POC: enable logical decoding when wal_level = 'replica' without a server restart

Hi all,

Logical decoding (and logical replication) are available only when
wal_level = logical. As the documentation says[1], Using the 'logical'
level increases the WAL volume which could negatively affect the
performance. For that reason, users might want to start with using
'replica', but when they want to use logical decoding they need a
server restart to increase wal_level to 'logical'. My goal is to allow
users who are using 'replica' level to use logical decoding without a
server restart. There are other GUC parameters related to logical
decoding and logical replication such as max_wal_senders,
max_logical_replication_workers, and max_replication_slots, but even
if users set these parameters >0, there would not be a noticeable
performance impact. And their default values are already >0. So I'd
like to focus on making only the wal_level dynamic GUC parameter.
There are several earlier discussions[2][3] but no one has submitted
patches unless I'm missing something.

The first idea I came up with is to make the wal_level a PGC_SIGHUP
parameter. However, it affects not only setting 'replica' to 'logical'
but also setting 'minimal' to 'replica' or higher. I'm not sure the
latter case is common and it might require a checkpoint. I don't want
to make the patch complex for uncommon cases.

The second idea is to somehow allow both WAL-logging logical info and
logical decoding even when wal_level is 'replica'. I've attached a PoC
patch for that. The patch introduces new SQL functions such as
pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
These functions are available only when wal_level is 'repilca'(or
higher). In pg_activate_logical_decoding(), we set the status of
logical decoding stored on the shared memory from 'disabled' to
'xlog-logical-info', allowing all processes to write logical
information to WAL records for logical decoding. But the logical
decoding is still not allowed. Once we confirm all in-progress
transactions completed, we switch the status to
'logical-decoding-ready', meaning that users can create logical
replication slots and use logical decoding.

Overall, with the patch, there are two ways to enable logical
decoding: setting wal_level to 'logical' and calling
pg_activate_logical_decoding() when wal_level is 'replica'. I left the
'logical' level for backward compatibility and for users who want to
enable the logical decoding without calling that SQL function. If we
can automatically enable the logical decoding when creating the first
logical replication slot, probably we no longer need the 'logical'
level. There is room to discuss the user interface. Feedback is very
welcome.

Regards,

[1] https://www.postgresql.org/docs/devel/runtime-config-wal.html#RUNTIME-CONFIG-WAL-SETTINGS
[2] https://www.postgresql.org/message-id/flat/CAKU4AWrv6zuywe1VBv6kwFmtaxyi5XYqpBkAG_B46cp4s4KoSw%40mail.gmail.com
[3] https://www.postgresql.org/message-id/20200608213215.mgk3cctlzvfuaqm6%40alap3.anarazel.de

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On Mon, Dec 30, 2024 at 10:44:38PM -0600, Masahiko Sawada wrote:
> Hi all,
>
> Logical decoding (and logical replication) are available only when
> wal_level = logical. As the documentation says[1], Using the 'logical'
> level increases the WAL volume which could negatively affect the
> performance. For that reason, users might want to start with using
> 'replica', but when they want to use logical decoding they need a
> server restart to increase wal_level to 'logical'. My goal is to allow
> users who are using 'replica' level to use logical decoding without a
> server restart.

Thanks for starting that thread and +1 for the idea!

> The first idea I came up with is to make the wal_level a PGC_SIGHUP
> parameter. However, it affects not only setting 'replica' to 'logical'
> but also setting 'minimal' to 'replica' or higher. I'm not sure the
> latter case is common and it might require a checkpoint. I don't want
> to make the patch complex for uncommon cases.
>
> The second idea is to somehow allow both WAL-logging logical info and
> logical decoding even when wal_level is 'replica'. I've attached a PoC
> patch for that. The patch introduces new SQL functions such as
> pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> These functions are available only when wal_level is 'repilca'(or
> higher). In pg_activate_logical_decoding(), we set the status of
> logical decoding stored on the shared memory from 'disabled' to
> 'xlog-logical-info', allowing all processes to write logical
> information to WAL records for logical decoding. But the logical
> decoding is still not allowed. Once we confirm all in-progress
> transactions completed, we switch the status to
> 'logical-decoding-ready', meaning that users can create logical
> replication slots and use logical decoding.
>
> Overall, with the patch, there are two ways to enable logical
> decoding: setting wal_level to 'logical' and calling
> pg_activate_logical_decoding() when wal_level is 'replica'. I left the
> 'logical' level for backward compatibility and for users who want to
> enable the logical decoding without calling that SQL function. If we
> can automatically enable the logical decoding when creating the first
> logical replication slot, probably we no longer need the 'logical'
> level. There is room to discuss the user interface. Feedback is very
> welcome.

If we don't want to force wal_level = logical to enable logical decoding (your
second idea) then I think that it would be better to "hide" everything behind
logical replication slot creation / deletion. That would mean that having at
least one logical replication slot created would be synonym to "activate logical
decoding" and zero logical replication slot created would be synonym to "deactivate
logical decoding".

That way:

1. an end user don't need to manipulate new functions and would just rely on
replication slots existence
2. we ensure that no extra WAL data is generated if not absolutely "needed"

Thoughts?

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Fri, Jan 3, 2025, at 10:14 AM, Bertrand Drouvot wrote:
> If we don't want to force wal_level = logical to enable logical decoding (your
> second idea) then I think that it would be better to "hide" everything behind
> logical replication slot creation / deletion. That would mean that having at
> least one logical replication slot created would be synonym to "activate logical
> decoding" and zero logical replication slot created would be synonym to "deactivate
> logical decoding".

I like this idea. The logical replication slot existence already has the
required protections and guarantees (no running transactions from the past while
creating) for logical decoding.

ERROR: logical decoding requires "wal_level" >= "logical"
STATEMENT: select pg_create_logical_replication_slot('test', 'pgoutput');

FATAL: logical replication slot "test" exists, but "wal_level" < "logical"
HINT: Change "wal_level" to be "logical" or higher.

Having said that, you are basically folding 'logical' machinery into 'replica'.
The 'logical' option can still exists but it will be less attractive because it
increases the WAL volume even if you are not using logical replication. I don't
know if the current 'logical' behavior (WAL records for logical decoding even
if there is no active logical replication) has any use case (maybe someone
inspects these extra records for analysis) but one suggestion (separate patch)
is to make 'logical' synonymous with the new 'replica' behavior (logical
decoding capability). This opens the door to remove 'logical' in future
releases (accepted as synonym but not documented).

--
Euler Taveira
EDB https://www.enterprisedb.com/

On Fri, Jan 3, 2025 at 6:31 AM Euler Taveira <euler(at)eulerto(dot)com> wrote:
>
> On Fri, Jan 3, 2025, at 10:14 AM, Bertrand Drouvot wrote:
>
> If we don't want to force wal_level = logical to enable logical decoding (your
> second idea) then I think that it would be better to "hide" everything behind
> logical replication slot creation / deletion. That would mean that having at
> least one logical replication slot created would be synonym to "activate logical
> decoding" and zero logical replication slot created would be synonym to "deactivate
> logical decoding".
>
>
> I like this idea. The logical replication slot existence already has the
> required protections and guarantees (no running transactions from the past while
> creating) for logical decoding.

I agree that it's better behavior.

>
> Having said that, you are basically folding 'logical' machinery into 'replica'.
> The 'logical' option can still exists but it will be less attractive because it
> increases the WAL volume even if you are not using logical replication. I don't
> know if the current 'logical' behavior (WAL records for logical decoding even
> if there is no active logical replication) has any use case (maybe someone
> inspects these extra records for analysis) but one suggestion (separate patch)
> is to make 'logical' synonymous with the new 'replica' behavior (logical
> decoding capability). This opens the door to remove 'logical' in future
> releases (accepted as synonym but not documented).

To enable the logical decoding automatically when the first slot is
created, probably we need to do something like:

1. enable WAL-logging logical info.
2. wait for all running transactions to finish.
3. enable logical decoding, and write a XLOG_LOGICAL_DECODING_STATUS record,
4. create a logical slot
4-1. reserve the WAL and write an XLOG_RUNNING_XACTS record.
4-2. wait for all transactions in the XLOG_RUNNING_XACTS record to
finish during creating the initial snapshot.

That is, we could need to wait for different sets of transactions to
finish (at 2 and 4-2). Which could surprise users since the first slot
creation could have users wait for a longer time. Using the 'logical'
level would avoid such waits.

Also, if we make the 'logical' level synonymous with the new 'replica'
behavior, we need to adjust regression tests as well. In some
regression tests (e.g., ondisk_startup.spec), we create a logical slot
while a transaction is concurrently running. We expect we wait for the
concurrent transaction during the slot creation but with the above
algorithm, we would wait for them to activate logical decoding. So I
agree that making the 'logical' level synonymous with the new
'replica' behavior is done in a separate patch. It would be a good
start to implement the new 'replica' level behavior.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Sat, Jan 4, 2025 at 6:03 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Fri, Jan 3, 2025 at 6:31 AM Euler Taveira <euler(at)eulerto(dot)com> wrote:
> >
> > On Fri, Jan 3, 2025, at 10:14 AM, Bertrand Drouvot wrote:
> >
> > If we don't want to force wal_level = logical to enable logical decoding (your
> > second idea) then I think that it would be better to "hide" everything behind
> > logical replication slot creation / deletion. That would mean that having at
> > least one logical replication slot created would be synonym to "activate logical
> > decoding" and zero logical replication slot created would be synonym to "deactivate
> > logical decoding".
> >
> >
> > I like this idea. The logical replication slot existence already has the
> > required protections and guarantees (no running transactions from the past while
> > creating) for logical decoding.
>
> I agree that it's better behavior.
>
> >
> > Having said that, you are basically folding 'logical' machinery into 'replica'.
> > The 'logical' option can still exists but it will be less attractive because it
> > increases the WAL volume even if you are not using logical replication. I don't
> > know if the current 'logical' behavior (WAL records for logical decoding even
> > if there is no active logical replication) has any use case (maybe someone
> > inspects these extra records for analysis) but one suggestion (separate patch)
> > is to make 'logical' synonymous with the new 'replica' behavior (logical
> > decoding capability). This opens the door to remove 'logical' in future
> > releases (accepted as synonym but not documented).
>
> To enable the logical decoding automatically when the first slot is
> created, probably we need to do something like:
>
> 1. enable WAL-logging logical info.
> 2. wait for all running transactions to finish.
> 3. enable logical decoding, and write a XLOG_LOGICAL_DECODING_STATUS record,
> 4. create a logical slot
> 4-1. reserve the WAL and write an XLOG_RUNNING_XACTS record.
> 4-2. wait for all transactions in the XLOG_RUNNING_XACTS record to
> finish during creating the initial snapshot.

A naive question: Can we combine step 2 and 4-2 - may be we need to
combine 4-1 and 3 too.

--
Best Wishes,
Ashutosh Bapat

On Mon, Jan 6, 2025 at 3:20 AM Ashutosh Bapat
<ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
>
> On Sat, Jan 4, 2025 at 6:03 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Fri, Jan 3, 2025 at 6:31 AM Euler Taveira <euler(at)eulerto(dot)com> wrote:
> > >
> > > On Fri, Jan 3, 2025, at 10:14 AM, Bertrand Drouvot wrote:
> > >
> > > If we don't want to force wal_level = logical to enable logical decoding (your
> > > second idea) then I think that it would be better to "hide" everything behind
> > > logical replication slot creation / deletion. That would mean that having at
> > > least one logical replication slot created would be synonym to "activate logical
> > > decoding" and zero logical replication slot created would be synonym to "deactivate
> > > logical decoding".
> > >
> > >
> > > I like this idea. The logical replication slot existence already has the
> > > required protections and guarantees (no running transactions from the past while
> > > creating) for logical decoding.
> >
> > I agree that it's better behavior.
> >
> > >
> > > Having said that, you are basically folding 'logical' machinery into 'replica'.
> > > The 'logical' option can still exists but it will be less attractive because it
> > > increases the WAL volume even if you are not using logical replication. I don't
> > > know if the current 'logical' behavior (WAL records for logical decoding even
> > > if there is no active logical replication) has any use case (maybe someone
> > > inspects these extra records for analysis) but one suggestion (separate patch)
> > > is to make 'logical' synonymous with the new 'replica' behavior (logical
> > > decoding capability). This opens the door to remove 'logical' in future
> > > releases (accepted as synonym but not documented).

FYI one possible use case of the 'logical' level would be that if we
support retrieving WAL records for logical decoding from somewhere
like restore_command in the future, users would like to keep the
wal_level 'logical' or keep the logical decoding active.

> >
> > To enable the logical decoding automatically when the first slot is
> > created, probably we need to do something like:
> >
> > 1. enable WAL-logging logical info.
> > 2. wait for all running transactions to finish.
> > 3. enable logical decoding, and write a XLOG_LOGICAL_DECODING_STATUS record,
> > 4. create a logical slot
> > 4-1. reserve the WAL and write an XLOG_RUNNING_XACTS record.
> > 4-2. wait for all transactions in the XLOG_RUNNING_XACTS record to
> > finish during creating the initial snapshot.
>
> A naive question: Can we combine step 2 and 4-2 - may be we need to
> combine 4-1 and 3 too.

Yeah, maybe we can somewhat optimize this process. Probably we can do like:

1. create a logical slot.
2. enable WAL-logging logical info.
3. reserve the WAL and write an XLOG_RUNNING_XACTS record.
4. wait for all transactions in the XLOG_RUNNING_XACTS record to
finish during creating the initial snapshot.
4-1. write a XLOG_LOGICAL_DECODING_STATUS record if a process
detects all transactions in the XLOG_RUNNING_XACTS record to finish.

However, we need to note that 4-1 should be done by only one process.
Also, it's a bit weird and concerns to me that the snapbuild
initialization contains the process of activating logical decoding.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi Sawada-San.

FWIW, I also thought it was a good idea suggested by Bertrand [1] to
"hide" everything behind the slot create/delete, and thereby eliminate
the need for user intervention using those new
pg_activate/deactivate_logical_decoding functions.

But, one concern doing it this way is how to prevent a user
(accidentally?) getting themselves into a different replication mode
without realising it? Or say they change the mode and then "forget"
that they did that.

For example, If wal_level=replica and then somebody does CREATE
PUBLICATION/SUBSCRIPTION won't that implicitly enable the logical
decoding and then leave it enabled until all the logical replication
slots are eventually dropped? Now, when the user examines wal_level it
is still going to show 'replica', which could be misleading --- e.g.
how will they even know that they can't really trust that anymore, and
they must also check the pg_get_logical_decoding_status?

At least, there should be some big NOTICE/WARNING logs written
whenever the logical decoding status is getting implicitly changed by
logical replication slot create/delete..

~~~

Meanwhile, although it is premature to give detailed code review
comments for what is still a PoC, during my reading of your patch I
made copious notes for myself, so I thought I might as well post what
I have regardless. See the details below in this post. Ignore, or do
with them as you wish. But, hopefully, a lot of the following feedback
will still be relevant regardless of any design changes.

//////////

Commit message

1.
Patch description is missing

======
GENERAL

2. git apply warning

[postgres(at)CentOS7-x64 oss_postgres_misc]$ git apply
../patches_misc/PoC_online_activate_logical_decoding.patch
../patches_misc/PoC_online_activate_logical_decoding.patch:93: new
blank line at EOF.
+
warning: 1 line adds whitespace errors.

~~~

3. Copyright date

All the new files of this patch have the wrong copyright date (2024
instead of 2025).

~~~

4. Missing documentation

There are missing docs changes, but that is to be expected while the
design is in flux.

~~~

5. Terminology

There are multiple places where similar terms like "active" and
"enabled" and "ready" are being used apparently interchangeably making
it quite confusing. Perhaps it was your intention to convey the idea
that the logical mode might be "enabled" but is not yet "active", but
I didn't think that was the case. Mostly it just seemed arbitrary to
me.

IMO the state starting out as "disabled" should end up as "enabled"
(not "ready").

Meanwhile, the API to change the status is currently called
"activate/deactivate" instead of "enable/disable" (??)

Meanwhile, many/most of the error messages talk about "enabled" (not
"activated" or "ready"). e.g. "logical decoding needs to be enabled to
publish logical changes"

Meanwhile, macros are called 'IsLogicalDecodingActive'; why isn't that
called 'IsLogicalDecodingEnabled'.

~

OTOH, maybe you chose the term "active" because master already had the
macro XLogLogicalInfoActive(). But in that case maybe it should be
saying inactive/active everywhere instead of disabled/enabled/ready...

~~~

6. Typedefs

There are some new typedefs introduced in the new files, So these
should be updated in the typedefs.list file.

~~~

7. Tri-state logical decoding status

Currently there is a tri-state logical decoding status: "disabled" ->
"xlog-logicalinfo" -> "ready"

AFAICT that middle state is just transitory. It is easier for me to
visualize it like:
logical decoding status: "disabled" -> "pending" -> "enabled".

IIUC the "pending" part is when it is just waiting for any running
transaction to complete.

IMO it becomes simpler if you do not make any distinction between WAL
logging and logical decoding ability. E.g. Do not allow logical WAL
logging for that 'middle' transitory state; only allow it after
logical decoding becomes fully activated.

IIUC, it would just a small change to do this:

CURRENT (pseudocode):
XLogLogicalInfoEnabled(void) { return XLogLogicalInfoCtl->status >=
LOGICAL_DECODING_STATUS_XLOG_LOGICALINFO; }

SUGGESTION (pseudocode)
XLogLogicalInfoEnabled(void) { return XLogLogicalInfoCtl->status ==
LOGICAL_DECODING_STATUS_READY; }

======
contrib/test_decoding/t/002_online_activation.pl

8.
+# Copyright (c) 2021-2024, PostgreSQL Global Development Group

/2024/2025/

~~~

9.
+# check if a logical replication slot cannot be craeted while logical

/check/Check/

/cannot/can/

/craeted/created/

~~~

10.
+ok( $stderr =~ /ERROR: logical decoding requires "wal_level" >= "logical"/,
+ "logical replication slot cannot be created while logical
decoding is disableed");

/disableed/disabled/

~~~

11.
+# Activate logical info logging.
+$node->safe_psql('postgres',
+ 'SELECT pg_activate_logical_decoding()');
+$ret = $node->safe_psql('postgres', 'SELECT pg_get_logical_decoding_status()');
+is($ret, 'ready', 'logical decoding gets activated');

Perhaps there should be another "show wal_level" done here to show
that even after logical decoding is enabled, the wal_level remains the
same.

~~~

12.
+$node->safe_psql(
+ 'postgres',
+ q[
+CREATE TABLE test (a int primary key, b int);
+INSERT INTO test values (1, 100), (2, 200);
+UPDATE test SET b = b + 10 WHERE a = 1;
+ ]);
+
+$ret = $node->safe_psql(
+ 'postgres',
+ q[SELECT data FROM
pg_logical_slot_get_changes('regression_slot1', NULL, NULL,
'include-xids', '0', 'skip-empty-xacts', '1');]
+ );
+is($ret, 'BEGIN
+table public.test: INSERT: a[integer]:1 b[integer]:100
+table public.test: INSERT: a[integer]:2 b[integer]:200
+COMMIT
+BEGIN
+table public.test: UPDATE: a[integer]:1 b[integer]:110
+COMMIT', 'logical decoding works fine');
+

There seems to be some comment missing for all this part. I expected
it to say something like "Check that logical decoding is working OK".

======
src/backend/access/heap/heapam.c

13.
- * This is only used in wal_level >= WAL_LEVEL_LOGICAL, and only for catalog
- * tuples.
+ * This is only used when XLogLogicalInfoActive() is true, and only for
+ * catalog tuples.

I felt it would be better to Assert this in the code, instead of just
mentioning it in the function comment.

======
src/backend/access/rmgrdesc/xlogdesc.c

14.
+ else if (info == XLOG_LOGICAL_DECODING_STATUS)
+ {
+ bool logical_decoding;
+
+ memcpy(&logical_decoding, rec, sizeof(bool));
+ appendStringInfoString(buf, logical_decoding ? "true" : "false");
+ }

Should there be some textual context for this bool value, like wal_level has?

SUGGESTION
appendStringInfoString(buf, "logical decoding %s", logical_decoding ?
"true" : "false");

~~~

15.
I noticed there are lots of other descriptions in this function that
are showing the value of 'wal_level' via the static function call:
get_wal_level_string(wal_level).

But, since the wal_level now might be out-of-step with the
logcal_decoding boolean value, shouldn't you be showing the logical
decoding state everywhere too?

e.g. you might have a combination like:
wal_level replica
logical decoding true

======
src/backend/access/transam/xlog.c

BootStrapXLOG:

16.
+ checkPoint.logicalDecoding = false;
checkPoint.wal_level = wal_level;

Shouldn't the default for 'logicalDecoding' be consistent with the
assigned wal_level? It seems strange if it is possible to have a
situation where wal_level = logical but simultaneously logicalDecoding
= false.

~~~

xlog_redo:

17.
+ else if (info == XLOG_LOGICAL_DECODING_STATUS)
+ {
+ bool enabled;

A less generic variable name like "logical_decoding_enabled" might be better.

======
src/backend/commands/publicationcmds.c

CreatePublication:

18.
- if (wal_level != WAL_LEVEL_LOGICAL)
+ if (!XLogLogicalInfoActive())
ereport(WARNING,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("\"wal_level\" is insufficient to publish logical changes"),
- errhint("Set \"wal_level\" to \"logical\" before creating subscriptions.")));
+ errmsg("logical decoding needs to be enabled to publish logical changes"),
+ errhint("Set \"wal_level\" to \"logical\" or activate logical
decoding before creating subscriptions.")));

18a.
/needs to be enabled/must be enabled/

~

18b.
The hint seems less useful than it could be because it does not say
*how* the user can activate logical decoding.

SUGGESTION
Set \"wal_level\" to \"logical\" or use pg_activate_logical_decoding
before creating subscriptions.

======
src/backend/replication/logical/logical.c

CheckLogicalDecodingRequirements:

19.
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("logical decoding on standby requires \"wal_level\" >=
\"logical\" on the primary")));
+ errmsg("logical decoding on standby requires to enable logical
decoding on the primary")));
19a
/requires to enable logical decoding/requires logical decoding to be enabled/

~

19b.
This message is like what was in CreatePublication, so might be better
split into an errmsg and errhint.

e.g.
errmsg("logical decoding on standby requires logical decoding to be
enabled on the primary");
errhint("Set \"wal_level\" to \"logical\" or use
pg_activate_logical_decoding.");

======
src/backend/replication/logical/logicalxlog.c

20. General

Would it be better if all the exposed functions from here used the
common "LogicalXLog" prefix?

These ones do:
- LogicalXlogShmemSize
- LogicalXlogShmemInit

These ones don't:
- StartupLogicalDecodingStatus
- UpdateLogicalDecodingStatus
- XLogLogicalInfoEnabled (this one is backwards)
- IsLogicalDecodingActive

~~~

21.
+ * logicalxlog.c
+ * This module contains the codes for enabling or disabling to include
+ * logical information into WAL records and logical decoding.

/codes/code/

Also, this is worded as if you can set those states independently, but
AFAICT the "logical information into WAL" is just a transition-state
to the LOGICAL_DECODING_STATUS_READY.

SUGGESTION
This module contains code to enable/disable and fetch the logical
decoding status.

~~~

22.
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group

/2024/2025/

~~~

23.
+XLogLogicalInfoCtlData *XLogLogicalInfoCtl = NULL;

Add blank line above this.

~~~

LogicalXlogShmemSize:

24.
+/*
+ * Initialization of shared memory.
+ */
+
+Size
+LogicalXlogShmemSize(void)
+{
+ return sizeof(XLogLogicalInfoCtlData);
+}

Unnecessary blank line after the function comment.

~~~

LogicalXlogShmemInit:

25.
+void
+LogicalXlogShmemInit(void)

Missing function comment.

~~~

StartupLogicalDecodingStatus:

26.
+/*
+ * This must be called ONCE during postmaster or standalone-backend startup,
+ * before calling StartupReplicationSlots().
+ */
+void
+StartupLogicalDecodingStatus(bool enabled_at_last_checkpoint)

The function comment does not actually say what this function does.
Maybe add a first sentence like:
"Assign the initial LogicalDecodingStatus value."

~~~

27.
+ /*
+ * On standbys, we always start with the status in the last checkpoint
+ * record. If changes of wal_level or logical decoding status is sent from
+ * the primary, we will enable or disable the logical decoding while
+ * replaying the WAL record and invalidate slots if necessary.
+ */

/is sent from/are sent from/

/invalidate/invalidating/

~~~

28.
+ if (!StandbyMode)
+ {
+ /*
+ * Disable logical decoding if replication slots are not available.
+ */
+ if (max_replication_slots == 0)
+ status = LOGICAL_DECODING_STATUS_DISABLED;
+
+ /*
+ * If previously the logical decoding was active but the server
+ * restarted with wal_level < 'replica', we disable the logical
+ * decoding.
+ */
+ if (wal_level < WAL_LEVEL_REPLICA)
+ status = LOGICAL_DECODING_STATUS_DISABLED;
+
+ /*
+ * Setting wal_level to 'logical' immediately enables logical decoding
+ * and WAL-logging logical info.
+ */
+ if (wal_level >= WAL_LEVEL_LOGICAL)
+ status = LOGICAL_DECODING_STATUS_READY;
+ }

28a.
The comment is a bit confusing because it is only talking about
Standby, whereas the code block is all for !StandbyMode.

It would be more easy to read if refactored to put the comment in an
empty code block like below:

if (StandByMode)
{
/*
* On standbys, we always start with the status in the last checkpoint
* record...
* ...
*/
}
else
{
<code>
}

~

28b.
What if the first condition (max_replication_slots is 0) and last
condition (WAL_LEVEL_LOGICAL) are true at the same time? Currently
this initializes the status as READY. Is it OK?

~~~

UpdateLogicalDecodingStatus:

29.
+void
+UpdateLogicalDecodingStatus(bool activate)
+{
+ XLogLogicalInfoCtl->status = activate
+ ? LOGICAL_DECODING_STATUS_READY
+ : LOGICAL_DECODING_STATUS_DISABLED;
+}

29a.
Missing function comment.

~

29b.
IMO the parameter should be in past tense, but I also think should be
called "enabled" (same as what it was called in the extern in
logicalxlog.h)

~~~

XLogLogicalInfoEnabled:

30.
+/*
+ * Is WAL-Logging logical info enabled?
+ */
+bool inline
+XLogLogicalInfoEnabled(void)
+{
+ /*
+ * Use volatile pointer to make sure we make a fresh read of the shared
+ * variable.
+ */
+ volatile XLogLogicalInfoCtlData *ctl = XLogLogicalInfoCtl;
+
+ return ctl->status >= LOGICAL_DECODING_STATUS_XLOG_LOGICALINFO;
+}

How necessary is this function?

I thought that LOGICAL_DECODING_STATUS_XLOG_LOGICALINFO is more like
just a transitory state between LOGICAL_DECODING_STATUS_DISABLED and
LOGICAL_DECODING_STATUS_READY. So in that case, isn't it simpler just
to wait for the READY? e.g. remove this function and just use
IsLogicalDecodingActive() instead?

~~~

do_activate_logical_decoding:

31.
Missing function comment.

~~~

32.
+ /*
+ * Get the latest status of logical info logging. If it's already
+ * activated, quick return.
+ */
+ SpinLockAcquire(&XLogLogicalInfoCtl->mutex);
+ if (XLogLogicalInfoCtl->status >= LOGICAL_DECODING_STATUS_XLOG_LOGICALINFO)
+ {
+ SpinLockRelease(&XLogLogicalInfoCtl->mutex);
+ return;
+ }

32a.
I didn't think the comment should mention "logical info logging" here.

SUGGESTION
Quick return if logical decoding is already activated (or activation
was already requested and is pending).

~

32b.
Related to this, it seems simpler to check condition "if
(XLogLogicalInfoCtl->status > LOGICAL_DECODING_STATUS_DISABLED)"

~~~

33.
+ PG_ENSURE_ERROR_CLEANUP(logical_decoding_activation_abort_callback, 0);
+ {
...
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(logical_decoding_activation_abort_callback, 0);

IIUC, this whole chunk of code lies between the status of
"xlog-logicalinfo" and "ready".

I think it needs a comment something like:
/* Confirm that all in-progress transactions have completed before
logical decoding can be enabled. */

~~~

logical_decoding_activation_abort_callback:

34.
Missing function comment.

~~~

35.
Since this is a callback for the function
do_activate_logical_decoding(), I thought the function should be
slightly renamed to match the caller.

/logical_decoding_activation_abort_callback/activate_logical_decoding_abort_callback/

~~~

36.
IIUC, the status could have reached
LOGICAL_DECODING_STATUS_XLOG_LOGICALINFO before we encountered some
problem trying to get to the READY status, and so reset everything
back to DISABLED. So some (unusable?) logical WAL records could have
been written while in that intermediate state? It is another reason
why I was unsure about the need for the earlier XLogLogicalInfoEnabled
function.

Won't it be simpler to just have a DISABLED and ENABLED state and just
keep the transition phase for internal logic? IOW, make
XLogLogicalInfoEnabled() return true only when READY/ENABLED.

~~~
pg_activate_logical_decoding:

37.
Missing function comment.

~~~

38.
+ if (IsTransactionState() &&
+ GetTopTransactionIdIfAny() != InvalidTransactionId)
+ ereport(ERROR,
+ (errcode(ERRCODE_ACTIVE_SQL_TRANSACTION),
+ errmsg("cannot activate logical decoding in transaction that has
performed writes")));
+
+

Double blank lines.

~~~

39.
+ SpinLockAcquire(&(XLogLogicalInfoCtl->mutex));
+ status = XLogLogicalInfoCtl->status;
+ SpinLockRelease(&(XLogLogicalInfoCtl->mutex));
+
+ if (status == LOGICAL_DECODING_STATUS_READY)
+ PG_RETURN_VOID();

Should have some comment like: "If logical decoding is already enabled
then there is nothing to do."

OTOH, can't all this code be removed, since the top-of-loop code will
do the same thing anyway (I didn't think the extra sleep
prepare/cancel that it is doing was something to worry about).

~~~

40.
+ ConditionVariableSleep(&XLogLogicalInfoCtl->cv,
+ WAIT_EVENT_LOGICAL_DECODING_ACTIVATION);
+
+ continue;

Remove the blank line.

~~~

41.
+ /* Activate logical decoding */
+ do_activate_logical_decoding();

The comment is not helpful because it is practically the same as the
function name.

~~~

pg_deactivate_logical_decoding:

42.
Missing function comment.

~~~

43.
+ /*
+ * Hold ReplicationSlotAllocationLock to prevent slots from newly
+ * being created while deactivating the logical decoding.
+ */
+ LWLockAcquire(ReplicationSlotAllocationLock, LW_SHARED);

/newly being created/being created/

~~~

44.
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (!s->in_use)
+ continue;
+
+ if (SlotIsPhysical(s))
+ continue;
+
+ if (s->data.invalidated != RS_INVAL_NONE)
+ continue;
+
+ valid_logical_slots++;
+ }

IMO it is more natural here to check "if (!SlotIsLogical(s))" instead
of "if (SlotIsPhysical(s))".

~~~

45.
+ if (valid_logical_slots > 0)
+ ereport(ERROR,
+ (errmsg("cannot deactivate logical decoding while having valid
logical replication slots")));

/while having valid logical replication slots/while valid logical
replication slots are present/

~~~

46.
+ if (XLogStandbyInfoActive() && !recoveryInProgress)
+ {
+ bool enabled = false;
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&enabled), sizeof(bool));
+
+ XLogInsert(RM_XLOG_ID, XLOG_LOGICAL_DECODING_STATUS);
+ }
+
+ SpinLockAcquire(&(XLogLogicalInfoCtl->mutex));
+ XLogLogicalInfoCtl->status = LOGICAL_DECODING_STATUS_DISABLED;
+ SpinLockRelease(&(XLogLogicalInfoCtl->mutex));

In the pg_activate_logical_decoding function, the assignment of the
status and the writing of the XLog record was done within a critical
section. Why isn't the code doing the same here?

~~~

pg_get_logical_decoding_status:

47.
Missing function comment.

======
src/backend/replication/logical/slotsync.c

ValidateSlotSyncParams:

48.
- if (wal_level < WAL_LEVEL_LOGICAL)
+ if (!XLogLogicalInfoActive())
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("replication slot synchronization requires \"wal_level\" >=
\"logical\""));
+ errmsg("replication slot synchronization requires \"wal_level\" >=
\"logical\" or to activate logical decoding"));

This is another message like what was in CreatePublication, and that
might be better split into an errmsg and errhint.

e.g.
errmsg("replication slot synchronization requires logical decoding to
be enabled");
errhint("Set \"wal_level\" to \"logical\" or use
pg_activate_logical_decoding.");

======
src/backend/replication/slot.c

RestoreSlotFromDisk:

49.
+ if (cp.slotdata.database != InvalidOid && !XLogLogicalInfoActive())
ereport(FATAL,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("logical replication slot \"%s\" exists, but \"wal_level\" <
\"logical\"",
NameStr(cp.slotdata.name)),
- errhint("Change \"wal_level\" to be \"logical\" or higher.")));
+ errhint("Change \"wal_level\" to be \"logical\" or higher, or
activate logical decoding with \"replica\".")));

Another message that could be made more consistent with the similar
ones from CreatePublication, ValidateSlotSyncParams, etc...

e.g.
errmsg("logical replication slot \"%s\" exists, but logical decoding
is not enabled");
errhint("Set \"wal_level\" to \"logical\" or use
pg_activate_logical_decoding.");

======
src/backend/utils/activity/wait_event_names.txt

50.
+LOGICAL_DECODING_ACTIVATION "Waiting for logical decoding to be activated."

Should this be more like "Waiting for a requested logical decoding
activation to complete."

======
src/include/access/xlog.h

51.
-#define XLogLogicalInfoActive() (wal_level >= WAL_LEVEL_LOGICAL)
+#define XLogLogicalInfoActive() (XLogLogicalInfoEnabled())

Those extra parentheses seem redundant now.

======
src/include/replication/logicalxlog.h

52.
+/*-------------------------------------------------------------------------
+ * logicalxlog.h
+ * Exports from replication/logical/logicalxlog.c.
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------

/2024/2025/

~~~

53.
+typedef enum LogicalDecodingStatus
+{
+ LOGICAL_DECODING_STATUS_DISABLED = 0,
+ LOGICAL_DECODING_STATUS_XLOG_LOGICALINFO,
+ LOGICAL_DECODING_STATUS_READY,
+} LogicalDecodingStatus;

Maybe some more comments about these status phases would be helpful.
Specifically, it's not immediately clear if the
LOGICAL_DECODING_STATUS_XLOG_LOGICALINFO is a real state we could find
ourselves permanently, or if it is more like just a temporary
"pending" state while transitioning from DISABLED to READY.

AFAICT it is just transitory, so in that case, maybe
LOGICAL_DECODING_PENDING might be a more appropriate name.

~~~

54.
+typedef struct XLogLogicalInfoCtlData XLogLogicalInfoCtlData;
+extern XLogLogicalInfoCtlData *XLogLogicalInfoCtl;
+extern bool XLogLogicalInfo;

Add a blank line between the typedef and the externs.

======
[1] Bertrand's hide everything idea -
https://www.postgresql.org/message-id/Z3fimYj0fbkLmWJb%40ip-10-97-1-34.eu-west-3.compute.internal

Kind Regards,
Peter Smith.
Fujitsu Australia

On Tue, Dec 31, 2024 at 10:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> Hi all,
>
> Logical decoding (and logical replication) are available only when
> wal_level = logical. As the documentation says[1], Using the 'logical'
> level increases the WAL volume which could negatively affect the
> performance. For that reason, users might want to start with using
> 'replica', but when they want to use logical decoding they need a
> server restart to increase wal_level to 'logical'. My goal is to allow
> users who are using 'replica' level to use logical decoding without a
> server restart. There are other GUC parameters related to logical
> decoding and logical replication such as max_wal_senders,
> max_logical_replication_workers, and max_replication_slots, but even
> if users set these parameters >0, there would not be a noticeable
> performance impact. And their default values are already >0. So I'd
> like to focus on making only the wal_level dynamic GUC parameter.
> There are several earlier discussions[2][3] but no one has submitted
> patches unless I'm missing something.
>
> The first idea I came up with is to make the wal_level a PGC_SIGHUP
> parameter. However, it affects not only setting 'replica' to 'logical'
> but also setting 'minimal' to 'replica' or higher. I'm not sure the
> latter case is common and it might require a checkpoint. I don't want
> to make the patch complex for uncommon cases.
>
> The second idea is to somehow allow both WAL-logging logical info and
> logical decoding even when wal_level is 'replica'. I've attached a PoC
> patch for that. The patch introduces new SQL functions such as
> pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> These functions are available only when wal_level is 'repilca'(or
> higher). In pg_activate_logical_decoding(), we set the status of
> logical decoding stored on the shared memory from 'disabled' to
> 'xlog-logical-info', allowing all processes to write logical
> information to WAL records for logical decoding. But the logical
> decoding is still not allowed. Once we confirm all in-progress
> transactions completed, we switch the status to
> 'logical-decoding-ready', meaning that users can create logical
> replication slots and use logical decoding.
>
> Overall, with the patch, there are two ways to enable logical
> decoding: setting wal_level to 'logical' and calling
> pg_activate_logical_decoding() when wal_level is 'replica'. I left the
> 'logical' level for backward compatibility and for users who want to
> enable the logical decoding without calling that SQL function. If we
> can automatically enable the logical decoding when creating the first
> logical replication slot, probably we no longer need the 'logical'
> level. There is room to discuss the user interface. Feedback is very
> welcome.
>

If a server is running at minimal wal_level and they want to enable
logical replication, they would still need a server restart. That
would be rare but not completely absent.

Our documentation says "wal_level determines how much information is
written to the WAL.". Users would may not expect that the WAL amount
changes while wal_level = replica depending upon whether logical
decoding is possible. It may be possible to set the expectations right
by changing the documentation. It's not in the patch, so I am not sure
whether this is considered.

Cloud providers do not like multiple ways of changing configuration
esp. when they can not control it. See [1]. Changing wal_level through
a SQL function may fit the same category.

I agree that it would be a lot of work to make all combinations of
wal_level changes work, but changing wal_level through SIGHUP looks
like a cleaner solution. Is there way that we make the GUC SIGHUP but
disallow certain combinations of old and new values?

[1] https://www.postgresql.org/message-id/flat/CA%2BVUV5rEKt2%2BCdC_KUaPoihMu%2Bi5ChT4WVNTr4CD5-xXZUfuQw%40mail.gmail.com

--
Best Wishes,
Ashutosh Bapat

On Thu, Jan 9, 2025 at 3:29 AM Ashutosh Bapat
<ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
>
> On Tue, Dec 31, 2024 at 10:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > Hi all,
> >
> > Logical decoding (and logical replication) are available only when
> > wal_level = logical. As the documentation says[1], Using the 'logical'
> > level increases the WAL volume which could negatively affect the
> > performance. For that reason, users might want to start with using
> > 'replica', but when they want to use logical decoding they need a
> > server restart to increase wal_level to 'logical'. My goal is to allow
> > users who are using 'replica' level to use logical decoding without a
> > server restart. There are other GUC parameters related to logical
> > decoding and logical replication such as max_wal_senders,
> > max_logical_replication_workers, and max_replication_slots, but even
> > if users set these parameters >0, there would not be a noticeable
> > performance impact. And their default values are already >0. So I'd
> > like to focus on making only the wal_level dynamic GUC parameter.
> > There are several earlier discussions[2][3] but no one has submitted
> > patches unless I'm missing something.
> >
> > The first idea I came up with is to make the wal_level a PGC_SIGHUP
> > parameter. However, it affects not only setting 'replica' to 'logical'
> > but also setting 'minimal' to 'replica' or higher. I'm not sure the
> > latter case is common and it might require a checkpoint. I don't want
> > to make the patch complex for uncommon cases.
> >
> > The second idea is to somehow allow both WAL-logging logical info and
> > logical decoding even when wal_level is 'replica'. I've attached a PoC
> > patch for that. The patch introduces new SQL functions such as
> > pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> > These functions are available only when wal_level is 'repilca'(or
> > higher). In pg_activate_logical_decoding(), we set the status of
> > logical decoding stored on the shared memory from 'disabled' to
> > 'xlog-logical-info', allowing all processes to write logical
> > information to WAL records for logical decoding. But the logical
> > decoding is still not allowed. Once we confirm all in-progress
> > transactions completed, we switch the status to
> > 'logical-decoding-ready', meaning that users can create logical
> > replication slots and use logical decoding.
> >
> > Overall, with the patch, there are two ways to enable logical
> > decoding: setting wal_level to 'logical' and calling
> > pg_activate_logical_decoding() when wal_level is 'replica'. I left the
> > 'logical' level for backward compatibility and for users who want to
> > enable the logical decoding without calling that SQL function. If we
> > can automatically enable the logical decoding when creating the first
> > logical replication slot, probably we no longer need the 'logical'
> > level. There is room to discuss the user interface. Feedback is very
> > welcome.
> >
>
> If a server is running at minimal wal_level and they want to enable
> logical replication, they would still need a server restart. That
> would be rare but not completely absent.

Currently we don't allow the server to start with the 'minimal' level
and max_wal_senders > 0. Even if we support changing 'minimal' to
'logical' without a server restart, we still need a server restart to
increase max_wal_senders for users who want to use logical
replication. Or we need to eliminate this restriction too. I guess it
would be too complex for such uncommon use cases.

>
> Our documentation says "wal_level determines how much information is
> written to the WAL.". Users would may not expect that the WAL amount
> changes while wal_level = replica depending upon whether logical
> decoding is possible. It may be possible to set the expectations right
> by changing the documentation. It's not in the patch, so I am not sure
> whether this is considered.

We should mention that in the doc. The WAL amount changes depending on
not only wal_level but also other parameters such as wal_log_hints and
full_page_writes.

> Cloud providers do not like multiple ways of changing configuration
> esp. when they can not control it. See [1]. Changing wal_level through
> a SQL function may fit the same category.

Thank you for pointing it out. This would support the idea of
automatically enabling logical decoding.

> I agree that it would be a lot of work to make all combinations of
> wal_level changes work, but changing wal_level through SIGHUP looks
> like a cleaner solution. Is there way that we make the GUC SIGHUP but
> disallow certain combinations of old and new values?

While I agree that it's cleaner I think there is no way today. I think
we need to invent something for that.

Another idea would be to have another SIGHUP GUC parameter to control
logical info as another way to enable logical info WAL-logging while
trying to deprecate the 'logical' level over some releases. While it
doesn't need a SQL function, it could confuse users since we will
require two GUC parameters for doing things that we used to use one
GUC parameter.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On Tue, Jan 7, 2025 at 11:30 PM Peter Smith <smithpb2250(at)gmail(dot)com> wrote:
>
> Hi Sawada-San.
>
> FWIW, I also thought it was a good idea suggested by Bertrand [1] to
> "hide" everything behind the slot create/delete, and thereby eliminate
> the need for user intervention using those new
> pg_activate/deactivate_logical_decoding functions.
>
> But, one concern doing it this way is how to prevent a user
> (accidentally?) getting themselves into a different replication mode
> without realising it? Or say they change the mode and then "forget"
> that they did that.

I think that as long as there is at least one logical replication slot
users still have the will to execute logical decoding, so they require
logical info WAL-logging. I think it would rather be good for users to
be able to forget about that.

> For example, If wal_level=replica and then somebody does CREATE
> PUBLICATION/SUBSCRIPTION won't that implicitly enable the logical
> decoding and then leave it enabled until all the logical replication
> slots are eventually dropped?

Yes.

> Now, when the user examines wal_level it
> is still going to show 'replica', which could be misleading --- e.g.
> how will they even know that they can't really trust that anymore, and
> they must also check the pg_get_logical_decoding_status?

I think that if we automatically enable logical decoding even when
wal_level=replica, we would not need pg_get_logical_decoding_status().
These changes would break compatibility tools checking if wal_level is
'logical', but I guess that tools will not need to check that anymore.
Users can simply understand that if there is at least one logical
replication slot the system writes necessary information to use it.
Having said that we anyway need to mention it in the doc and raising
NOTICE/WARNING would also be a good idea as you mentioned.

>
> Meanwhile, although it is premature to give detailed code review
> comments for what is still a PoC, during my reading of your patch I
> made copious notes for myself, so I thought I might as well post what
> I have regardless. See the details below in this post. Ignore, or do
> with them as you wish. But, hopefully, a lot of the following feedback
> will still be relevant regardless of any design changes.

Thank you for reviewing the patch! It seems that most comments are
about wording, typos, suggestions for variable names and function
names etc. Since the patch is in the PoC/design phase where we might
need to change the whole design, I'll refer to these comments when
updating the patch next time.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Fri, Jan 10, 2025 at 8:28 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Tue, Jan 7, 2025 at 11:30 PM Peter Smith <smithpb2250(at)gmail(dot)com> wrote:
> >
> > Hi Sawada-San.
> >
> > FWIW, I also thought it was a good idea suggested by Bertrand [1] to
> > "hide" everything behind the slot create/delete, and thereby eliminate
> > the need for user intervention using those new
> > pg_activate/deactivate_logical_decoding functions.
> >
> > But, one concern doing it this way is how to prevent a user
> > (accidentally?) getting themselves into a different replication mode
> > without realising it? Or say they change the mode and then "forget"
> > that they did that.
>
> I think that as long as there is at least one logical replication slot
> users still have the will to execute logical decoding, so they require
> logical info WAL-logging. I think it would rather be good for users to
> be able to forget about that.
>
> > For example, If wal_level=replica and then somebody does CREATE
> > PUBLICATION/SUBSCRIPTION won't that implicitly enable the logical
> > decoding and then leave it enabled until all the logical replication
> > slots are eventually dropped?
>
> Yes.
>
> > Now, when the user examines wal_level it
> > is still going to show 'replica', which could be misleading --- e.g.
> > how will they even know that they can't really trust that anymore, and
> > they must also check the pg_get_logical_decoding_status?
>
> I think that if we automatically enable logical decoding even when
> wal_level=replica, we would not need pg_get_logical_decoding_status().
> These changes would break compatibility tools checking if wal_level is
> 'logical', but I guess that tools will not need to check that anymore.
> Users can simply understand that if there is at least one logical
> replication slot the system writes necessary information to use it.
> Having said that we anyway need to mention it in the doc and raising
> NOTICE/WARNING would also be a good idea as you mentioned.
>

Hi Sawada-San

IMO it seems misleading to be having logical replication behaviour
while the wal_level still displays as 'replica'.

So, I was wondering if it would be worthwhile to introduce a
'show_hook' for this GUC.

I hacked a quick implementation (attached) which now displays a middle
value for this scenario as "replica-logical". See below:

test_pub=# show wal_level;
wal_level
-----------
replica
(1 row)

test_pub=# select pg_get_logical_decoding_status();
pg_get_logical_decoding_status
--------------------------------
disabled
(1 row)

test_pub=# select pg_activate_logical_decoding();
pg_activate_logical_decoding
------------------------------

(1 row)

test_pub=# show wal_level;
wal_level
-----------------
replica-logical
(1 row)

test_pub=# select pg_get_logical_decoding_status();
pg_get_logical_decoding_status
--------------------------------
ready

(1 row)
test_pub=# \x
Expanded display is on.
test_pub=# select * from pg_settings where name in ('wal_level');
-[ RECORD 1 ]---+--------------------------------------------------
name | wal_level
setting | replica-logical
unit |
category | Write-Ahead Log / Settings
short_desc | Sets the level of information written to the WAL.
extra_desc |
context | postmaster
vartype | enum
source | configuration file
min_val |
max_val |
enumvals | {minimal,replica,logical}
boot_val | replica
reset_val | replica
sourcefile | /home/postgres/MYDATAOSS_2PC_PUB/postgresql.conf
sourceline | 199
pending_restart | f

~~

OTOH, you might prefer to display the "replica-logical" case as
"logical". (doing so might make things easier for existing tools that
are already checking if wal_level is "logical").

Of course, it doesn't really change the 'wal_level' variable value, it
just changes how it is *displayed*. Anyway, maybe it's an idea to
consider.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Tue, 31 Dec 2024 at 10:15, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> Hi all,
>
> Logical decoding (and logical replication) are available only when
> wal_level = logical. As the documentation says[1], Using the 'logical'
> level increases the WAL volume which could negatively affect the
> performance. For that reason, users might want to start with using
> 'replica', but when they want to use logical decoding they need a
> server restart to increase wal_level to 'logical'. My goal is to allow
> users who are using 'replica' level to use logical decoding without a
> server restart. There are other GUC parameters related to logical
> decoding and logical replication such as max_wal_senders,
> max_logical_replication_workers, and max_replication_slots, but even
> if users set these parameters >0, there would not be a noticeable
> performance impact. And their default values are already >0. So I'd
> like to focus on making only the wal_level dynamic GUC parameter.
> There are several earlier discussions[2][3] but no one has submitted
> patches unless I'm missing something.
>
> The first idea I came up with is to make the wal_level a PGC_SIGHUP
> parameter. However, it affects not only setting 'replica' to 'logical'
> but also setting 'minimal' to 'replica' or higher. I'm not sure the
> latter case is common and it might require a checkpoint. I don't want
> to make the patch complex for uncommon cases.
>
> The second idea is to somehow allow both WAL-logging logical info and
> logical decoding even when wal_level is 'replica'. I've attached a PoC
> patch for that. The patch introduces new SQL functions such as
> pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> These functions are available only when wal_level is 'repilca'(or
> higher). In pg_activate_logical_decoding(), we set the status of
> logical decoding stored on the shared memory from 'disabled' to
> 'xlog-logical-info', allowing all processes to write logical
> information to WAL records for logical decoding. But the logical
> decoding is still not allowed. Once we confirm all in-progress
> transactions completed, we switch the status to
> 'logical-decoding-ready', meaning that users can create logical
> replication slots and use logical decoding.

I felt that the only disadvantage with this approach is that we
currently wait for all in-progress transactions to complete before
enabling logical decoding. If a long-running transaction exists and
the session enabling logical decoding fails—due to factors like
statement_timeout, transaction_timeout,
idle_in_transaction_session_timeout, idle_session_timeout, or any
other failure. This would require restarting the wait. During this
time, there's a risk that a new long-running transaction could start,
further delaying the process. Probably this could be solved if the
waiting is done from any of the background processes through
PGC_SIGHUP. While this type of failure is likely rare, I’m unsure
whether we should consider this scenario.

Thoughts?

Regards,
Vignesh

On Mon, Jan 13, 2025 at 2:52 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
>
> I felt that the only disadvantage with this approach is that we
> currently wait for all in-progress transactions to complete before
> enabling logical decoding. If a long-running transaction exists and
> the session enabling logical decoding fails—due to factors like
> statement_timeout, transaction_timeout,
> idle_in_transaction_session_timeout, idle_session_timeout, or any
> other failure. This would require restarting the wait. During this
> time, there's a risk that a new long-running transaction could start,
> further delaying the process. Probably this could be solved if the
> waiting is done from any of the background processes through
> PGC_SIGHUP. While this type of failure is likely rare, I’m unsure
> whether we should consider this scenario.

A related question: While the operation is waiting for already running
transactions to end, the backends whose transactions have finished may
start new transactions. When we switch WAL from 'replica' to
'logical', there may be some transactions running. Will this lead to
WAL stream with mixes WAL records - some with logical information and
some without? Do we need to adjust logical decoding to tackle this
situation? Is there a chance that some WAL from same transaction have
logical information and some do not?

--
Best Wishes,
Ashutosh Bapat

On Mon, Jan 13, 2025 at 1:22 AM vignesh C <vignesh21(at)gmail(dot)com> wrote:
>
> On Tue, 31 Dec 2024 at 10:15, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > Hi all,
> >
> > Logical decoding (and logical replication) are available only when
> > wal_level = logical. As the documentation says[1], Using the 'logical'
> > level increases the WAL volume which could negatively affect the
> > performance. For that reason, users might want to start with using
> > 'replica', but when they want to use logical decoding they need a
> > server restart to increase wal_level to 'logical'. My goal is to allow
> > users who are using 'replica' level to use logical decoding without a
> > server restart. There are other GUC parameters related to logical
> > decoding and logical replication such as max_wal_senders,
> > max_logical_replication_workers, and max_replication_slots, but even
> > if users set these parameters >0, there would not be a noticeable
> > performance impact. And their default values are already >0. So I'd
> > like to focus on making only the wal_level dynamic GUC parameter.
> > There are several earlier discussions[2][3] but no one has submitted
> > patches unless I'm missing something.
> >
> > The first idea I came up with is to make the wal_level a PGC_SIGHUP
> > parameter. However, it affects not only setting 'replica' to 'logical'
> > but also setting 'minimal' to 'replica' or higher. I'm not sure the
> > latter case is common and it might require a checkpoint. I don't want
> > to make the patch complex for uncommon cases.
> >
> > The second idea is to somehow allow both WAL-logging logical info and
> > logical decoding even when wal_level is 'replica'. I've attached a PoC
> > patch for that. The patch introduces new SQL functions such as
> > pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> > These functions are available only when wal_level is 'repilca'(or
> > higher). In pg_activate_logical_decoding(), we set the status of
> > logical decoding stored on the shared memory from 'disabled' to
> > 'xlog-logical-info', allowing all processes to write logical
> > information to WAL records for logical decoding. But the logical
> > decoding is still not allowed. Once we confirm all in-progress
> > transactions completed, we switch the status to
> > 'logical-decoding-ready', meaning that users can create logical
> > replication slots and use logical decoding.
>
> I felt that the only disadvantage with this approach is that we
> currently wait for all in-progress transactions to complete before
> enabling logical decoding. If a long-running transaction exists and
> the session enabling logical decoding fails—due to factors like
> statement_timeout, transaction_timeout,
> idle_in_transaction_session_timeout, idle_session_timeout, or any
> other failure. This would require restarting the wait. During this
> time, there's a risk that a new long-running transaction could start,
> further delaying the process. Probably this could be solved if the
> waiting is done from any of the background processes through
> PGC_SIGHUP. While this type of failure is likely rare, I’m unsure
> whether we should consider this scenario.
>
> Thoughts?

Yeah, delegating the activation to the background process such as the
checkpointer would also be one solution. This would work with the
approach that we enable the logical decoding via
pg_activate_logical_decoding(). On the other hand, if we support
automatically enabling the logical decoding (and logical info logging)
when the first logical slot is created, we might want to have the
process who is creating the logical slot activate the logical decoding
too.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Mon, Jan 13, 2025 at 1:31 AM Ashutosh Bapat
<ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
>
> On Mon, Jan 13, 2025 at 2:52 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
> >
> > I felt that the only disadvantage with this approach is that we
> > currently wait for all in-progress transactions to complete before
> > enabling logical decoding. If a long-running transaction exists and
> > the session enabling logical decoding fails—due to factors like
> > statement_timeout, transaction_timeout,
> > idle_in_transaction_session_timeout, idle_session_timeout, or any
> > other failure. This would require restarting the wait. During this
> > time, there's a risk that a new long-running transaction could start,
> > further delaying the process. Probably this could be solved if the
> > waiting is done from any of the background processes through
> > PGC_SIGHUP. While this type of failure is likely rare, I’m unsure
> > whether we should consider this scenario.
>
> A related question: While the operation is waiting for already running
> transactions to end, the backends whose transactions have finished may
> start new transactions. When we switch WAL from 'replica' to
> 'logical', there may be some transactions running. Will this lead to
> WAL stream with mixes WAL records - some with logical information and
> some without?

Yes. There could be mixed WAL records until all running transactions complete.

> Do we need to adjust logical decoding to tackle this
> situation? Is there a chance that some WAL from same transaction have
> logical information and some do not?

In the current approach, we first enable logical info WAL-logging to
let newly started transactions do logical info WAL-logging, then allow
the logical decoding only after all running transactions complete.
Therefore, at the time when we allow the logical decoding, all WAL
records are written with logical information.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, 14 Jan 2025 at 03:03, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> Yeah, delegating the activation to the background process such as the
> checkpointer would also be one solution. This would work with the
> approach that we enable the logical decoding via
> pg_activate_logical_decoding(). On the other hand, if we support
> automatically enabling the logical decoding (and logical info logging)
> when the first logical slot is created, we might want to have the
> process who is creating the logical slot activate the logical decoding
> too.

One possible approach is to delegate the task of updating the wal
level to include logical decoding information during the creation of
the first logical replication slot also to a background process (e.g.,
the checkpoint process) and wait until the wal level has been updated
and reaches ready state. Additionally, we could introduce a new
optional parameter during slot creation to specify whether we should
roll back changes related to updating the WAL level in case of
failure. If a rollback is needed, we could invoke the
logical_decoding_activation_abort_callback. By default, the rollback
could be set to false, since in most failure scenarios, the user will
likely attempt to create the slot again immediately. By then, the WAL
level may have already been updated, eliminating the need to wait for
any new long-running transactions.
An alternative approach could be to skip the rollback of logical info
logging during logical replication slot creation in case of failure,
without introducing a rollback parameter. This is because there may be
concurrent transactions (e.g., parallel slot creation or
pg_activate_logical_decoding from different sessions) that would also
fail. In such cases, the user could be allowed to call
pg_deactivate_logical_decoding if needed.

Regards,
Vignesh

On Tue, Jan 14, 2025 at 3:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Mon, Jan 13, 2025 at 1:31 AM Ashutosh Bapat
> <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> >
> > On Mon, Jan 13, 2025 at 2:52 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
> > >
> > > I felt that the only disadvantage with this approach is that we
> > > currently wait for all in-progress transactions to complete before
> > > enabling logical decoding. If a long-running transaction exists and
> > > the session enabling logical decoding fails—due to factors like
> > > statement_timeout, transaction_timeout,
> > > idle_in_transaction_session_timeout, idle_session_timeout, or any
> > > other failure. This would require restarting the wait. During this
> > > time, there's a risk that a new long-running transaction could start,
> > > further delaying the process. Probably this could be solved if the
> > > waiting is done from any of the background processes through
> > > PGC_SIGHUP. While this type of failure is likely rare, I’m unsure
> > > whether we should consider this scenario.
> >
> > A related question: While the operation is waiting for already running
> > transactions to end, the backends whose transactions have finished may
> > start new transactions. When we switch WAL from 'replica' to
> > 'logical', there may be some transactions running. Will this lead to
> > WAL stream with mixes WAL records - some with logical information and
> > some without?
>
> Yes. There could be mixed WAL records until all running transactions complete.
>
> > Do we need to adjust logical decoding to tackle this
> > situation? Is there a chance that some WAL from same transaction have
> > logical information and some do not?
>
> In the current approach, we first enable logical info WAL-logging to
> let newly started transactions do logical info WAL-logging, then allow
> the logical decoding only after all running transactions complete.
> Therefore, at the time when we allow the logical decoding, all WAL
> records are written with logical information.

Thanks for the clarification. Let's consider that T1, T2, T3 were
running when the request to enable logical decoding came in. Let's say
T1 finished first, then T2 and then T3. But in-the mean time T4, T5
and T6 were started, which logged WAL with logical information. While
T2, T3, T4, T5, T6 all are running simultaneously, there will be mixed
WAL logs (some with logical information and some without). When T3
ends, logical decoding will be allowed. If a logical decoding process
is started at this point, it will start reading WAL from the earliest
of T4, T5 or T6 and hence it will encounter WAL written by T2 or T3
which do not have logical information. That would create a problem.
Thus in order for the logical process to start The transactions T4, T5
and T6 need to be finished as well. Am I correct? If that is so, my
earlier proposal to combine the step to wait for running transactions
to complete for both logical decoding as well as replication slot
creation would not be possible.

Will the mixed logs create a problem for an ongoing physical stream replication?

--
Best Wishes,
Ashutosh Bapat

On Sun, Jan 12, 2025 at 10:52 PM Peter Smith <smithpb2250(at)gmail(dot)com> wrote:
>
> On Fri, Jan 10, 2025 at 8:28 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > Hi,
> >
> > On Tue, Jan 7, 2025 at 11:30 PM Peter Smith <smithpb2250(at)gmail(dot)com> wrote:
> > >
> > > Hi Sawada-San.
> > >
> > > FWIW, I also thought it was a good idea suggested by Bertrand [1] to
> > > "hide" everything behind the slot create/delete, and thereby eliminate
> > > the need for user intervention using those new
> > > pg_activate/deactivate_logical_decoding functions.
> > >
> > > But, one concern doing it this way is how to prevent a user
> > > (accidentally?) getting themselves into a different replication mode
> > > without realising it? Or say they change the mode and then "forget"
> > > that they did that.
> >
> > I think that as long as there is at least one logical replication slot
> > users still have the will to execute logical decoding, so they require
> > logical info WAL-logging. I think it would rather be good for users to
> > be able to forget about that.
> >
> > > For example, If wal_level=replica and then somebody does CREATE
> > > PUBLICATION/SUBSCRIPTION won't that implicitly enable the logical
> > > decoding and then leave it enabled until all the logical replication
> > > slots are eventually dropped?
> >
> > Yes.
> >
> > > Now, when the user examines wal_level it
> > > is still going to show 'replica', which could be misleading --- e.g.
> > > how will they even know that they can't really trust that anymore, and
> > > they must also check the pg_get_logical_decoding_status?
> >
> > I think that if we automatically enable logical decoding even when
> > wal_level=replica, we would not need pg_get_logical_decoding_status().
> > These changes would break compatibility tools checking if wal_level is
> > 'logical', but I guess that tools will not need to check that anymore.
> > Users can simply understand that if there is at least one logical
> > replication slot the system writes necessary information to use it.
> > Having said that we anyway need to mention it in the doc and raising
> > NOTICE/WARNING would also be a good idea as you mentioned.
> >
>
> Hi Sawada-San
>
> IMO it seems misleading to be having logical replication behaviour
> while the wal_level still displays as 'replica'.
>
> So, I was wondering if it would be worthwhile to introduce a
> 'show_hook' for this GUC.
>
> I hacked a quick implementation (attached) which now displays a middle
> value for this scenario as "replica-logical". See below:
>
> test_pub=# show wal_level;
> wal_level
> -----------
> replica
> (1 row)
>
> test_pub=# select pg_get_logical_decoding_status();
> pg_get_logical_decoding_status
> --------------------------------
> disabled
> (1 row)
>
> test_pub=# select pg_activate_logical_decoding();
> pg_activate_logical_decoding
> ------------------------------
>
> (1 row)
>
> test_pub=# show wal_level;
> wal_level
> -----------------
> replica-logical
> (1 row)

I think this would break the compatibility for monitoring tools
anyway. Given that the logical decoding is enabled if there is at
least one logical slot, they just need to check the number of existing
logical slots to know whether logical info WAL-logging is enabled or
not.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, Jan 14, 2025 at 9:46 PM Ashutosh Bapat
<ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
>
> On Tue, Jan 14, 2025 at 3:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Mon, Jan 13, 2025 at 1:31 AM Ashutosh Bapat
> > <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> > >
> > > On Mon, Jan 13, 2025 at 2:52 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
> > > >
> > > > I felt that the only disadvantage with this approach is that we
> > > > currently wait for all in-progress transactions to complete before
> > > > enabling logical decoding. If a long-running transaction exists and
> > > > the session enabling logical decoding fails—due to factors like
> > > > statement_timeout, transaction_timeout,
> > > > idle_in_transaction_session_timeout, idle_session_timeout, or any
> > > > other failure. This would require restarting the wait. During this
> > > > time, there's a risk that a new long-running transaction could start,
> > > > further delaying the process. Probably this could be solved if the
> > > > waiting is done from any of the background processes through
> > > > PGC_SIGHUP. While this type of failure is likely rare, I’m unsure
> > > > whether we should consider this scenario.
> > >
> > > A related question: While the operation is waiting for already running
> > > transactions to end, the backends whose transactions have finished may
> > > start new transactions. When we switch WAL from 'replica' to
> > > 'logical', there may be some transactions running. Will this lead to
> > > WAL stream with mixes WAL records - some with logical information and
> > > some without?
> >
> > Yes. There could be mixed WAL records until all running transactions complete.
> >
> > > Do we need to adjust logical decoding to tackle this
> > > situation? Is there a chance that some WAL from same transaction have
> > > logical information and some do not?
> >
> > In the current approach, we first enable logical info WAL-logging to
> > let newly started transactions do logical info WAL-logging, then allow
> > the logical decoding only after all running transactions complete.
> > Therefore, at the time when we allow the logical decoding, all WAL
> > records are written with logical information.
>
> Thanks for the clarification. Let's consider that T1, T2, T3 were
> running when the request to enable logical decoding came in. Let's say
> T1 finished first, then T2 and then T3. But in-the mean time T4, T5
> and T6 were started, which logged WAL with logical information. While
> T2, T3, T4, T5, T6 all are running simultaneously, there will be mixed
> WAL logs (some with logical information and some without). When T3
> ends, logical decoding will be allowed. If a logical decoding process
> is started at this point, it will start reading WAL from the earliest
> of T4, T5 or T6 and hence it will encounter WAL written by T2 or T3
> which do not have logical information.

IIUC we should prevent a logical slot from reserving the WAL until T1,
T2, and T3 finishes. That is, we wait for T1, T2, and T3 to finish
before calling ReplicationSlotReserveWal() where we decide the start
point for reading WAL. That way, the logical decoding will start
reading WAL from the point after the latest commit record of T1, T2,
and T3 (i.e. T3 in this case), meaning that there is no mixed WAL
after this point and T4, T5, and T6 would not be included in the
logical decoding.

> Will the mixed logs create a problem for an ongoing physical stream replication?

I don't see any problem for physical stream replication in terms of
mixed logs. We write a WAL record to enable the logical decoding after
the point where we can ensure there are no mixed WAL records. So the
logical decoding on standbys also can start reading WAL records from
that point.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Jan 16, 2025 at 4:28 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Sun, Jan 12, 2025 at 10:52 PM Peter Smith <smithpb2250(at)gmail(dot)com> wrote:
> >
> > On Fri, Jan 10, 2025 at 8:28 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > Hi,
> > >
> > > On Tue, Jan 7, 2025 at 11:30 PM Peter Smith <smithpb2250(at)gmail(dot)com> wrote:
> > > >
> > > > Hi Sawada-San.
> > > >
> > > > FWIW, I also thought it was a good idea suggested by Bertrand [1] to
> > > > "hide" everything behind the slot create/delete, and thereby eliminate
> > > > the need for user intervention using those new
> > > > pg_activate/deactivate_logical_decoding functions.
> > > >
> > > > But, one concern doing it this way is how to prevent a user
> > > > (accidentally?) getting themselves into a different replication mode
> > > > without realising it? Or say they change the mode and then "forget"
> > > > that they did that.
> > >
> > > I think that as long as there is at least one logical replication slot
> > > users still have the will to execute logical decoding, so they require
> > > logical info WAL-logging. I think it would rather be good for users to
> > > be able to forget about that.
> > >
> > > > For example, If wal_level=replica and then somebody does CREATE
> > > > PUBLICATION/SUBSCRIPTION won't that implicitly enable the logical
> > > > decoding and then leave it enabled until all the logical replication
> > > > slots are eventually dropped?
> > >
> > > Yes.
> > >
> > > > Now, when the user examines wal_level it
> > > > is still going to show 'replica', which could be misleading --- e.g.
> > > > how will they even know that they can't really trust that anymore, and
> > > > they must also check the pg_get_logical_decoding_status?
> > >
> > > I think that if we automatically enable logical decoding even when
> > > wal_level=replica, we would not need pg_get_logical_decoding_status().
> > > These changes would break compatibility tools checking if wal_level is
> > > 'logical', but I guess that tools will not need to check that anymore.
> > > Users can simply understand that if there is at least one logical
> > > replication slot the system writes necessary information to use it.
> > > Having said that we anyway need to mention it in the doc and raising
> > > NOTICE/WARNING would also be a good idea as you mentioned.
> > >
> >
> > Hi Sawada-San
> >
> > IMO it seems misleading to be having logical replication behaviour
> > while the wal_level still displays as 'replica'.
> >
> > So, I was wondering if it would be worthwhile to introduce a
> > 'show_hook' for this GUC.
> >
> > I hacked a quick implementation (attached) which now displays a middle
> > value for this scenario as "replica-logical". See below:
> >
> > test_pub=# show wal_level;
> > wal_level
> > -----------
> > replica
> > (1 row)
> >
> > test_pub=# select pg_get_logical_decoding_status();
> > pg_get_logical_decoding_status
> > --------------------------------
> > disabled
> > (1 row)
> >
> > test_pub=# select pg_activate_logical_decoding();
> > pg_activate_logical_decoding
> > ------------------------------
> >
> > (1 row)
> >
> > test_pub=# show wal_level;
> > wal_level
> > -----------------
> > replica-logical
> > (1 row)
>
> I think this would break the compatibility for monitoring tools
> anyway. Given that the logical decoding is enabled if there is at
> least one logical slot, they just need to check the number of existing
> logical slots to know whether logical info WAL-logging is enabled or
> not.
>

But, that "replica-logical" string value was just for my demonstration.

From your reply, I cannot tell if you accidentally overlooked the next
part where I said:
OTOH, you might prefer to display the "replica-logical" case as "logical".

~~~

e.g. then the above example would look like this:

test_pub=# show wal_level;
wal_level
-----------
replica
(1 row)

test_pub=# select pg_get_logical_decoding_status();
pg_get_logical_decoding_status
--------------------------------
disabled
(1 row)

test_pub=# select pg_activate_logical_decoding();
pg_activate_logical_decoding
------------------------------
(1 row)

test_pub=# show wal_level;
wal_level
-----------
logical
(1 row)

Eventually, if the number of logical replication slots falls to 0, the
wal_level value displayed reverts to "replica".

Since there are no *new* wal_level strings introduced how would that
break compatibility for monitoring tools?

And, since the (fudged) wal_level value "logical" already indicates
whether logical info WAL-logging is enabled or not, no additional
kinds of checking are needed.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi,

On Mon, Jan 06, 2025 at 10:32:52PM -0800, Masahiko Sawada wrote:
> On Mon, Jan 6, 2025 at 3:20 AM Ashutosh Bapat
> <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> >
> > On Sat, Jan 4, 2025 at 6:03 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Fri, Jan 3, 2025 at 6:31 AM Euler Taveira <euler(at)eulerto(dot)com> wrote:
> > > >
> > > > On Fri, Jan 3, 2025, at 10:14 AM, Bertrand Drouvot wrote:
> > > >
> > > > If we don't want to force wal_level = logical to enable logical decoding (your
> > > > second idea) then I think that it would be better to "hide" everything behind
> > > > logical replication slot creation / deletion. That would mean that having at
> > > > least one logical replication slot created would be synonym to "activate logical
> > > > decoding" and zero logical replication slot created would be synonym to "deactivate
> > > > logical decoding".
> > > >
> > > >
> > > > I like this idea. The logical replication slot existence already has the
> > > > required protections and guarantees (no running transactions from the past while
> > > > creating) for logical decoding.
> > >
> > > I agree that it's better behavior.
> > >
> > > >
> > > > Having said that, you are basically folding 'logical' machinery into 'replica'.
> > > > The 'logical' option can still exists but it will be less attractive because it
> > > > increases the WAL volume even if you are not using logical replication. I don't
> > > > know if the current 'logical' behavior (WAL records for logical decoding even
> > > > if there is no active logical replication) has any use case (maybe someone
> > > > inspects these extra records for analysis) but one suggestion (separate patch)
> > > > is to make 'logical' synonymous with the new 'replica' behavior (logical
> > > > decoding capability). This opens the door to remove 'logical' in future
> > > > releases (accepted as synonym but not documented).
>
> FYI one possible use case of the 'logical' level would be that if we
> support retrieving WAL records for logical decoding from somewhere
> like restore_command in the future, users would like to keep the
> wal_level 'logical' or keep the logical decoding active.

Another use case is to allow logical decoding from standby. One could set
wal_level to logical on the primary to be able to create a logical replication
slot on the standby (without having the need to create one on the primary).

BTW, currently it's mandatory to set wal_level to logical on the primary to
be able to create a logical replication slot on the standby. Should we extend
this to "logical decoding is enabled on the primary" too?

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Wed, Jan 22, 2025 at 2:04 AM Bertrand Drouvot
<bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Mon, Jan 06, 2025 at 10:32:52PM -0800, Masahiko Sawada wrote:
> > On Mon, Jan 6, 2025 at 3:20 AM Ashutosh Bapat
> > <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> > >
> > > On Sat, Jan 4, 2025 at 6:03 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > On Fri, Jan 3, 2025 at 6:31 AM Euler Taveira <euler(at)eulerto(dot)com> wrote:
> > > > >
> > > > > On Fri, Jan 3, 2025, at 10:14 AM, Bertrand Drouvot wrote:
> > > > >
> > > > > If we don't want to force wal_level = logical to enable logical decoding (your
> > > > > second idea) then I think that it would be better to "hide" everything behind
> > > > > logical replication slot creation / deletion. That would mean that having at
> > > > > least one logical replication slot created would be synonym to "activate logical
> > > > > decoding" and zero logical replication slot created would be synonym to "deactivate
> > > > > logical decoding".
> > > > >
> > > > >
> > > > > I like this idea. The logical replication slot existence already has the
> > > > > required protections and guarantees (no running transactions from the past while
> > > > > creating) for logical decoding.
> > > >
> > > > I agree that it's better behavior.
> > > >
> > > > >
> > > > > Having said that, you are basically folding 'logical' machinery into 'replica'.
> > > > > The 'logical' option can still exists but it will be less attractive because it
> > > > > increases the WAL volume even if you are not using logical replication. I don't
> > > > > know if the current 'logical' behavior (WAL records for logical decoding even
> > > > > if there is no active logical replication) has any use case (maybe someone
> > > > > inspects these extra records for analysis) but one suggestion (separate patch)
> > > > > is to make 'logical' synonymous with the new 'replica' behavior (logical
> > > > > decoding capability). This opens the door to remove 'logical' in future
> > > > > releases (accepted as synonym but not documented).
> >
> > FYI one possible use case of the 'logical' level would be that if we
> > support retrieving WAL records for logical decoding from somewhere
> > like restore_command in the future, users would like to keep the
> > wal_level 'logical' or keep the logical decoding active.
>
> Another use case is to allow logical decoding from standby. One could set
> wal_level to logical on the primary to be able to create a logical replication
> slot on the standby (without having the need to create one on the primary).

Good point. It seems to be a substantial disadvantage for users.

>
> BTW, currently it's mandatory to set wal_level to logical on the primary to
> be able to create a logical replication slot on the standby. Should we extend
> this to "logical decoding is enabled on the primary" too?

Yes, in order to use the logical decoding on standbys, we need to
enable it on the primary. When enabling the logical decoding on the
primary we can write a WAL record so that standby servers can enable
the logical decoding too.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Fri, Jan 10, 2025 at 12:33 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Thu, Jan 9, 2025 at 3:29 AM Ashutosh Bapat
> <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> >
> > On Tue, Dec 31, 2024 at 10:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > Hi all,
> > >
> > > Logical decoding (and logical replication) are available only when
> > > wal_level = logical. As the documentation says[1], Using the 'logical'
> > > level increases the WAL volume which could negatively affect the
> > > performance. For that reason, users might want to start with using
> > > 'replica', but when they want to use logical decoding they need a
> > > server restart to increase wal_level to 'logical'. My goal is to allow
> > > users who are using 'replica' level to use logical decoding without a
> > > server restart. There are other GUC parameters related to logical
> > > decoding and logical replication such as max_wal_senders,
> > > max_logical_replication_workers, and max_replication_slots, but even
> > > if users set these parameters >0, there would not be a noticeable
> > > performance impact. And their default values are already >0. So I'd
> > > like to focus on making only the wal_level dynamic GUC parameter.
> > > There are several earlier discussions[2][3] but no one has submitted
> > > patches unless I'm missing something.
> > >
> > > The first idea I came up with is to make the wal_level a PGC_SIGHUP
> > > parameter. However, it affects not only setting 'replica' to 'logical'
> > > but also setting 'minimal' to 'replica' or higher. I'm not sure the
> > > latter case is common and it might require a checkpoint. I don't want
> > > to make the patch complex for uncommon cases.
> > >
> > > The second idea is to somehow allow both WAL-logging logical info and
> > > logical decoding even when wal_level is 'replica'. I've attached a PoC
> > > patch for that. The patch introduces new SQL functions such as
> > > pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> > > These functions are available only when wal_level is 'repilca'(or
> > > higher). In pg_activate_logical_decoding(), we set the status of
> > > logical decoding stored on the shared memory from 'disabled' to
> > > 'xlog-logical-info', allowing all processes to write logical
> > > information to WAL records for logical decoding. But the logical
> > > decoding is still not allowed. Once we confirm all in-progress
> > > transactions completed, we switch the status to
> > > 'logical-decoding-ready', meaning that users can create logical
> > > replication slots and use logical decoding.
> > >
> > > Overall, with the patch, there are two ways to enable logical
> > > decoding: setting wal_level to 'logical' and calling
> > > pg_activate_logical_decoding() when wal_level is 'replica'. I left the
> > > 'logical' level for backward compatibility and for users who want to
> > > enable the logical decoding without calling that SQL function. If we
> > > can automatically enable the logical decoding when creating the first
> > > logical replication slot, probably we no longer need the 'logical'
> > > level. There is room to discuss the user interface. Feedback is very
> > > welcome.
> > >
> >
> > If a server is running at minimal wal_level and they want to enable
> > logical replication, they would still need a server restart. That
> > would be rare but not completely absent.
>
> Currently we don't allow the server to start with the 'minimal' level
> and max_wal_senders > 0. Even if we support changing 'minimal' to
> 'logical' without a server restart, we still need a server restart to
> increase max_wal_senders for users who want to use logical
> replication. Or we need to eliminate this restriction too. I guess it
> would be too complex for such uncommon use cases.
>
> >
> > Our documentation says "wal_level determines how much information is
> > written to the WAL.". Users would may not expect that the WAL amount
> > changes while wal_level = replica depending upon whether logical
> > decoding is possible. It may be possible to set the expectations right
> > by changing the documentation. It's not in the patch, so I am not sure
> > whether this is considered.
>
> We should mention that in the doc. The WAL amount changes depending on
> not only wal_level but also other parameters such as wal_log_hints and
> full_page_writes.
>
> > Cloud providers do not like multiple ways of changing configuration
> > esp. when they can not control it. See [1]. Changing wal_level through
> > a SQL function may fit the same category.
>
> Thank you for pointing it out. This would support the idea of
> automatically enabling logical decoding.
>
> > I agree that it would be a lot of work to make all combinations of
> > wal_level changes work, but changing wal_level through SIGHUP looks
> > like a cleaner solution. Is there way that we make the GUC SIGHUP but
> > disallow certain combinations of old and new values?
>
> While I agree that it's cleaner I think there is no way today. I think
> we need to invent something for that.

I would like to summarize the proposed approaches thus far:

Regarding the user interface, there are three approaches:

1. Implementing SQL function controls (e.g.,
pg_activate_logical_decoding() and pg_deactivate_logical_decoding()).
This would enable users to activate logical decoding even with
wal_level=replica by calling the SQL function. While cloud providers
seem not like having multiple configuration methods, this could
potentially be managed through appropriate EXECUTE privileges. Another
drawback is the user confusion when 'SHOW wal_level' displays
'replica' despite processes writing WAL records with logical
information. This might be dealt with by implementing a show_hook
function for wal_level.

2. Implementing automatic logical decoding activation. This would
trigger upon creation of the first logical slot and deactivate upon
removal of the final slot. This approach shares the user confusion
concern of the first proposal. Moreover, it presents a significant
limitation: users would be unable to utilize logical decoding on
standby servers without maintaining at least one logical slot on the
primary -- a substantial disadvantage.

3. Converting wal_level to a SIGHUP parameter, thereby supporting all
possible wal_level transition combinations. While this represents the
most elegant solution among the proposals, it necessitates additional
development effort for less common scenarios, such as transitioning
between 'minimal' and 'replica' levels. Such transitions require
specific handling -- for instance, changing between 'minimal' and
'replica' requires a checkpoint, while decreasing from 'replica' to
'minimal' necessitates terminating certain processes like WAL senders
and archiver.

We also had discussion (and I did some research) on the implementation
of increasing/decreasing wal_level online. The basic idea is that we
first enable logical information WAL-logging to all processes while
maintaining the logical decoding in an inactive state. Once we can
guarantee that all processes are writing WAL records with logical
information, we enable the logical decoding. This guarantee can be
achieved by waiting for all concurrent transactions to finish, which
could make us wait for a long time if a transaction is long-running.
Another way is to send a global barrier signal and wait for all
processes to start writing WAL records with logical information. We
have a good facility for that: EmitProcSignalBarrier() and
WaitForProcSignalBarrier(). That way, we don't need to wait for
transaction finishes.

Based on the discussion so far, the idea 3 appears most promising. I
welcome any additional suggestions or preferences.

BTW I'm writing a PoC patch for the idea 3 and using global barriers,
and will share the patch.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On Wed, Jan 22, 2025 at 04:46:00PM -0800, Masahiko Sawada wrote:
> I would like to summarize the proposed approaches thus far:

Thanks!

> Regarding the user interface, there are three approaches:
>
> 1. Implementing SQL function controls (e.g.,
> pg_activate_logical_decoding() and pg_deactivate_logical_decoding()).
> This would enable users to activate logical decoding even with
> wal_level=replica by calling the SQL function. While cloud providers
> seem not like having multiple configuration methods, this could
> potentially be managed through appropriate EXECUTE privileges. Another
> drawback is the user confusion when 'SHOW wal_level' displays
> 'replica' despite processes writing WAL records with logical
> information. This might be dealt with by implementing a show_hook
> function for wal_level.
>
> 2. Implementing automatic logical decoding activation. This would
> trigger upon creation of the first logical slot and deactivate upon
> removal of the final slot. This approach shares the user confusion
> concern of the first proposal. Moreover, it presents a significant
> limitation: users would be unable to utilize logical decoding on
> standby servers without maintaining at least one logical slot on the
> primary -- a substantial disadvantage.

Yeah, unless we keep wal_level around but I agree that the following (3.) looks
like the way to go (as it removes any confusion).

> 3. Converting wal_level to a SIGHUP parameter, thereby supporting all
> possible wal_level transition combinations. While this represents the
> most elegant solution among the proposals,

+1

> it necessitates additional
> development effort for less common scenarios, such as transitioning
> between 'minimal' and 'replica' levels. Such transitions require
> specific handling -- for instance, changing between 'minimal' and
> 'replica' requires a checkpoint, while decreasing from 'replica' to
> 'minimal' necessitates terminating certain processes like WAL senders
> and archiver.

Yeah. OTOH switching from replica to minimal is "dangerous" as it makes
previous base backups unusable for point-in-time recovery. So I wonder if it
wouldn't be better to keep a restart mandatory depending of the transition
state (that would probably make users thinking "twice" before doing the
transition that requires a restart). I don't think any GUC does that already but
that might be something to explore, thoughts?

> We also had discussion (and I did some research) on the implementation
> of increasing/decreasing wal_level online. The basic idea is that we
> first enable logical information WAL-logging to all processes while
> maintaining the logical decoding in an inactive state. Once we can
> guarantee that all processes are writing WAL records with logical
> information, we enable the logical decoding. This guarantee can be
> achieved by waiting for all concurrent transactions to finish, which
> could make us wait for a long time if a transaction is long-running.
> Another way is to send a global barrier signal and wait for all
> processes to start writing WAL records with logical information. We
> have a good facility for that: EmitProcSignalBarrier() and
> WaitForProcSignalBarrier(). That way, we don't need to wait for
> transaction finishes.

That sounds like a plan.

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Thu, Jan 23, 2025 at 6:16 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Fri, Jan 10, 2025 at 12:33 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Thu, Jan 9, 2025 at 3:29 AM Ashutosh Bapat
> > <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> > >
> > > On Tue, Dec 31, 2024 at 10:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Logical decoding (and logical replication) are available only when
> > > > wal_level = logical. As the documentation says[1], Using the 'logical'
> > > > level increases the WAL volume which could negatively affect the
> > > > performance. For that reason, users might want to start with using
> > > > 'replica', but when they want to use logical decoding they need a
> > > > server restart to increase wal_level to 'logical'. My goal is to allow
> > > > users who are using 'replica' level to use logical decoding without a
> > > > server restart. There are other GUC parameters related to logical
> > > > decoding and logical replication such as max_wal_senders,
> > > > max_logical_replication_workers, and max_replication_slots, but even
> > > > if users set these parameters >0, there would not be a noticeable
> > > > performance impact. And their default values are already >0. So I'd
> > > > like to focus on making only the wal_level dynamic GUC parameter.
> > > > There are several earlier discussions[2][3] but no one has submitted
> > > > patches unless I'm missing something.
> > > >
> > > > The first idea I came up with is to make the wal_level a PGC_SIGHUP
> > > > parameter. However, it affects not only setting 'replica' to 'logical'
> > > > but also setting 'minimal' to 'replica' or higher. I'm not sure the
> > > > latter case is common and it might require a checkpoint. I don't want
> > > > to make the patch complex for uncommon cases.
> > > >
> > > > The second idea is to somehow allow both WAL-logging logical info and
> > > > logical decoding even when wal_level is 'replica'. I've attached a PoC
> > > > patch for that. The patch introduces new SQL functions such as
> > > > pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> > > > These functions are available only when wal_level is 'repilca'(or
> > > > higher). In pg_activate_logical_decoding(), we set the status of
> > > > logical decoding stored on the shared memory from 'disabled' to
> > > > 'xlog-logical-info', allowing all processes to write logical
> > > > information to WAL records for logical decoding. But the logical
> > > > decoding is still not allowed. Once we confirm all in-progress
> > > > transactions completed, we switch the status to
> > > > 'logical-decoding-ready', meaning that users can create logical
> > > > replication slots and use logical decoding.
> > > >
> > > > Overall, with the patch, there are two ways to enable logical
> > > > decoding: setting wal_level to 'logical' and calling
> > > > pg_activate_logical_decoding() when wal_level is 'replica'. I left the
> > > > 'logical' level for backward compatibility and for users who want to
> > > > enable the logical decoding without calling that SQL function. If we
> > > > can automatically enable the logical decoding when creating the first
> > > > logical replication slot, probably we no longer need the 'logical'
> > > > level. There is room to discuss the user interface. Feedback is very
> > > > welcome.
> > > >
> > >
> > > If a server is running at minimal wal_level and they want to enable
> > > logical replication, they would still need a server restart. That
> > > would be rare but not completely absent.
> >
> > Currently we don't allow the server to start with the 'minimal' level
> > and max_wal_senders > 0. Even if we support changing 'minimal' to
> > 'logical' without a server restart, we still need a server restart to
> > increase max_wal_senders for users who want to use logical
> > replication. Or we need to eliminate this restriction too. I guess it
> > would be too complex for such uncommon use cases.
> >
> > >
> > > Our documentation says "wal_level determines how much information is
> > > written to the WAL.". Users would may not expect that the WAL amount
> > > changes while wal_level = replica depending upon whether logical
> > > decoding is possible. It may be possible to set the expectations right
> > > by changing the documentation. It's not in the patch, so I am not sure
> > > whether this is considered.
> >
> > We should mention that in the doc. The WAL amount changes depending on
> > not only wal_level but also other parameters such as wal_log_hints and
> > full_page_writes.
> >
> > > Cloud providers do not like multiple ways of changing configuration
> > > esp. when they can not control it. See [1]. Changing wal_level through
> > > a SQL function may fit the same category.
> >
> > Thank you for pointing it out. This would support the idea of
> > automatically enabling logical decoding.
> >
> > > I agree that it would be a lot of work to make all combinations of
> > > wal_level changes work, but changing wal_level through SIGHUP looks
> > > like a cleaner solution. Is there way that we make the GUC SIGHUP but
> > > disallow certain combinations of old and new values?
> >
> > While I agree that it's cleaner I think there is no way today. I think
> > we need to invent something for that.
>
> I would like to summarize the proposed approaches thus far:
>
> Regarding the user interface, there are three approaches:
>
> 1. Implementing SQL function controls (e.g.,
> pg_activate_logical_decoding() and pg_deactivate_logical_decoding()).
> This would enable users to activate logical decoding even with
> wal_level=replica by calling the SQL function. While cloud providers
> seem not like having multiple configuration methods, this could
> potentially be managed through appropriate EXECUTE privileges. Another
> drawback is the user confusion when 'SHOW wal_level' displays
> 'replica' despite processes writing WAL records with logical
> information. This might be dealt with by implementing a show_hook
> function for wal_level.
>
> 2. Implementing automatic logical decoding activation. This would
> trigger upon creation of the first logical slot and deactivate upon
> removal of the final slot. This approach shares the user confusion
> concern of the first proposal. Moreover, it presents a significant
> limitation: users would be unable to utilize logical decoding on
> standby servers without maintaining at least one logical slot on the
> primary -- a substantial disadvantage.
>
> 3. Converting wal_level to a SIGHUP parameter, thereby supporting all
> possible wal_level transition combinations. While this represents the
> most elegant solution among the proposals, it necessitates additional
> development effort for less common scenarios, such as transitioning
> between 'minimal' and 'replica' levels. Such transitions require
> specific handling -- for instance, changing between 'minimal' and
> 'replica' requires a checkpoint, while decreasing from 'replica' to
> 'minimal' necessitates terminating certain processes like WAL senders
> and archiver.
>
> We also had discussion (and I did some research) on the implementation
> of increasing/decreasing wal_level online. The basic idea is that we
> first enable logical information WAL-logging to all processes while
> maintaining the logical decoding in an inactive state. Once we can
> guarantee that all processes are writing WAL records with logical
> information, we enable the logical decoding. This guarantee can be
> achieved by waiting for all concurrent transactions to finish, which
> could make us wait for a long time if a transaction is long-running.
> Another way is to send a global barrier signal and wait for all
> processes to start writing WAL records with logical information. We
> have a good facility for that: EmitProcSignalBarrier() and
> WaitForProcSignalBarrier(). That way, we don't need to wait for
> transaction finishes.
>
> Based on the discussion so far, the idea 3 appears most promising. I
> welcome any additional suggestions or preferences.

I think this is the cleanest solution but harder to implement.
Performing any heavy lifting like waiting for other transactions to
finish or a barrier inside pg_reload_conf() increases the chances of
delaying conf reload. Further, if there are errors, it might cause
some configurations to be not loaded. So the actual processing needs
to happen after the configurations have been loaded.

--
Best Wishes,
Ashutosh Bapat

On Thu, Jan 23, 2025 at 3:24 AM Ashutosh Bapat
<ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
>
> On Thu, Jan 23, 2025 at 6:16 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Fri, Jan 10, 2025 at 12:33 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Thu, Jan 9, 2025 at 3:29 AM Ashutosh Bapat
> > > <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> > > >
> > > > On Tue, Dec 31, 2024 at 10:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Logical decoding (and logical replication) are available only when
> > > > > wal_level = logical. As the documentation says[1], Using the 'logical'
> > > > > level increases the WAL volume which could negatively affect the
> > > > > performance. For that reason, users might want to start with using
> > > > > 'replica', but when they want to use logical decoding they need a
> > > > > server restart to increase wal_level to 'logical'. My goal is to allow
> > > > > users who are using 'replica' level to use logical decoding without a
> > > > > server restart. There are other GUC parameters related to logical
> > > > > decoding and logical replication such as max_wal_senders,
> > > > > max_logical_replication_workers, and max_replication_slots, but even
> > > > > if users set these parameters >0, there would not be a noticeable
> > > > > performance impact. And their default values are already >0. So I'd
> > > > > like to focus on making only the wal_level dynamic GUC parameter.
> > > > > There are several earlier discussions[2][3] but no one has submitted
> > > > > patches unless I'm missing something.
> > > > >
> > > > > The first idea I came up with is to make the wal_level a PGC_SIGHUP
> > > > > parameter. However, it affects not only setting 'replica' to 'logical'
> > > > > but also setting 'minimal' to 'replica' or higher. I'm not sure the
> > > > > latter case is common and it might require a checkpoint. I don't want
> > > > > to make the patch complex for uncommon cases.
> > > > >
> > > > > The second idea is to somehow allow both WAL-logging logical info and
> > > > > logical decoding even when wal_level is 'replica'. I've attached a PoC
> > > > > patch for that. The patch introduces new SQL functions such as
> > > > > pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> > > > > These functions are available only when wal_level is 'repilca'(or
> > > > > higher). In pg_activate_logical_decoding(), we set the status of
> > > > > logical decoding stored on the shared memory from 'disabled' to
> > > > > 'xlog-logical-info', allowing all processes to write logical
> > > > > information to WAL records for logical decoding. But the logical
> > > > > decoding is still not allowed. Once we confirm all in-progress
> > > > > transactions completed, we switch the status to
> > > > > 'logical-decoding-ready', meaning that users can create logical
> > > > > replication slots and use logical decoding.
> > > > >
> > > > > Overall, with the patch, there are two ways to enable logical
> > > > > decoding: setting wal_level to 'logical' and calling
> > > > > pg_activate_logical_decoding() when wal_level is 'replica'. I left the
> > > > > 'logical' level for backward compatibility and for users who want to
> > > > > enable the logical decoding without calling that SQL function. If we
> > > > > can automatically enable the logical decoding when creating the first
> > > > > logical replication slot, probably we no longer need the 'logical'
> > > > > level. There is room to discuss the user interface. Feedback is very
> > > > > welcome.
> > > > >
> > > >
> > > > If a server is running at minimal wal_level and they want to enable
> > > > logical replication, they would still need a server restart. That
> > > > would be rare but not completely absent.
> > >
> > > Currently we don't allow the server to start with the 'minimal' level
> > > and max_wal_senders > 0. Even if we support changing 'minimal' to
> > > 'logical' without a server restart, we still need a server restart to
> > > increase max_wal_senders for users who want to use logical
> > > replication. Or we need to eliminate this restriction too. I guess it
> > > would be too complex for such uncommon use cases.
> > >
> > > >
> > > > Our documentation says "wal_level determines how much information is
> > > > written to the WAL.". Users would may not expect that the WAL amount
> > > > changes while wal_level = replica depending upon whether logical
> > > > decoding is possible. It may be possible to set the expectations right
> > > > by changing the documentation. It's not in the patch, so I am not sure
> > > > whether this is considered.
> > >
> > > We should mention that in the doc. The WAL amount changes depending on
> > > not only wal_level but also other parameters such as wal_log_hints and
> > > full_page_writes.
> > >
> > > > Cloud providers do not like multiple ways of changing configuration
> > > > esp. when they can not control it. See [1]. Changing wal_level through
> > > > a SQL function may fit the same category.
> > >
> > > Thank you for pointing it out. This would support the idea of
> > > automatically enabling logical decoding.
> > >
> > > > I agree that it would be a lot of work to make all combinations of
> > > > wal_level changes work, but changing wal_level through SIGHUP looks
> > > > like a cleaner solution. Is there way that we make the GUC SIGHUP but
> > > > disallow certain combinations of old and new values?
> > >
> > > While I agree that it's cleaner I think there is no way today. I think
> > > we need to invent something for that.
> >
> > I would like to summarize the proposed approaches thus far:
> >
> > Regarding the user interface, there are three approaches:
> >
> > 1. Implementing SQL function controls (e.g.,
> > pg_activate_logical_decoding() and pg_deactivate_logical_decoding()).
> > This would enable users to activate logical decoding even with
> > wal_level=replica by calling the SQL function. While cloud providers
> > seem not like having multiple configuration methods, this could
> > potentially be managed through appropriate EXECUTE privileges. Another
> > drawback is the user confusion when 'SHOW wal_level' displays
> > 'replica' despite processes writing WAL records with logical
> > information. This might be dealt with by implementing a show_hook
> > function for wal_level.
> >
> > 2. Implementing automatic logical decoding activation. This would
> > trigger upon creation of the first logical slot and deactivate upon
> > removal of the final slot. This approach shares the user confusion
> > concern of the first proposal. Moreover, it presents a significant
> > limitation: users would be unable to utilize logical decoding on
> > standby servers without maintaining at least one logical slot on the
> > primary -- a substantial disadvantage.
> >
> > 3. Converting wal_level to a SIGHUP parameter, thereby supporting all
> > possible wal_level transition combinations. While this represents the
> > most elegant solution among the proposals, it necessitates additional
> > development effort for less common scenarios, such as transitioning
> > between 'minimal' and 'replica' levels. Such transitions require
> > specific handling -- for instance, changing between 'minimal' and
> > 'replica' requires a checkpoint, while decreasing from 'replica' to
> > 'minimal' necessitates terminating certain processes like WAL senders
> > and archiver.
> >
> > We also had discussion (and I did some research) on the implementation
> > of increasing/decreasing wal_level online. The basic idea is that we
> > first enable logical information WAL-logging to all processes while
> > maintaining the logical decoding in an inactive state. Once we can
> > guarantee that all processes are writing WAL records with logical
> > information, we enable the logical decoding. This guarantee can be
> > achieved by waiting for all concurrent transactions to finish, which
> > could make us wait for a long time if a transaction is long-running.
> > Another way is to send a global barrier signal and wait for all
> > processes to start writing WAL records with logical information. We
> > have a good facility for that: EmitProcSignalBarrier() and
> > WaitForProcSignalBarrier(). That way, we don't need to wait for
> > transaction finishes.
> >
> > Based on the discussion so far, the idea 3 appears most promising. I
> > welcome any additional suggestions or preferences.
>
> I think this is the cleanest solution but harder to implement.
> Performing any heavy lifting like waiting for other transactions to
> finish or a barrier inside pg_reload_conf() increases the chances of
> delaying conf reload. Further, if there are errors, it might cause
> some configurations to be not loaded. So the actual processing needs
> to happen after the configurations have been loaded.

Here is an idea: we can use a background worker who does everything
for changing wal_level online. I thought we could delegate the work
for changing wal_level to the checkpointer process, but given that the
work involves creating a checkpoint, invalidating logical slots, and
terminating (physical) walsenders etc I think it would be better to
have a dedicated worker for that. I've attached a PoC patch for
discussion. It's still dirty, uncommented, and untested enough, but I
hope it will help move this project forward. In the patch, when the
checkpointer realizes that wal_level has been changed, it launches a
background worker, wal-level control worker. The worker changes
wal_level online using global barriers etc. and terminates
functionality such as WAL archiving, replication and logical decoding
if necessary.

While it might be too much to use a background worker just for
changing wal_level online, I think it's technically okay. Users would
need to have one open background worker slot, or we can implement it
using an auxiliary process. Such a worker process might be able to be
used for other parameters too in the future.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, Jan 22, 2025 at 11:42 PM Bertrand Drouvot
<bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Wed, Jan 22, 2025 at 04:46:00PM -0800, Masahiko Sawada wrote:
> > I would like to summarize the proposed approaches thus far:
>
> Thanks!
>
> > Regarding the user interface, there are three approaches:
> >
> > 1. Implementing SQL function controls (e.g.,
> > pg_activate_logical_decoding() and pg_deactivate_logical_decoding()).
> > This would enable users to activate logical decoding even with
> > wal_level=replica by calling the SQL function. While cloud providers
> > seem not like having multiple configuration methods, this could
> > potentially be managed through appropriate EXECUTE privileges. Another
> > drawback is the user confusion when 'SHOW wal_level' displays
> > 'replica' despite processes writing WAL records with logical
> > information. This might be dealt with by implementing a show_hook
> > function for wal_level.
> >
> > 2. Implementing automatic logical decoding activation. This would
> > trigger upon creation of the first logical slot and deactivate upon
> > removal of the final slot. This approach shares the user confusion
> > concern of the first proposal. Moreover, it presents a significant
> > limitation: users would be unable to utilize logical decoding on
> > standby servers without maintaining at least one logical slot on the
> > primary -- a substantial disadvantage.
>
> Yeah, unless we keep wal_level around but I agree that the following (3.) looks
> like the way to go (as it removes any confusion).
>
> > 3. Converting wal_level to a SIGHUP parameter, thereby supporting all
> > possible wal_level transition combinations. While this represents the
> > most elegant solution among the proposals,
>
> +1
>
> > it necessitates additional
> > development effort for less common scenarios, such as transitioning
> > between 'minimal' and 'replica' levels. Such transitions require
> > specific handling -- for instance, changing between 'minimal' and
> > 'replica' requires a checkpoint, while decreasing from 'replica' to
> > 'minimal' necessitates terminating certain processes like WAL senders
> > and archiver.
>
> Yeah. OTOH switching from replica to minimal is "dangerous" as it makes
> previous base backups unusable for point-in-time recovery. So I wonder if it
> wouldn't be better to keep a restart mandatory depending of the transition
> state (that would probably make users thinking "twice" before doing the
> transition that requires a restart). I don't think any GUC does that already but
> that might be something to explore, thoughts?

I'm concerned that such inconsistency could introduce confusion for
users. The outcome by changing wal_level to minimal is already
documented, and I guess users would check what parameters are going to
be changed even before reloading the config file. I'm not sure that
requiring a server restart only for lowering wal_level to 'minimal'
would really be a protection for unexpectedly making previous base
backups unusable.

It might make sense to raise a WARNING when ALTER SYSTEM lowers the
wal_level to 'minimal'.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Dear Sawada-san,

I love the idea. I've roughly tested the patch and worked on my env.
Here are initial comments...

1. xloglevelworker.c
```
+#include "replication/logicalxlog.h"
```

xloglevelworker.c includes replication/logicalxlog.h, but it does not exist.
The line had to be removed to build and test it.

2.
```
+static void
+writeUpdateWalLevel(int new_wal_level)
+{
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&new_wal_level), sizeof(bool));
+ XLogInsert(RM_XLOG_ID, XLOG_UPDATE_WAL_LEVEL);
+}
```

IIUC the data length should be sizeof(int) instead of sizeof(bool).

3.
Is there a reason why the process does not wait till the archiver exits?

4.
When I dumped wal files, I found that XLOG_UPDATE_WAL_LEVEL cannot be recognized:

```
rmgr: XLOG len (rec/tot): 27/ 27, tx: 0, lsn: 0/03050838, prev 0/03050800, desc: UNKNOWN (f0) wal_level logical
```

xlog_identify() must be updated as well.

5.
When I changed "logical" to "replica", postgres outputs like below:

```
LOG: received SIGHUP, reloading configuration files
LOG: parameter "wal_level" changed to "replica"
LOG: wal_level control worker started
LOG: changing wal_level from "logical" to "replica"
LOG: wal_level has been decreased to "replica"
LOG: successfully changed wal_level from "logical" to "replica"
```

ISTM that both postmaster and the wal_level control worker said something like
"wal_level changed", which is bit strange for me. Since GUC can't be renamed,
can we use another name for the wal_level control state?

6.
With the patch present, the wal_level can be changed to the minimal even when the
streaming replication is going. If we do that, the walsender exits immediately and
the below FATAL appears periodically until the standby stops. Same things can be
said for the logical replication:

```
FATAL: streaming replication receiver "walreceiver" could not connect to the primary server:
connection to server on socket "/tmp/.s.PGSQL.oooo" failed:
FATAL: WAL senders require "wal_level" to be "replica" or "logical
```

I know this is not a perfect, but can we avoid the issue by reject the GUC update
if the walsender exists? Another approach is not to update the value when replication
slots need to be invalidated.

----------
Best regards,
Haato Kuroda

On Tue, Jan 28, 2025 at 1:39 AM Hayato Kuroda (Fujitsu)
<kuroda(dot)hayato(at)fujitsu(dot)com> wrote:
>
> Dear Sawada-san,
>
> I love the idea. I've roughly tested the patch and worked on my env.
> Here are initial comments...

Thank you for looking at the patch!

>
> 1. xloglevelworker.c
> ```
> +#include "replication/logicalxlog.h"
> ```
>
> xloglevelworker.c includes replication/logicalxlog.h, but it does not exist.
> The line had to be removed to build and test it.
>
> 2.
> ```
> +static void
> +writeUpdateWalLevel(int new_wal_level)
> +{
> + XLogBeginInsert();
> + XLogRegisterData((char *) (&new_wal_level), sizeof(bool));
> + XLogInsert(RM_XLOG_ID, XLOG_UPDATE_WAL_LEVEL);
> +}
> ```
>
> IIUC the data length should be sizeof(int) instead of sizeof(bool).

Agreed to fix them.

>
> 3.
> Is there a reason why the process does not wait till the archiver exits?

No. I didn't implement this part as the patch was just for
proof-of-concept. I think it would be better to wait for it to exit.

>
> 4.
> When I dumped wal files, I found that XLOG_UPDATE_WAL_LEVEL cannot be recognized:
>
> ```
> rmgr: XLOG len (rec/tot): 27/ 27, tx: 0, lsn: 0/03050838, prev 0/03050800, desc: UNKNOWN (f0) wal_level logical
> ```
>
> xlog_identify() must be updated as well.

Will fix.

>
> 5.
> When I changed "logical" to "replica", postgres outputs like below:
>
> ```
> LOG: received SIGHUP, reloading configuration files
> LOG: parameter "wal_level" changed to "replica"
> LOG: wal_level control worker started
> LOG: changing wal_level from "logical" to "replica"
> LOG: wal_level has been decreased to "replica"
> LOG: successfully changed wal_level from "logical" to "replica"
> ```
>
> ISTM that both postmaster and the wal_level control worker said something like
> "wal_level changed", which is bit strange for me. Since GUC can't be renamed,
> can we use another name for the wal_level control state?

I'm concerned that users could be confused if two different names
refer to substantially the same thing.

Having said that, I guess that we need to drastically change the
messages. For example, I think that the wal_level worker should say
something like "successfully made 'logical' wal_level effective"
instead of saying something like "changed wal_level value". Also,
users might not need gradual messages when increasing 'minimal' to
'logical' or decreasing 'logical' to 'minimal'.

>
> 6.
> With the patch present, the wal_level can be changed to the minimal even when the
> streaming replication is going. If we do that, the walsender exits immediately and
> the below FATAL appears periodically until the standby stops. Same things can be
> said for the logical replication:
>
> ```
> FATAL: streaming replication receiver "walreceiver" could not connect to the primary server:
> connection to server on socket "/tmp/.s.PGSQL.oooo" failed:
> FATAL: WAL senders require "wal_level" to be "replica" or "logical
> ```
>
> I know this is not a perfect, but can we avoid the issue by reject the GUC update
> if the walsender exists? Another approach is not to update the value when replication
> slots need to be invalidated.

Does it mean that we reject the config file from being reloaded in
that case? I have no idea how to reject it in a case where the
wal_level in postgresql.conf changed and the user did 'pg_ctl reload'.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Dear Sawada-san,

> I'm concerned that users could be confused if two different names
> refer to substantially the same thing.
>
> Having said that, I guess that we need to drastically change the
> messages. For example, I think that the wal_level worker should say
> something like "successfully made 'logical' wal_level effective"
> instead of saying something like "changed wal_level value". Also,
> users might not need gradual messages when increasing 'minimal' to
> 'logical' or decreasing 'logical' to 'minimal'.

+1 for something like "successfully made 'logical' wal_level effective", and
removing gradual messages.

> > 6.
> > With the patch present, the wal_level can be changed to the minimal even when
> the
> > streaming replication is going. If we do that, the walsender exits immediately
> and
> > the below FATAL appears periodically until the standby stops. Same things can
> be
> > said for the logical replication:
> >
> > ```
> > FATAL: streaming replication receiver "walreceiver" could not connect to the
> primary server:
> > connection to server on socket "/tmp/.s.PGSQL.oooo" failed:
> > FATAL: WAL senders require "wal_level" to be "replica" or "logical
> > ```
> >
> > I know this is not a perfect, but can we avoid the issue by reject the GUC update
> > if the walsender exists? Another approach is not to update the value when
> replication
> > slots need to be invalidated.
>
> Does it mean that we reject the config file from being reloaded in
> that case? I have no idea how to reject it in a case where the
> wal_level in postgresql.conf changed and the user did 'pg_ctl reload'.

I imagined like attached. When I modified wal_level to minimal and send SIGHUP,
postmaster reported below lines and failed to update wal_level.

```
LOG: received SIGHUP, reloading configuration files
LOG: wal_level cannot be set to "minimal" while walsender exists
LOG: configuration file "...postgresql.conf" contains errors; unaffected changes were applied
```

Best regards,
Hayato Kuroda
FUJITSU LIMITED

On Mon, Feb 3, 2025 at 3:40 AM Hayato Kuroda (Fujitsu)
<kuroda(dot)hayato(at)fujitsu(dot)com> wrote:
>
> Dear Sawada-san,
>
> > I'm concerned that users could be confused if two different names
> > refer to substantially the same thing.
> >
> > Having said that, I guess that we need to drastically change the
> > messages. For example, I think that the wal_level worker should say
> > something like "successfully made 'logical' wal_level effective"
> > instead of saying something like "changed wal_level value". Also,
> > users might not need gradual messages when increasing 'minimal' to
> > 'logical' or decreasing 'logical' to 'minimal'.
>
> +1 for something like "successfully made 'logical' wal_level effective", and
> removing gradual messages.
>
> > > 6.
> > > With the patch present, the wal_level can be changed to the minimal even when
> > the
> > > streaming replication is going. If we do that, the walsender exits immediately
> > and
> > > the below FATAL appears periodically until the standby stops. Same things can
> > be
> > > said for the logical replication:
> > >
> > > ```
> > > FATAL: streaming replication receiver "walreceiver" could not connect to the
> > primary server:
> > > connection to server on socket "/tmp/.s.PGSQL.oooo" failed:
> > > FATAL: WAL senders require "wal_level" to be "replica" or "logical
> > > ```
> > >
> > > I know this is not a perfect, but can we avoid the issue by reject the GUC update
> > > if the walsender exists? Another approach is not to update the value when
> > replication
> > > slots need to be invalidated.
> >
> > Does it mean that we reject the config file from being reloaded in
> > that case? I have no idea how to reject it in a case where the
> > wal_level in postgresql.conf changed and the user did 'pg_ctl reload'.
>
> I imagined like attached. When I modified wal_level to minimal and send SIGHUP,
> postmaster reported below lines and failed to update wal_level.
>
> ```
> LOG: received SIGHUP, reloading configuration files
> LOG: wal_level cannot be set to "minimal" while walsender exists
> LOG: configuration file "...postgresql.conf" contains errors; unaffected changes were applied
> ```

Interesting, and thanks for sharing the patch. But I think that when
we change the wal_level to 'minimal', there is a window where a new
walsender can launch after passing the check_wal_level() check.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Fri, Jan 24, 2025 at 11:16 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Thu, Jan 23, 2025 at 3:24 AM Ashutosh Bapat
> <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> >
> > On Thu, Jan 23, 2025 at 6:16 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Fri, Jan 10, 2025 at 12:33 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > On Thu, Jan 9, 2025 at 3:29 AM Ashutosh Bapat
> > > > <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> > > > >
> > > > > On Tue, Dec 31, 2024 at 10:15 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Logical decoding (and logical replication) are available only when
> > > > > > wal_level = logical. As the documentation says[1], Using the 'logical'
> > > > > > level increases the WAL volume which could negatively affect the
> > > > > > performance. For that reason, users might want to start with using
> > > > > > 'replica', but when they want to use logical decoding they need a
> > > > > > server restart to increase wal_level to 'logical'. My goal is to allow
> > > > > > users who are using 'replica' level to use logical decoding without a
> > > > > > server restart. There are other GUC parameters related to logical
> > > > > > decoding and logical replication such as max_wal_senders,
> > > > > > max_logical_replication_workers, and max_replication_slots, but even
> > > > > > if users set these parameters >0, there would not be a noticeable
> > > > > > performance impact. And their default values are already >0. So I'd
> > > > > > like to focus on making only the wal_level dynamic GUC parameter.
> > > > > > There are several earlier discussions[2][3] but no one has submitted
> > > > > > patches unless I'm missing something.
> > > > > >
> > > > > > The first idea I came up with is to make the wal_level a PGC_SIGHUP
> > > > > > parameter. However, it affects not only setting 'replica' to 'logical'
> > > > > > but also setting 'minimal' to 'replica' or higher. I'm not sure the
> > > > > > latter case is common and it might require a checkpoint. I don't want
> > > > > > to make the patch complex for uncommon cases.
> > > > > >
> > > > > > The second idea is to somehow allow both WAL-logging logical info and
> > > > > > logical decoding even when wal_level is 'replica'. I've attached a PoC
> > > > > > patch for that. The patch introduces new SQL functions such as
> > > > > > pg_activate_logical_decoding() and pg_deactivate_logical_decoding().
> > > > > > These functions are available only when wal_level is 'repilca'(or
> > > > > > higher). In pg_activate_logical_decoding(), we set the status of
> > > > > > logical decoding stored on the shared memory from 'disabled' to
> > > > > > 'xlog-logical-info', allowing all processes to write logical
> > > > > > information to WAL records for logical decoding. But the logical
> > > > > > decoding is still not allowed. Once we confirm all in-progress
> > > > > > transactions completed, we switch the status to
> > > > > > 'logical-decoding-ready', meaning that users can create logical
> > > > > > replication slots and use logical decoding.
> > > > > >
> > > > > > Overall, with the patch, there are two ways to enable logical
> > > > > > decoding: setting wal_level to 'logical' and calling
> > > > > > pg_activate_logical_decoding() when wal_level is 'replica'. I left the
> > > > > > 'logical' level for backward compatibility and for users who want to
> > > > > > enable the logical decoding without calling that SQL function. If we
> > > > > > can automatically enable the logical decoding when creating the first
> > > > > > logical replication slot, probably we no longer need the 'logical'
> > > > > > level. There is room to discuss the user interface. Feedback is very
> > > > > > welcome.
> > > > > >
> > > > >
> > > > > If a server is running at minimal wal_level and they want to enable
> > > > > logical replication, they would still need a server restart. That
> > > > > would be rare but not completely absent.
> > > >
> > > > Currently we don't allow the server to start with the 'minimal' level
> > > > and max_wal_senders > 0. Even if we support changing 'minimal' to
> > > > 'logical' without a server restart, we still need a server restart to
> > > > increase max_wal_senders for users who want to use logical
> > > > replication. Or we need to eliminate this restriction too. I guess it
> > > > would be too complex for such uncommon use cases.
> > > >
> > > > >
> > > > > Our documentation says "wal_level determines how much information is
> > > > > written to the WAL.". Users would may not expect that the WAL amount
> > > > > changes while wal_level = replica depending upon whether logical
> > > > > decoding is possible. It may be possible to set the expectations right
> > > > > by changing the documentation. It's not in the patch, so I am not sure
> > > > > whether this is considered.
> > > >
> > > > We should mention that in the doc. The WAL amount changes depending on
> > > > not only wal_level but also other parameters such as wal_log_hints and
> > > > full_page_writes.
> > > >
> > > > > Cloud providers do not like multiple ways of changing configuration
> > > > > esp. when they can not control it. See [1]. Changing wal_level through
> > > > > a SQL function may fit the same category.
> > > >
> > > > Thank you for pointing it out. This would support the idea of
> > > > automatically enabling logical decoding.
> > > >
> > > > > I agree that it would be a lot of work to make all combinations of
> > > > > wal_level changes work, but changing wal_level through SIGHUP looks
> > > > > like a cleaner solution. Is there way that we make the GUC SIGHUP but
> > > > > disallow certain combinations of old and new values?
> > > >
> > > > While I agree that it's cleaner I think there is no way today. I think
> > > > we need to invent something for that.
> > >
> > > I would like to summarize the proposed approaches thus far:
> > >
> > > Regarding the user interface, there are three approaches:
> > >
> > > 1. Implementing SQL function controls (e.g.,
> > > pg_activate_logical_decoding() and pg_deactivate_logical_decoding()).
> > > This would enable users to activate logical decoding even with
> > > wal_level=replica by calling the SQL function. While cloud providers
> > > seem not like having multiple configuration methods, this could
> > > potentially be managed through appropriate EXECUTE privileges. Another
> > > drawback is the user confusion when 'SHOW wal_level' displays
> > > 'replica' despite processes writing WAL records with logical
> > > information. This might be dealt with by implementing a show_hook
> > > function for wal_level.
> > >
> > > 2. Implementing automatic logical decoding activation. This would
> > > trigger upon creation of the first logical slot and deactivate upon
> > > removal of the final slot. This approach shares the user confusion
> > > concern of the first proposal. Moreover, it presents a significant
> > > limitation: users would be unable to utilize logical decoding on
> > > standby servers without maintaining at least one logical slot on the
> > > primary -- a substantial disadvantage.
> > >
> > > 3. Converting wal_level to a SIGHUP parameter, thereby supporting all
> > > possible wal_level transition combinations. While this represents the
> > > most elegant solution among the proposals, it necessitates additional
> > > development effort for less common scenarios, such as transitioning
> > > between 'minimal' and 'replica' levels. Such transitions require
> > > specific handling -- for instance, changing between 'minimal' and
> > > 'replica' requires a checkpoint, while decreasing from 'replica' to
> > > 'minimal' necessitates terminating certain processes like WAL senders
> > > and archiver.
> > >
> > > We also had discussion (and I did some research) on the implementation
> > > of increasing/decreasing wal_level online. The basic idea is that we
> > > first enable logical information WAL-logging to all processes while
> > > maintaining the logical decoding in an inactive state. Once we can
> > > guarantee that all processes are writing WAL records with logical
> > > information, we enable the logical decoding. This guarantee can be
> > > achieved by waiting for all concurrent transactions to finish, which
> > > could make us wait for a long time if a transaction is long-running.
> > > Another way is to send a global barrier signal and wait for all
> > > processes to start writing WAL records with logical information. We
> > > have a good facility for that: EmitProcSignalBarrier() and
> > > WaitForProcSignalBarrier(). That way, we don't need to wait for
> > > transaction finishes.
> > >
> > > Based on the discussion so far, the idea 3 appears most promising. I
> > > welcome any additional suggestions or preferences.
> >
> > I think this is the cleanest solution but harder to implement.
> > Performing any heavy lifting like waiting for other transactions to
> > finish or a barrier inside pg_reload_conf() increases the chances of
> > delaying conf reload. Further, if there are errors, it might cause
> > some configurations to be not loaded. So the actual processing needs
> > to happen after the configurations have been loaded.
>
> Here is an idea: we can use a background worker who does everything
> for changing wal_level online. I thought we could delegate the work
> for changing wal_level to the checkpointer process, but given that the
> work involves creating a checkpoint, invalidating logical slots, and
> terminating (physical) walsenders etc I think it would be better to
> have a dedicated worker for that. I've attached a PoC patch for
> discussion. It's still dirty, uncommented, and untested enough, but I
> hope it will help move this project forward. In the patch, when the
> checkpointer realizes that wal_level has been changed, it launches a
> background worker, wal-level control worker. The worker changes
> wal_level online using global barriers etc. and terminates
> functionality such as WAL archiving, replication and logical decoding
> if necessary.
>
> While it might be too much to use a background worker just for
> changing wal_level online, I think it's technically okay. Users would
> need to have one open background worker slot, or we can implement it
> using an auxiliary process. Such a worker process might be able to be
> used for other parameters too in the future.

I've updated the patch that includes comment updates and bug fixes.

The main idea of changing WAL level online is to decouple two aspects:
(1) the information included in WAL records and (2) the
functionalities available at each WAL level. With that, we can change
the WAL level gradually. For example, when increasing the WAL level
from 'replica' to 'logical', we first switch the WAL level on the
shared memory to a new higher level where we allow processes to write
WAL records with additional information required by the logical
decoding, while keeping the logical decoding unavailable. The new
level is something between 'replica' and 'logical'. Once we confirm
all processes have synchronized to the new level, we increase the WAL
level further to 'logical', allowing us to start logical decoding. The
patch supports all combinations of WAL level transitions. It makes
sense to me to use a background worker to proceed with this transition
work since we need to wait at some points, rather than delegating it
to the checkpointer process.

With the patch, wal_level global variable is no longer reliable to get
the actual WAL level since it introduces the authoritative WAL level
is stored in the shared memory that is protected by a lwlock. However,
processes cache flags about the information they need to include to
the WAL records so they don't need to access the shared memory if they
check XLogStandbyInfoActive() or XLogLogicalInfoActive(). The cache is
updated when the process startup or absorbing the global signal
barrier. On the other hand, if they need to check the current active
WAL level, they need to get the WAL level using GetActiveWalLevel()
that accesses the shared memory with locks. The reason I made such
changes is that while the information (1) could be accessed very
frequently for example every XLOG_HEAP_INSERT insertion (via
RelationIsAccessibleInLogicalDecoding()), the information (2) is not
(for example when starting the logical decoding).

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On Tue, Feb 11, 2025 at 02:11:10PM -0800, Masahiko Sawada wrote:
> I've updated the patch that includes comment updates and bug fixes.

Thanks!

> The main idea of changing WAL level online is to decouple two aspects:
> (1) the information included in WAL records and (2) the
> functionalities available at each WAL level. With that, we can change
> the WAL level gradually. For example, when increasing the WAL level
> from 'replica' to 'logical', we first switch the WAL level on the
> shared memory to a new higher level where we allow processes to write
> WAL records with additional information required by the logical
> decoding, while keeping the logical decoding unavailable. The new
> level is something between 'replica' and 'logical'. Once we confirm
> all processes have synchronized to the new level, we increase the WAL
> level further to 'logical', allowing us to start logical decoding. The
> patch supports all combinations of WAL level transitions. It makes
> sense to me to use a background worker to proceed with this transition
> work since we need to wait at some points, rather than delegating it
> to the checkpointer process.

The background worker being added is "wal_level control worker". I wonder if
it would make sense to create a more "generic" one instead (to whom we could
assign more "tasks" later on, as suggested in the past in [1]).

+ /*
+ * XXX: Perhaps it's not okay that we failed to launch a bgworker and give
+ * up wal_level change because we already reported that the change has
+ * been accepted. Do we need to use aux process instead for that purpose?
+ */
+ if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
+ ereport(WARNING,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg("out of background worker slots"),
+ errhint("You might need to increase \"%s\".", "max_worker_processes")));

Not sure it has to be an aux process instead as it should be busy in rare occasions.

Maybe we could add some mechanism for ensuring that a bgworker slot is available
when needed (as suggested in [2])?

Not saying it has to be done that way. I just thought that the "wal_level control worker"
could be a perfect use case/starting point for a more generic one but I don't want
to over complicate that thread though.

So maybe just rename "wal_level control worker" to say "custodian worker" and
we could also think about [2]? Feel free to consider all of this as Nits if you
feel it deviates too much from the initial intend of this thread.

[1]: https://www.postgresql.org/message-id/flat/C1EE64B0-D4DB-40F3-98C8-0CED324D34CB%40amazon.com
[2]: https://www.postgresql.org/message-id/1058306.1680467858%40sss.pgh.pa.us

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Tue, Feb 11, 2025 at 11:44 PM Bertrand Drouvot
<bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Tue, Feb 11, 2025 at 02:11:10PM -0800, Masahiko Sawada wrote:
> > I've updated the patch that includes comment updates and bug fixes.
>
> Thanks!
>
> > The main idea of changing WAL level online is to decouple two aspects:
> > (1) the information included in WAL records and (2) the
> > functionalities available at each WAL level. With that, we can change
> > the WAL level gradually. For example, when increasing the WAL level
> > from 'replica' to 'logical', we first switch the WAL level on the
> > shared memory to a new higher level where we allow processes to write
> > WAL records with additional information required by the logical
> > decoding, while keeping the logical decoding unavailable. The new
> > level is something between 'replica' and 'logical'. Once we confirm
> > all processes have synchronized to the new level, we increase the WAL
> > level further to 'logical', allowing us to start logical decoding. The
> > patch supports all combinations of WAL level transitions. It makes
> > sense to me to use a background worker to proceed with this transition
> > work since we need to wait at some points, rather than delegating it
> > to the checkpointer process.
>
> The background worker being added is "wal_level control worker". I wonder if
> it would make sense to create a more "generic" one instead (to whom we could
> assign more "tasks" later on, as suggested in the past in [1]).
>
> + /*
> + * XXX: Perhaps it's not okay that we failed to launch a bgworker and give
> + * up wal_level change because we already reported that the change has
> + * been accepted. Do we need to use aux process instead for that purpose?
> + */
> + if (!RegisterDynamicBackgroundWorker(&bgw, &bgw_handle))
> + ereport(WARNING,
> + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
> + errmsg("out of background worker slots"),
> + errhint("You might need to increase \"%s\".", "max_worker_processes")));
>
> Not sure it has to be an aux process instead as it should be busy in rare occasions.

Thank you for referring to the custodian worker thread. I'm not sure
that online wal_level change work would fit the concept of custodian
worker, which offloads some work for time-critical works such as
checkpointing, but this idea made me think of other possible
directions of this work.

Looking at the latest custodian worker patch, the basic architecture
is to have a single custodian worker and processes can ask it for some
work such as removing logical decoding related files. The online
wal_level change will be the one of the tasks that processes (eps.
checkpointer) can ask for it. On the other hand, one point that I
think might not fit this wal_level work well is that while the
custodian worker is a long-lived worker process, it's sufficient for
the online wal_level change work to have a bgworker that does its work
and then exits. IOW, from the perspective of this work, I prefer the
idea of having one short-lived worker for one task over having one
long-lived worker for multiple tasks. Reading that thread, while we
need to resolve the XID wraparound issue for the work of removing
logical decoding related files, the work of removing temporary files
seems to fit a short-lived worker style. So I thought as one of the
directions, it might be worth considering to have an infrastructure
where we can launch a bgworker just for one task, and we implement the
online wal_level change and temporary files removal on top of it.

> Maybe we could add some mechanism for ensuring that a bgworker slot is available
> when needed (as suggested in [2])?

Yeah, we need this mechanism if we use a bgworker for these works.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On Fri, Feb 14, 2025 at 12:17:48AM -0800, Masahiko Sawada wrote:
> On Tue, Feb 11, 2025 at 11:44 PM Bertrand Drouvot
> <bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:

> Looking at the latest custodian worker patch, the basic architecture
> is to have a single custodian worker and processes can ask it for some
> work such as removing logical decoding related files. The online
> wal_level change will be the one of the tasks that processes (eps.
> checkpointer) can ask for it. On the other hand, one point that I
> think might not fit this wal_level work well is that while the
> custodian worker is a long-lived worker process,

That was the case initialy but it looks like it would not have been the case
at the end. See, Tom's comment in [1]:

"
I wonder if a single long-lived custodian task is the right model at all.
At least for RemovePgTempFiles, it'd make more sense to write it as a
background worker that spawns, does its work, and then exits,
independently of anything else
"

> it's sufficient for
> the online wal_level change work to have a bgworker that does its work
> and then exits.

Fully agree and I did not think about changing this behavior.

> IOW, from the perspective of this work, I prefer the
> idea of having one short-lived worker for one task over having one
> long-lived worker for multiple tasks.

Yeah, or one short-lived worker for multiple tasks could work too. It just
starts when it has something to do and then exit.

> Reading that thread, while we
> need to resolve the XID wraparound issue for the work of removing
> logical decoding related files, the work of removing temporary files
> seems to fit a short-lived worker style. So I thought as one of the
> directions, it might be worth considering to have an infrastructure
> where we can launch a bgworker just for one task, and we implement the
> online wal_level change and temporary files removal on top of it.

Yeap, that was exactly my point when I mentioned the custodian thread (taking
into account Tom's comment quoted above).

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Fri, Feb 14, 2025 at 2:35 AM Bertrand Drouvot
<bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Fri, Feb 14, 2025 at 12:17:48AM -0800, Masahiko Sawada wrote:
> > On Tue, Feb 11, 2025 at 11:44 PM Bertrand Drouvot
> > <bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
>
> > Looking at the latest custodian worker patch, the basic architecture
> > is to have a single custodian worker and processes can ask it for some
> > work such as removing logical decoding related files. The online
> > wal_level change will be the one of the tasks that processes (eps.
> > checkpointer) can ask for it. On the other hand, one point that I
> > think might not fit this wal_level work well is that while the
> > custodian worker is a long-lived worker process,
>
> That was the case initialy but it looks like it would not have been the case
> at the end. See, Tom's comment in [1]:
>
> "
> I wonder if a single long-lived custodian task is the right model at all.
> At least for RemovePgTempFiles, it'd make more sense to write it as a
> background worker that spawns, does its work, and then exits,
> independently of anything else
> "
>
> > it's sufficient for
> > the online wal_level change work to have a bgworker that does its work
> > and then exits.
>
> Fully agree and I did not think about changing this behavior.
>
> > IOW, from the perspective of this work, I prefer the
> > idea of having one short-lived worker for one task over having one
> > long-lived worker for multiple tasks.
>
> Yeah, or one short-lived worker for multiple tasks could work too. It just
> starts when it has something to do and then exit.
>
> > Reading that thread, while we
> > need to resolve the XID wraparound issue for the work of removing
> > logical decoding related files, the work of removing temporary files
> > seems to fit a short-lived worker style. So I thought as one of the
> > directions, it might be worth considering to have an infrastructure
> > where we can launch a bgworker just for one task, and we implement the
> > online wal_level change and temporary files removal on top of it.
>
> Yeap, that was exactly my point when I mentioned the custodian thread (taking
> into account Tom's comment quoted above).
>

I've written PoC patches to have the online wal_level change work use
a more generic infrastructure. These patches are still in PoC state
but seem like a good direction to me. Here is a brief explanation for
each patch.

* The 0001 patch introduces "reserved background worker slots". We
allocate max_process_workers + BGWORKER_CLASS_RESERVED at startup, and
if the number of running bgworker exceeds max_worker_processes, only
workers using the reserved slots can be launched. We can request to
use the reserved slots by adding BGWORKER_CLASS_RESERVED flag at
bgworker registration.

* The 0002 patch introduces "bgtask worker". The bgtask infrastructure
is designed to execute internal tasks in background in
one-worker-per-one-task style. Internally, bgtask workers use the
reserved bgworker so it's guaranteed that they can launch. The
internal tasks that we can request are predefined and this patch has a
dummy task as a placeholder. This patch implements only the minimal
functionality for the online wal_level change work. I've not tested if
this bgtask infrastructure can be used for tasks that we wanted to
offload to the custodian worker.

* The 0003 patch makes wal_level a SIGHUP parameter. We do the online
wal_level change work using the bgtask infrastructure. There are no
major changes from the previous version other than that.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On Mon, Feb 17, 2025 at 12:07:56PM -0800, Masahiko Sawada wrote:
> On Fri, Feb 14, 2025 at 2:35 AM Bertrand Drouvot
> <bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
> > Yeap, that was exactly my point when I mentioned the custodian thread (taking
> > into account Tom's comment quoted above).
> >
>
> I've written PoC patches to have the online wal_level change work use
> a more generic infrastructure. These patches are still in PoC state
> but seem like a good direction to me. Here is a brief explanation for
> each patch.

Thanks for the patches!

> * The 0001 patch introduces "reserved background worker slots". We
> allocate max_process_workers + BGWORKER_CLASS_RESERVED at startup, and
> if the number of running bgworker exceeds max_worker_processes, only
> workers using the reserved slots can be launched. We can request to
> use the reserved slots by adding BGWORKER_CLASS_RESERVED flag at
> bgworker registration.

I had a quick look at 0001 and I think the way that's implemented is reasonnable.
I thought this could be defined through a GUC so that extensions can benefit
from it. But OTOH the core code should ensure the value is > as the number of
reserved slots needed by the core so not using a GUC looks ok to me.

> * The 0002 patch introduces "bgtask worker". The bgtask infrastructure
> is designed to execute internal tasks in background in
> one-worker-per-one-task style. Internally, bgtask workers use the
> reserved bgworker so it's guaranteed that they can launch.

Yeah.

> The
> internal tasks that we can request are predefined and this patch has a
> dummy task as a placeholder. This patch implements only the minimal
> functionality for the online wal_level change work. I've not tested if
> this bgtask infrastructure can be used for tasks that we wanted to
> offload to the custodian worker.

Again, I had a quick look and looks simple enough of our need here. It "just"
executes "(void) InternalBgTasks[type].func()" and then exists. That's, I think,
a good starting point to add more tasks in the future (if we want to).

> * The 0003 patch makes wal_level a SIGHUP parameter. We do the online
> wal_level change work using the bgtask infrastructure. There are no
> major changes from the previous version other than that.

It replaces the dummy task introduced in 0002 by the one that suits our needs
here (through the new BgTaskWalLevelChange() function).

The design looks reasonable to me. Waiting to see if others disagree before
looking more closely at the code.

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Wed, Feb 19, 2025 at 1:56 AM Bertrand Drouvot
<bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
>
> Hi,

Thank you for looking at the patches.

>
> On Mon, Feb 17, 2025 at 12:07:56PM -0800, Masahiko Sawada wrote:
> > On Fri, Feb 14, 2025 at 2:35 AM Bertrand Drouvot
> > <bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
> > > Yeap, that was exactly my point when I mentioned the custodian thread (taking
> > > into account Tom's comment quoted above).
> > >
> >
> > I've written PoC patches to have the online wal_level change work use
> > a more generic infrastructure. These patches are still in PoC state
> > but seem like a good direction to me. Here is a brief explanation for
> > each patch.
>
> Thanks for the patches!
>
> > * The 0001 patch introduces "reserved background worker slots". We
> > allocate max_process_workers + BGWORKER_CLASS_RESERVED at startup, and
> > if the number of running bgworker exceeds max_worker_processes, only
> > workers using the reserved slots can be launched. We can request to
> > use the reserved slots by adding BGWORKER_CLASS_RESERVED flag at
> > bgworker registration.
>
> I had a quick look at 0001 and I think the way that's implemented is reasonnable.
> I thought this could be defined through a GUC so that extensions can benefit
> from it. But OTOH the core code should ensure the value is > as the number of
> reserved slots needed by the core so not using a GUC looks ok to me.

Interesting idea. I kept the reserved slots only for internal use but
it would be worth considering to use GUC instead.

> > * The 0002 patch introduces "bgtask worker". The bgtask infrastructure
> > is designed to execute internal tasks in background in
> > one-worker-per-one-task style. Internally, bgtask workers use the
> > reserved bgworker so it's guaranteed that they can launch.
>
> Yeah.
>
> > The
> > internal tasks that we can request are predefined and this patch has a
> > dummy task as a placeholder. This patch implements only the minimal
> > functionality for the online wal_level change work. I've not tested if
> > this bgtask infrastructure can be used for tasks that we wanted to
> > offload to the custodian worker.
>
> Again, I had a quick look and looks simple enough of our need here. It "just"
> executes "(void) InternalBgTasks[type].func()" and then exists. That's, I think,
> a good starting point to add more tasks in the future (if we want to).

Yeah, we might want to extend it further, for example to pass an
argument to the background task or to ask multiple tasks for the
single bgtask worker. As far as I can read the custodian patch set,
the work of removing temp files seems not to require any argument
though.

>
> > * The 0003 patch makes wal_level a SIGHUP parameter. We do the online
> > wal_level change work using the bgtask infrastructure. There are no
> > major changes from the previous version other than that.
>
> It replaces the dummy task introduced in 0002 by the one that suits our needs
> here (through the new BgTaskWalLevelChange() function).
>
> The design looks reasonable to me. Waiting to see if others disagree before
> looking more closely at the code.

Thanks.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Feb 20, 2025 at 10:05 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Wed, Feb 19, 2025 at 1:56 AM Bertrand Drouvot
> <bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
> >
> > Hi,
>
> Thank you for looking at the patches.
>
> >
> > On Mon, Feb 17, 2025 at 12:07:56PM -0800, Masahiko Sawada wrote:
> > > On Fri, Feb 14, 2025 at 2:35 AM Bertrand Drouvot
> > > <bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
> > > > Yeap, that was exactly my point when I mentioned the custodian thread (taking
> > > > into account Tom's comment quoted above).
> > > >
> > >
> > > I've written PoC patches to have the online wal_level change work use
> > > a more generic infrastructure. These patches are still in PoC state
> > > but seem like a good direction to me. Here is a brief explanation for
> > > each patch.
> >
> > Thanks for the patches!
> >
> > > * The 0001 patch introduces "reserved background worker slots". We
> > > allocate max_process_workers + BGWORKER_CLASS_RESERVED at startup, and
> > > if the number of running bgworker exceeds max_worker_processes, only
> > > workers using the reserved slots can be launched. We can request to
> > > use the reserved slots by adding BGWORKER_CLASS_RESERVED flag at
> > > bgworker registration.
> >
> > I had a quick look at 0001 and I think the way that's implemented is reasonnable.
> > I thought this could be defined through a GUC so that extensions can benefit
> > from it. But OTOH the core code should ensure the value is > as the number of
> > reserved slots needed by the core so not using a GUC looks ok to me.
>
> Interesting idea. I kept the reserved slots only for internal use but
> it would be worth considering to use GUC instead.
>
> > > * The 0002 patch introduces "bgtask worker". The bgtask infrastructure
> > > is designed to execute internal tasks in background in
> > > one-worker-per-one-task style. Internally, bgtask workers use the
> > > reserved bgworker so it's guaranteed that they can launch.
> >
> > Yeah.
> >
> > > The
> > > internal tasks that we can request are predefined and this patch has a
> > > dummy task as a placeholder. This patch implements only the minimal
> > > functionality for the online wal_level change work. I've not tested if
> > > this bgtask infrastructure can be used for tasks that we wanted to
> > > offload to the custodian worker.
> >
> > Again, I had a quick look and looks simple enough of our need here. It "just"
> > executes "(void) InternalBgTasks[type].func()" and then exists. That's, I think,
> > a good starting point to add more tasks in the future (if we want to).
>
> Yeah, we might want to extend it further, for example to pass an
> argument to the background task or to ask multiple tasks for the
> single bgtask worker. As far as I can read the custodian patch set,
> the work of removing temp files seems not to require any argument
> though.
>
> >
> > > * The 0003 patch makes wal_level a SIGHUP parameter. We do the online
> > > wal_level change work using the bgtask infrastructure. There are no
> > > major changes from the previous version other than that.
> >
> > It replaces the dummy task introduced in 0002 by the one that suits our needs
> > here (through the new BgTaskWalLevelChange() function).
> >
> > The design looks reasonable to me. Waiting to see if others disagree before
> > looking more closely at the code.
>
> Thanks.

I would like to discuss behavioral and user interface considerations.

Upon further analysis of this patch regarding the conversion of
wal_level to a SIGHUP parameter, I find that supporting all
combinations of wal_level value changes might make less sense.
Specifically, changing to or from 'minimal' would necessitate a
checkpoint, and reducing wal_level to 'minimal' would require
terminating physical replication, WAL archiving, and online backups.
While these operations demand careful consideration, there seems to be
no compelling use case for decreasing to 'minimal'. Furthermore,
increasing wal_level from 'minimal' is typically a one-time operation
during a database's lifetime. Therefore, we should weigh the benefits
against the implementation complexity.

One solution is to manage the effective WAL level using two distinct
GUC parameters: max_wal_level and wal_level. max_wal_level would be a
POSTMASTER parameter controlling the system's maximum allowable WAL
level, with values 'minimal', 'replica', and 'logical'. wal_level
would function as a SIGHUP parameter managing the runtime WAL level,
accepting values 'replica', 'logical', and 'auto'. The selected value
must be either 'auto' or not exceed max_wal_level. When set to 'auto',
wal_level automatically synchronizes with max_wal_level's value. This
approach would enable online WAL level transitions between 'replica'
and 'logical'.

Regarding logical decoding on standbys, currently both primary and
standby servers must have wal_level set to 'logical'. We need to
determine the appropriate behavior when users decrease the WAL level
from 'logical' to 'replica' through configuration file reload.

One approach would be to invalidate all logical replication slots on
the standby when transitioning to 'replica' WAL level. Although
incoming WAL records from the primary would still be written at
'logical' level, making logical decoding technically feasible, this
behavior seems logical as it reflects the user's intent to discontinue
logical decoding on the standby. For consistency, we might need to
invalidate logical slots during server startup if the WAL level is
insufficient.

Alternatively, we could permit logical decoding on the standby even
with wal_level set to 'replica'. However, this would necessitate
invalidating all logical replication slots during promotion,
potentially extending downtime during failover.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Mon, Apr 21, 2025 at 11:01 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> I would like to discuss behavioral and user interface considerations.
>
> Upon further analysis of this patch regarding the conversion of
> wal_level to a SIGHUP parameter, I find that supporting all
> combinations of wal_level value changes might make less sense.
> Specifically, changing to or from 'minimal' would necessitate a
> checkpoint, and reducing wal_level to 'minimal' would require
> terminating physical replication, WAL archiving, and online backups.
> While these operations demand careful consideration, there seems to be
> no compelling use case for decreasing to 'minimal'. Furthermore,
> increasing wal_level from 'minimal' is typically a one-time operation
> during a database's lifetime. Therefore, we should weigh the benefits
> against the implementation complexity.
>
> One solution is to manage the effective WAL level using two distinct
> GUC parameters: max_wal_level and wal_level. max_wal_level would be a
> POSTMASTER parameter controlling the system's maximum allowable WAL
> level, with values 'minimal', 'replica', and 'logical'. wal_level
> would function as a SIGHUP parameter managing the runtime WAL level,
> accepting values 'replica', 'logical', and 'auto'. The selected value
> must be either 'auto' or not exceed max_wal_level. When set to 'auto',
> wal_level automatically synchronizes with max_wal_level's value. This
> approach would enable online WAL level transitions between 'replica'
> and 'logical'.
>
>
> Regarding logical decoding on standbys, currently both primary and
> standby servers must have wal_level set to 'logical'. We need to
> determine the appropriate behavior when users decrease the WAL level
> from 'logical' to 'replica' through configuration file reload.
>
> One approach would be to invalidate all logical replication slots on
> the standby when transitioning to 'replica' WAL level. Although
> incoming WAL records from the primary would still be written at
> 'logical' level, making logical decoding technically feasible, this
> behavior seems logical as it reflects the user's intent to discontinue
> logical decoding on the standby. For consistency, we might need to
> invalidate logical slots during server startup if the WAL level is
> insufficient.
>
> Alternatively, we could permit logical decoding on the standby even
> with wal_level set to 'replica'. However, this would necessitate
> invalidating all logical replication slots during promotion,
> potentially extending downtime during failover.
>

BTW, did we consider the idea to automatically transition to 'logical'
when the first logical slot is created and transition back to
'replica' when last logical slot gets dropped? I see some ideas around
this last time we discussed this topic.

[1] - https://www.postgresql.org/message-id/CAA4eK1J0we5qsZ-ZOwXPbZyvwdWbnT43knO2Cxidia2aHxZSJw%40mail.gmail.com

--
With Regards,
Amit Kapila.

On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Mon, Apr 21, 2025 at 11:01 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > I would like to discuss behavioral and user interface considerations.
> >
> > Upon further analysis of this patch regarding the conversion of
> > wal_level to a SIGHUP parameter, I find that supporting all
> > combinations of wal_level value changes might make less sense.
> > Specifically, changing to or from 'minimal' would necessitate a
> > checkpoint, and reducing wal_level to 'minimal' would require
> > terminating physical replication, WAL archiving, and online backups.
> > While these operations demand careful consideration, there seems to be
> > no compelling use case for decreasing to 'minimal'. Furthermore,
> > increasing wal_level from 'minimal' is typically a one-time operation
> > during a database's lifetime. Therefore, we should weigh the benefits
> > against the implementation complexity.
> >
> > One solution is to manage the effective WAL level using two distinct
> > GUC parameters: max_wal_level and wal_level. max_wal_level would be a
> > POSTMASTER parameter controlling the system's maximum allowable WAL
> > level, with values 'minimal', 'replica', and 'logical'. wal_level
> > would function as a SIGHUP parameter managing the runtime WAL level,
> > accepting values 'replica', 'logical', and 'auto'. The selected value
> > must be either 'auto' or not exceed max_wal_level. When set to 'auto',
> > wal_level automatically synchronizes with max_wal_level's value. This
> > approach would enable online WAL level transitions between 'replica'
> > and 'logical'.
> >
> >
> > Regarding logical decoding on standbys, currently both primary and
> > standby servers must have wal_level set to 'logical'. We need to
> > determine the appropriate behavior when users decrease the WAL level
> > from 'logical' to 'replica' through configuration file reload.
> >
> > One approach would be to invalidate all logical replication slots on
> > the standby when transitioning to 'replica' WAL level. Although
> > incoming WAL records from the primary would still be written at
> > 'logical' level, making logical decoding technically feasible, this
> > behavior seems logical as it reflects the user's intent to discontinue
> > logical decoding on the standby. For consistency, we might need to
> > invalidate logical slots during server startup if the WAL level is
> > insufficient.
> >
> > Alternatively, we could permit logical decoding on the standby even
> > with wal_level set to 'replica'. However, this would necessitate
> > invalidating all logical replication slots during promotion,
> > potentially extending downtime during failover.
> >
>
> BTW, did we consider the idea to automatically transition to 'logical'
> when the first logical slot is created and transition back to
> 'replica' when last logical slot gets dropped? I see some ideas around
> this last time we discussed this topic.

Yes. Bertrand pointed out that a drawback is that the primary server
needs to create a logical slot in order to execute logical decoding on
the standbys[1].

Regards,

[1] https://www.postgresql.org/message-id/Z5DCm6xiBfbUdvX7%40ip-10-97-1-34.eu-west-3.compute.internal

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, Apr 23, 2025 at 9:35 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > BTW, did we consider the idea to automatically transition to 'logical'
> > when the first logical slot is created and transition back to
> > 'replica' when last logical slot gets dropped? I see some ideas around
> > this last time we discussed this topic.
>
> Yes. Bertrand pointed out that a drawback is that the primary server
> needs to create a logical slot in order to execute logical decoding on
> the standbys[1].
>

True, but if we want to avoid that, we can still keep 'logical' as
wal_level for the ease of users. We can also have another API like the
one you originally proposed (pg_activate_logical_decoding) for the
ease of users. But the minimum requirement would be that one creates a
logical slot to enable logical decoding/replication.

Additionally, shall we do some benchmarking, if not done already, to
show the cases where the performance and WAL volume can hurt users if
we make wal_level as 'logical'?

--
With Regards,
Amit Kapila.

On Thu, Apr 24, 2025 at 5:30 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Wed, Apr 23, 2025 at 9:35 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > BTW, did we consider the idea to automatically transition to 'logical'
> > > when the first logical slot is created and transition back to
> > > 'replica' when last logical slot gets dropped? I see some ideas around
> > > this last time we discussed this topic.
> >
> > Yes. Bertrand pointed out that a drawback is that the primary server
> > needs to create a logical slot in order to execute logical decoding on
> > the standbys[1].
> >
>
> True, but if we want to avoid that, we can still keep 'logical' as
> wal_level for the ease of users.

I think we'd like to cover the use case like where users start with
'replica' on the primary and execute logical decoding on the standby
without neither creating a logical slot on the primary nor restarting
the primary.

> We can also have another API like the
> one you originally proposed (pg_activate_logical_decoding) for the
> ease of users. But the minimum requirement would be that one creates a
> logical slot to enable logical decoding/replication.

I think we want to avoid the runtime WAL level automatically decreased
to 'replica' once all logical slots are removed, if users still want
to execute logical decoding on only the standby. One idea is that if
users enable logical decoding using pg_activate_logical_decoding(),
the runtime WAL level doesn't decrease to 'replica' even if all
logical slots are removed. But it would require for us to remember how
the logical decoding has been enabled in a permanent way. Also, I'm
concerned that having three ways to enable logical decoding could
confuse users: wal_level GUC parameter, creating at least one logical
slot, and pg_activate_logical_decoding().

> Additionally, shall we do some benchmarking, if not done already, to
> show the cases where the performance and WAL volume can hurt users if
> we make wal_level as 'logical'?

I believe it would be significant especially for REPLICA IDENTITY FULL
tables. I agree it's worth benchmarking it but I guess the result
would not convince us to make 'logical' default.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi,

On Mon, Apr 21, 2025 at 10:31:03AM -0700, Masahiko Sawada wrote:
> I would like to discuss behavioral and user interface considerations.
>
> Upon further analysis of this patch regarding the conversion of
> wal_level to a SIGHUP parameter, I find that supporting all
> combinations of wal_level value changes might make less sense.
> Specifically, changing to or from 'minimal' would necessitate a
> checkpoint, and reducing wal_level to 'minimal' would require
> terminating physical replication, WAL archiving, and online backups.
> While these operations demand careful consideration, there seems to be
> no compelling use case for decreasing to 'minimal'. Furthermore,
> increasing wal_level from 'minimal' is typically a one-time operation
> during a database's lifetime. Therefore, we should weigh the benefits
> against the implementation complexity.

Agree.

> One solution is to manage the effective WAL level using two distinct
> GUC parameters: max_wal_level and wal_level. max_wal_level would be a
> POSTMASTER parameter controlling the system's maximum allowable WAL
> level, with values 'minimal', 'replica', and 'logical'. wal_level
> would function as a SIGHUP parameter managing the runtime WAL level,
> accepting values 'replica', 'logical', and 'auto'. The selected value
> must be either 'auto' or not exceed max_wal_level. When set to 'auto',
> wal_level automatically synchronizes with max_wal_level's value. This
> approach would enable online WAL level transitions between 'replica'
> and 'logical'.

That makes sense to me. I think that 'logical' could be the default value
for max_wal_level and 'replica' the default for wal_level.
I think that would provide almost the same user experience as currently and would
allow replica->logical change without restart. Thoughts?

> Regarding logical decoding on standbys, currently both primary and
> standby servers must have wal_level set to 'logical'. We need to
> determine the appropriate behavior when users decrease the WAL level
> from 'logical' to 'replica' through configuration file reload.
>
> One approach would be to invalidate all logical replication slots on
> the standby when transitioning to 'replica' WAL level. Although
> incoming WAL records from the primary would still be written at
> 'logical' level, making logical decoding technically feasible, this
> behavior seems logical as it reflects the user's intent to discontinue
> logical decoding on the standby.

+1

> For consistency, we might need to
> invalidate logical slots during server startup if the WAL level is
> insufficient.

Not sure. Currently we'd not allow the standby to start:

"
LOG: entering standby mode
FATAL: logical replication slot "logical_slot" exists, but "wal_level" < "logical"
HINT: Change "wal_level" to be "logical" or higher.
LOG: startup process (PID 1790508) exited with exit code 1
"

I think that's a good guard for configuration change mistakes. If that's a mistake
change back to logical and start. If that's not a mistake then change back to
logical, start, change with SIGHUP. OTOH I also see the benefits of being consistent
between SIGHUP and start.

> Alternatively, we could permit logical decoding on the standby even
> with wal_level set to 'replica'.

Yeah, technically speaking we could as the WALs are coming from the primary (that
has wal_level set to logical).

> However, this would necessitate
> invalidating all logical replication slots during promotion,
> potentially extending downtime during failover.

Yeah, I'm tempted to vote to not allow logical decoding on the standby if the
wal_level is not logical.

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Thu, Apr 24, 2025 at 11:14 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Thu, Apr 24, 2025 at 5:30 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Wed, Apr 23, 2025 at 9:35 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > >
> > > > BTW, did we consider the idea to automatically transition to 'logical'
> > > > when the first logical slot is created and transition back to
> > > > 'replica' when last logical slot gets dropped? I see some ideas around
> > > > this last time we discussed this topic.
> > >
> > > Yes. Bertrand pointed out that a drawback is that the primary server
> > > needs to create a logical slot in order to execute logical decoding on
> > > the standbys[1].
> > >
> >
> > True, but if we want to avoid that, we can still keep 'logical' as
> > wal_level for the ease of users.
>
> I think we'd like to cover the use case like where users start with
> 'replica' on the primary and execute logical decoding on the standby
> without neither creating a logical slot on the primary nor restarting
> the primary.
>

Okay, if we introduce a SIGHUP GUC like max_wal_level as you are
proposing, the above requirement will be fulfilled, right? The other
way is by API pg_activate_logical_decoding().

> > We can also have another API like the
> > one you originally proposed (pg_activate_logical_decoding) for the
> > ease of users. But the minimum requirement would be that one creates a
> > logical slot to enable logical decoding/replication.
>
> I think we want to avoid the runtime WAL level automatically decreased
> to 'replica' once all logical slots are removed, if users still want
> to execute logical decoding on only the standby. One idea is that if
> users enable logical decoding using pg_activate_logical_decoding(),
> the runtime WAL level doesn't decrease to 'replica' even if all
> logical slots are removed.
>

That makes sense. If we are using an API like
pg_activate_*/pg_deactivate_*, then why add an additional dependency
on the slots?

--
With Regards,
Amit Kapila.

On Tue, May 6, 2025 at 12:19 AM Bertrand Drouvot
<bertranddrouvot(dot)pg(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Mon, Apr 21, 2025 at 10:31:03AM -0700, Masahiko Sawada wrote:
> > I would like to discuss behavioral and user interface considerations.
> >
> > Upon further analysis of this patch regarding the conversion of
> > wal_level to a SIGHUP parameter, I find that supporting all
> > combinations of wal_level value changes might make less sense.
> > Specifically, changing to or from 'minimal' would necessitate a
> > checkpoint, and reducing wal_level to 'minimal' would require
> > terminating physical replication, WAL archiving, and online backups.
> > While these operations demand careful consideration, there seems to be
> > no compelling use case for decreasing to 'minimal'. Furthermore,
> > increasing wal_level from 'minimal' is typically a one-time operation
> > during a database's lifetime. Therefore, we should weigh the benefits
> > against the implementation complexity.
>
> Agree.
>
> > One solution is to manage the effective WAL level using two distinct
> > GUC parameters: max_wal_level and wal_level. max_wal_level would be a
> > POSTMASTER parameter controlling the system's maximum allowable WAL
> > level, with values 'minimal', 'replica', and 'logical'. wal_level
> > would function as a SIGHUP parameter managing the runtime WAL level,
> > accepting values 'replica', 'logical', and 'auto'. The selected value
> > must be either 'auto' or not exceed max_wal_level. When set to 'auto',
> > wal_level automatically synchronizes with max_wal_level's value. This
> > approach would enable online WAL level transitions between 'replica'
> > and 'logical'.
>
> That makes sense to me. I think that 'logical' could be the default value
> for max_wal_level and 'replica' the default for wal_level.
> I think that would provide almost the same user experience as currently and would
> allow replica->logical change without restart. Thoughts?

Sounds reasonable default values. One thing we might want to note is
that when users want to set 'minimal', they would need to change both
parameters. With the defaults value wal_level='auto', users would need
to simply set max_wal_level='minimal'. However, given that it's more
common for users to start with 'replica' than 'minimal', probably it
would make sense to have the default values you suggested. This
combination of WAL levels maximize the benefit of this idea while
maintaining the default behavior.

One downside of this idea is that we introduce another GUC parameter,
resulting in that users would need to control WAL level using two
parameters, which might not be quite clear for users.

>
> > Regarding logical decoding on standbys, currently both primary and
> > standby servers must have wal_level set to 'logical'. We need to
> > determine the appropriate behavior when users decrease the WAL level
> > from 'logical' to 'replica' through configuration file reload.
> >
> > One approach would be to invalidate all logical replication slots on
> > the standby when transitioning to 'replica' WAL level. Although
> > incoming WAL records from the primary would still be written at
> > 'logical' level, making logical decoding technically feasible, this
> > behavior seems logical as it reflects the user's intent to discontinue
> > logical decoding on the standby.
>
> +1
>
> > For consistency, we might need to
> > invalidate logical slots during server startup if the WAL level is
> > insufficient.
>
> Not sure. Currently we'd not allow the standby to start:
>
> "
> LOG: entering standby mode
> FATAL: logical replication slot "logical_slot" exists, but "wal_level" < "logical"
> HINT: Change "wal_level" to be "logical" or higher.
> LOG: startup process (PID 1790508) exited with exit code 1
> "
>
> I think that's a good guard for configuration change mistakes. If that's a mistake
> change back to logical and start. If that's not a mistake then change back to
> logical, start, change with SIGHUP. OTOH I also see the benefits of being consistent
> between SIGHUP and start.

I see your point.

An alternative idea would be to somehow prevent wal_level from being
changed to 'replica' while there is at least one active logical slot
on the standby. I'll consider this idea too.

Regards,

--
Masahiko Sawada

Amazon Web Services: https://aws.amazon.com

On Tue, May 6, 2025 at 11:59 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Thu, Apr 24, 2025 at 11:14 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Thu, Apr 24, 2025 at 5:30 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > On Wed, Apr 23, 2025 at 9:35 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > >
> > > > > BTW, did we consider the idea to automatically transition to 'logical'
> > > > > when the first logical slot is created and transition back to
> > > > > 'replica' when last logical slot gets dropped? I see some ideas around
> > > > > this last time we discussed this topic.
> > > >
> > > > Yes. Bertrand pointed out that a drawback is that the primary server
> > > > needs to create a logical slot in order to execute logical decoding on
> > > > the standbys[1].
> > > >
> > >
> > > True, but if we want to avoid that, we can still keep 'logical' as
> > > wal_level for the ease of users.
> >
> > I think we'd like to cover the use case like where users start with
> > 'replica' on the primary and execute logical decoding on the standby
> > without neither creating a logical slot on the primary nor restarting
> > the primary.
> >
>
> Okay, if we introduce a SIGHUP GUC like max_wal_level as you are
> proposing, the above requirement will be fulfilled, right?

Right. Both the primary and the standby can increase WAL level to
'logical' without server restart nor creating a logical slot.

> The other
> way is by API pg_activate_logical_decoding().

Yes. This approach would be simpler than the current proposal as we
don't need other new infrastructure such as executing a task in the
background. However, we might want to note that wal_level value would
no longer show the actual runtime WAL level if the logical decoding is
activated via this API. Probably it's better to introduce a read-only
GUC, say runtime_wal_level, showing the actual WAL level. Also,
Ashutosh pointed out[1] before that cloud providers do not like
multiple ways of changing configuration esp. when they can not control
it. But I'm not sure this applies to the API as it's a SQL function
whose access privilege can be controlled.

>
> > > We can also have another API like the
> > > one you originally proposed (pg_activate_logical_decoding) for the
> > > ease of users. But the minimum requirement would be that one creates a
> > > logical slot to enable logical decoding/replication.
> >
> > I think we want to avoid the runtime WAL level automatically decreased
> > to 'replica' once all logical slots are removed, if users still want
> > to execute logical decoding on only the standby. One idea is that if
> > users enable logical decoding using pg_activate_logical_decoding(),
> > the runtime WAL level doesn't decrease to 'replica' even if all
> > logical slots are removed.
> >
>
> That makes sense. If we are using an API like
> pg_activate_*/pg_deactivate_*, then why add an additional dependency
> on the slots?

I thought that we need to remember how logical decoding got enabled
because otherwise even if we enable logical decoding using the API,
it's disabled to 'replica' if all logical slots get removed. So the
idea I mentioned above is that we somehow prevent logical decoding
from being disabled even if all logical slots are removed. If we're
using only these APIs to enable/disable logical decoding, we don't
need to add a dependency on the slots, although we probably want to
disallow disabling logical decoding if there is at least one active
logical slot.

Regards,

[1] https://www.postgresql.org/message-id/CAExHW5tyJrdjqKFQ%2BqDs8Yq3E_P1Fj_T4pwVW9WACmMznRtDuw%40mail.gmail.com

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, May 8, 2025 at 1:06 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Tue, May 6, 2025 at 11:59 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Thu, Apr 24, 2025 at 11:14 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Thu, Apr 24, 2025 at 5:30 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > >
> > > > On Wed, Apr 23, 2025 at 9:35 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > > >
> > > > > On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > > >
> > > > > > BTW, did we consider the idea to automatically transition to 'logical'
> > > > > > when the first logical slot is created and transition back to
> > > > > > 'replica' when last logical slot gets dropped? I see some ideas around
> > > > > > this last time we discussed this topic.
> > > > >
> > > > > Yes. Bertrand pointed out that a drawback is that the primary server
> > > > > needs to create a logical slot in order to execute logical decoding on
> > > > > the standbys[1].
> > > > >
> > > >
> > > > True, but if we want to avoid that, we can still keep 'logical' as
> > > > wal_level for the ease of users.
> > >
> > > I think we'd like to cover the use case like where users start with
> > > 'replica' on the primary and execute logical decoding on the standby
> > > without neither creating a logical slot on the primary nor restarting
> > > the primary.
> > >
> >
> > Okay, if we introduce a SIGHUP GUC like max_wal_level as you are
> > proposing, the above requirement will be fulfilled, right?
>
> Right. Both the primary and the standby can increase WAL level to
> 'logical' without server restart nor creating a logical slot.
>
> > The other
> > way is by API pg_activate_logical_decoding().
>
> Yes. This approach would be simpler than the current proposal as we
> don't need other new infrastructure such as executing a task in the
> background.
>

Right, but to an extent, this is also similar to having a requirement
of a logical slot on the primary. Now, it seems to me that the point
you are trying to make is that to allow logical decoding on standby,
it is okay to ask users to use pg_activate_logical_decoding() on
primary, but it would be inconvenient to ask them to have a logical
slot on primary instead. If my understanding is correct, then why do
you think so? We recommend that users have a physical slot on primary
and use it via primary_slot_name on standby to control resource
removal, so why can't we ask them to have a logical slot on primary to
allow logical decoding on standby?

> However, we might want to note that wal_level value would
> no longer show the actual runtime WAL level if the logical decoding is
> activated via this API. Probably it's better to introduce a read-only
> GUC, say runtime_wal_level, showing the actual WAL level.
>

Yeah, we need some way to show the correct value. In one of the
previous emails on this thread, you mentioned that we can use
show_hook to show the correct value. I see that show_in_hot_standby()
uses in_memory value to show the correct value. Do you have something
like that in your mind?

BTW, what is your idea to preserve the state to allow logical decoding
across server restart when the user uses the API, do we want to
persist the state in some way, if so, how? OTOH, if we use the idea to
have a logical slot to allow decoding, then the presence of a logical
slot can tell us whether we need to enable the new state to allow
logical decoding after restart.

> Also,
> Ashutosh pointed out[1] before that cloud providers do not like
> multiple ways of changing configuration esp. when they can not control
> it. But I'm not sure this applies to the API as it's a SQL function
> whose access privilege can be controlled.
>

By multiple ways, do we mean to say that one way for users would be to
use the existing way (change wal_level to logical and restart server),
and the other way would be to use the new API (or have a logical
slot)? But won't similarly users have multiple ways to retain WAL for
standby servers (either by using wal_keep_size or by having a
primary_slot_name). The other example is that one can either manually
change postgresql.conf file or use ALTER SYSTEM to change it, and then
reloadthe config or restart the server for the change to take effect.
There could be other similar examples as well if one tries to list all
such possibilities. I feel one should be concerned if we are trying to
make both wal_level GUC as SIGHUP, and also try to provide an API to
enable logical decoding.

> > >
> >
> > That makes sense. If we are using an API like
> > pg_activate_*/pg_deactivate_*, then why add an additional dependency
> > on the slots?
>
> I thought that we need to remember how logical decoding got enabled
> because otherwise even if we enable logical decoding using the API,
> it's disabled to 'replica' if all logical slots get removed. So the
> idea I mentioned above is that we somehow prevent logical decoding
> from being disabled even if all logical slots are removed. If we're
> using only these APIs to enable/disable logical decoding, we don't
> need to add a dependency on the slots, although we probably want to
> disallow disabling logical decoding if there is at least one active
> logical slot.
>

Yeah, this is a detail that should be discussed once we finalize the
API to enable logical decoding on both primary and standby without
restarting the primary server.

--
With Regards,
Amit Kapila.

On Sat, May 10, 2025 at 12:00 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Thu, May 8, 2025 at 1:06 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Tue, May 6, 2025 at 11:59 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > On Thu, Apr 24, 2025 at 11:14 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > On Thu, Apr 24, 2025 at 5:30 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > >
> > > > > On Wed, Apr 23, 2025 at 9:35 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > > > >
> > > > > > On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > > > >
> > > > > > > BTW, did we consider the idea to automatically transition to 'logical'
> > > > > > > when the first logical slot is created and transition back to
> > > > > > > 'replica' when last logical slot gets dropped? I see some ideas around
> > > > > > > this last time we discussed this topic.
> > > > > >
> > > > > > Yes. Bertrand pointed out that a drawback is that the primary server
> > > > > > needs to create a logical slot in order to execute logical decoding on
> > > > > > the standbys[1].
> > > > > >
> > > > >
> > > > > True, but if we want to avoid that, we can still keep 'logical' as
> > > > > wal_level for the ease of users.
> > > >
> > > > I think we'd like to cover the use case like where users start with
> > > > 'replica' on the primary and execute logical decoding on the standby
> > > > without neither creating a logical slot on the primary nor restarting
> > > > the primary.
> > > >
> > >
> > > Okay, if we introduce a SIGHUP GUC like max_wal_level as you are
> > > proposing, the above requirement will be fulfilled, right?
> >
> > Right. Both the primary and the standby can increase WAL level to
> > 'logical' without server restart nor creating a logical slot.
> >
> > > The other
> > > way is by API pg_activate_logical_decoding().
> >
> > Yes. This approach would be simpler than the current proposal as we
> > don't need other new infrastructure such as executing a task in the
> > background.
> >
>
> Right, but to an extent, this is also similar to having a requirement
> of a logical slot on the primary. Now, it seems to me that the point
> you are trying to make is that to allow logical decoding on standby,
> it is okay to ask users to use pg_activate_logical_decoding() on
> primary, but it would be inconvenient to ask them to have a logical
> slot on primary instead. If my understanding is correct, then why do
> you think so? We recommend that users have a physical slot on primary
> and use it via primary_slot_name on standby to control resource
> removal, so why can't we ask them to have a logical slot on primary to
> allow logical decoding on standby?

I was thinking of a simple use case where users do logical decoding
from the physical standby. That is, the primary has a physical slot
and the standby uses it via primary_slot_name, and the subscriber
connects the standby server for logical replication with a logical
slot on the standby. In this case, IIUC we need to require users to
create a logical slot on the primary in order just to increase WAL
level to 'logical', but it doesn't make sense to me. No one is going
to use this logical slot and the primary ends up accumulating WALs.

> > However, we might want to note that wal_level value would
> > no longer show the actual runtime WAL level if the logical decoding is
> > activated via this API. Probably it's better to introduce a read-only
> > GUC, say runtime_wal_level, showing the actual WAL level.
> >
>
> Yeah, we need some way to show the correct value. In one of the
> previous emails on this thread, you mentioned that we can use
> show_hook to show the correct value. I see that show_in_hot_standby()
> uses in_memory value to show the correct value. Do you have something
> like that in your mind?

Yes.

> BTW, what is your idea to preserve the state to allow logical decoding
> across server restart when the user uses the API, do we want to
> persist the state in some way, if so, how? OTOH, if we use the idea to
> have a logical slot to allow decoding, then the presence of a logical
> slot can tell us whether we need to enable the new state to allow
> logical decoding after restart.

I vaguely thought of storing such information in the control file or a
checkpoint record along with wal_level value. But I've not seriously
considered how to implement this idea as I don't think it's not a good
user interface.

>
> > Also,
> > Ashutosh pointed out[1] before that cloud providers do not like
> > multiple ways of changing configuration esp. when they can not control
> > it. But I'm not sure this applies to the API as it's a SQL function
> > whose access privilege can be controlled.
> >
>
> By multiple ways, do we mean to say that one way for users would be to
> use the existing way (change wal_level to logical and restart server),
> and the other way would be to use the new API (or have a logical
> slot)?

Yes, that's my understanding of his comment.

> But won't similarly users have multiple ways to retain WAL for
> standby servers (either by using wal_keep_size or by having a
> primary_slot_name). The other example is that one can either manually
> change postgresql.conf file or use ALTER SYSTEM to change it, and then
> reloadthe config or restart the server for the change to take effect.
> There could be other similar examples as well if one tries to list all
> such possibilities.

True. I think allow_alter_system was introduced for users who don't
want to let their end users change the configuration via ALTER SYSTEM
command. Since the new API we're considering is an SQL function we
already have a way to control its access for such users.

> I feel one should be concerned if we are trying to
> make both wal_level GUC as SIGHUP, and also try to provide an API to
> enable logical decoding.

Agreed.

>
> > > >
> > >
> > > That makes sense. If we are using an API like
> > > pg_activate_*/pg_deactivate_*, then why add an additional dependency
> > > on the slots?
> >
> > I thought that we need to remember how logical decoding got enabled
> > because otherwise even if we enable logical decoding using the API,
> > it's disabled to 'replica' if all logical slots get removed. So the
> > idea I mentioned above is that we somehow prevent logical decoding
> > from being disabled even if all logical slots are removed. If we're
> > using only these APIs to enable/disable logical decoding, we don't
> > need to add a dependency on the slots, although we probably want to
> > disallow disabling logical decoding if there is at least one active
> > logical slot.
> >
>
> Yeah, this is a detail that should be discussed once we finalize the
> API to enable logical decoding on both primary and standby without
> restarting the primary server.

Agreed.

Another approach we might need to consider is to convert wal_level to
a SIGHUP parameter. While I mentioned that supporting all combinations
of wal_level value changes might make less sense for its complexity, I
think this is the most straightforward approach and interface. So it
might be worth trying to implement this approach to figure out the
actual complexity.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Sat, May 10, 2025 at 1:05 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Sat, May 10, 2025 at 12:00 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> >
> > Right, but to an extent, this is also similar to having a requirement
> > of a logical slot on the primary. Now, it seems to me that the point
> > you are trying to make is that to allow logical decoding on standby,
> > it is okay to ask users to use pg_activate_logical_decoding() on
> > primary, but it would be inconvenient to ask them to have a logical
> > slot on primary instead. If my understanding is correct, then why do
> > you think so? We recommend that users have a physical slot on primary
> > and use it via primary_slot_name on standby to control resource
> > removal, so why can't we ask them to have a logical slot on primary to
> > allow logical decoding on standby?
>
> I was thinking of a simple use case where users do logical decoding
> from the physical standby. That is, the primary has a physical slot
> and the standby uses it via primary_slot_name, and the subscriber
> connects the standby server for logical replication with a logical
> slot on the standby. In this case, IIUC we need to require users to
> create a logical slot on the primary in order just to increase WAL
> level to 'logical', but it doesn't make sense to me. No one is going
> to use this logical slot and the primary ends up accumulating WALs.
>

Can we have a parameter like immediately_reserve in
create_logical_slot API, similar to what we have for physical slots?
We need to work out the details, but that should address the kind of
use case you are worried about, unless I am missing something.

--
With Regards,
Amit Kapila.

On Sat, May 10, 2025 at 7:08 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Sat, May 10, 2025 at 1:05 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Sat, May 10, 2025 at 12:00 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > >
> > > Right, but to an extent, this is also similar to having a requirement
> > > of a logical slot on the primary. Now, it seems to me that the point
> > > you are trying to make is that to allow logical decoding on standby,
> > > it is okay to ask users to use pg_activate_logical_decoding() on
> > > primary, but it would be inconvenient to ask them to have a logical
> > > slot on primary instead. If my understanding is correct, then why do
> > > you think so? We recommend that users have a physical slot on primary
> > > and use it via primary_slot_name on standby to control resource
> > > removal, so why can't we ask them to have a logical slot on primary to
> > > allow logical decoding on standby?
> >
> > I was thinking of a simple use case where users do logical decoding
> > from the physical standby. That is, the primary has a physical slot
> > and the standby uses it via primary_slot_name, and the subscriber
> > connects the standby server for logical replication with a logical
> > slot on the standby. In this case, IIUC we need to require users to
> > create a logical slot on the primary in order just to increase WAL
> > level to 'logical', but it doesn't make sense to me. No one is going
> > to use this logical slot and the primary ends up accumulating WALs.
> >
>
> Can we have a parameter like immediately_reserve in
> create_logical_slot API, similar to what we have for physical slots?
> We need to work out the details, but that should address the kind of
> use case you are worried about, unless I am missing something.

Interesting idea. One concern in my mind is that in the use case I
mentioned above, users would need to carefully manage the extra
logical slot to keep the logical decoding active. The logical decoding
is deactivated on the standby as soon as users drop all logical slots
on the primary.

Also, with this idea of automatically increasing WAL level, do we want
to keep the 'logical' WAL level? If so, it requires an extra step of
creating a non-reserved logical slot on the primary in order for the
standby to activate the logical decoding. On the other hand, we can
also keep the 'logical' WAL level for the compatibility and for making
the logical decoding enabled without the coordination of WAL level
transition. But wal_level GUC parameter would no longer tell the
actual WAL level to users when 'replica' + logical slots. Is it
sufficient to provide a read-only GUC parameter, say
effective_wal_level showing the actual WAL level being used?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Sun, May 18, 2025 at 1:09 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Sat, May 10, 2025 at 7:08 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Sat, May 10, 2025 at 1:05 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Sat, May 10, 2025 at 12:00 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > >
> > > >
> > > > Right, but to an extent, this is also similar to having a requirement
> > > > of a logical slot on the primary. Now, it seems to me that the point
> > > > you are trying to make is that to allow logical decoding on standby,
> > > > it is okay to ask users to use pg_activate_logical_decoding() on
> > > > primary, but it would be inconvenient to ask them to have a logical
> > > > slot on primary instead. If my understanding is correct, then why do
> > > > you think so? We recommend that users have a physical slot on primary
> > > > and use it via primary_slot_name on standby to control resource
> > > > removal, so why can't we ask them to have a logical slot on primary to
> > > > allow logical decoding on standby?
> > >
> > > I was thinking of a simple use case where users do logical decoding
> > > from the physical standby. That is, the primary has a physical slot
> > > and the standby uses it via primary_slot_name, and the subscriber
> > > connects the standby server for logical replication with a logical
> > > slot on the standby. In this case, IIUC we need to require users to
> > > create a logical slot on the primary in order just to increase WAL
> > > level to 'logical', but it doesn't make sense to me. No one is going
> > > to use this logical slot and the primary ends up accumulating WALs.
> > >
> >
> > Can we have a parameter like immediately_reserve in
> > create_logical_slot API, similar to what we have for physical slots?
> > We need to work out the details, but that should address the kind of
> > use case you are worried about, unless I am missing something.
>
> Interesting idea. One concern in my mind is that in the use case I
> mentioned above, users would need to carefully manage the extra
> logical slot to keep the logical decoding active. The logical decoding
> is deactivated on the standby as soon as users drop all logical slots
> on the primary.
>
> Also, with this idea of automatically increasing WAL level, do we want
> to keep the 'logical' WAL level? If so, it requires an extra step of
> creating a non-reserved logical slot on the primary in order for the
> standby to activate the logical decoding. On the other hand, we can
> also keep the 'logical' WAL level for the compatibility and for making
> the logical decoding enabled without the coordination of WAL level
> transition. But wal_level GUC parameter would no longer tell the
> actual WAL level to users when 'replica' + logical slots. Is it
> sufficient to provide a read-only GUC parameter, say
> effective_wal_level showing the actual WAL level being used?
>

Thanks for proposing the idea of making wal_level configurable at
runtime. But why isn't making the relevant GUCs SIGHUP-reloadable
sufficient?

For enabling logical replication, users are already familiar with the
wal_level and max_wal_senders settings. The main issue is that
changing them currently requires a server restart. If we can address
that by making the GUCs reloadable via SIGHUP, that might be enough.

On the other hand, if the goal is to make the behavior fully dynamic,
then we should go all the way, decouple it from wal_level. For
example, we could start logging the extra WAL needed for logical
decoding as soon as a logical slot is created, and stop once all
logical slots are dropped, even if wal_level is still set to logical.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Sun, May 18, 2025 at 1:09 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Sat, May 10, 2025 at 7:08 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> >
> > Can we have a parameter like immediately_reserve in
> > create_logical_slot API, similar to what we have for physical slots?
> > We need to work out the details, but that should address the kind of
> > use case you are worried about, unless I am missing something.
>
> Interesting idea. One concern in my mind is that in the use case I
> mentioned above, users would need to carefully manage the extra
> logical slot to keep the logical decoding active. The logical decoding
> is deactivated on the standby as soon as users drop all logical slots
> on the primary.
>

Yes, but the same is true for a physical slot in the case of physical
replication used via primary_slot_name parameter.

> Also, with this idea of automatically increasing WAL level, do we want
> to keep the 'logical' WAL level? If so, it requires an extra step of
> creating a non-reserved logical slot on the primary in order for the
> standby to activate the logical decoding. On the other hand, we can
> also keep the 'logical' WAL level for the compatibility and for making
> the logical decoding enabled without the coordination of WAL level
> transition.

Right, I also feel we should retain both ways to enable logical
replication at least initially. Once we get some feedback, we may
think of removing 'logical' as wal_level.

> But wal_level GUC parameter would no longer tell the
> actual WAL level to users when 'replica' + logical slots.
>

Right.

> Is it
> sufficient to provide a read-only GUC parameter, say
> effective_wal_level showing the actual WAL level being used?
>

I am not so sure about how we want to communicate this to the user,
but I guess to start with, this is a good idea.

--
With Regards,
Amit Kapila.

On Sun, May 18, 2025 at 4:06 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> Thanks for proposing the idea of making wal_level configurable at
> runtime. But why isn't making the relevant GUCs SIGHUP-reloadable
> sufficient?
>
> For enabling logical replication, users are already familiar with the
> wal_level and max_wal_senders settings. The main issue is that
> changing them currently requires a server restart. If we can address
> that by making the GUCs reloadable via SIGHUP, that might be enough.
>

Sure, but the challenges are huge. Consider cases like one wants to
change wal_level from 'logical' to 'minimal'. See some analysis of the
same in email [1]. I think it will be a much bigger and challenging
project with diminishing returns as compared to the alternative we are
discussing.

> On the other hand, if the goal is to make the behavior fully dynamic,
> then we should go all the way, decouple it from wal_level. For
> example, we could start logging the extra WAL needed for logical
> decoding as soon as a logical slot is created, and stop once all
> logical slots are dropped, even if wal_level is still set to logical.
>

Yeah, this is one thing that is still under consideration. It is
almost equivalent to removing wal_level as 'logical', which sounds
like a compatibility break, and even if we want to do that, it is
better to attempt that after the base version is committed and we get
a broader consensus on the same.

[1]- https://www.postgresql.org/message-id/CAD21AoAA%3DzuiajwXgXSCYQWo%3D6oY-%3DCGLaEqvpfNUTVsLen%2BCA%40mail.gmail.com

--
With Regards,
Amit Kapila.

On Mon, May 19, 2025 at 2:05 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Sun, May 18, 2025 at 1:09 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Sat, May 10, 2025 at 7:08 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > >
> > > Can we have a parameter like immediately_reserve in
> > > create_logical_slot API, similar to what we have for physical slots?
> > > We need to work out the details, but that should address the kind of
> > > use case you are worried about, unless I am missing something.
> >
> > Interesting idea. One concern in my mind is that in the use case I
> > mentioned above, users would need to carefully manage the extra
> > logical slot to keep the logical decoding active. The logical decoding
> > is deactivated on the standby as soon as users drop all logical slots
> > on the primary.
> >
>
> Yes, but the same is true for a physical slot in the case of physical
> replication used via primary_slot_name parameter.

Could you elaborate on this? IIUC the purpose of using a physical slot
in a physical replication case is obvious; users don't want to lose
WAL files necessary for replication. On the other hand, this empty
logical slot needs to be maintained just for keeping the logical
decoding active.

>
> > Also, with this idea of automatically increasing WAL level, do we want
> > to keep the 'logical' WAL level? If so, it requires an extra step of
> > creating a non-reserved logical slot on the primary in order for the
> > standby to activate the logical decoding. On the other hand, we can
> > also keep the 'logical' WAL level for the compatibility and for making
> > the logical decoding enabled without the coordination of WAL level
> > transition.
>
> Right, I also feel we should retain both ways to enable logical
> replication at least initially. Once we get some feedback, we may
> think of removing 'logical' as wal_level.
>
> > But wal_level GUC parameter would no longer tell the
> > actual WAL level to users when 'replica' + logical slots.
> >
>
> Right.
>
> > Is it
> > sufficient to provide a read-only GUC parameter, say
> > effective_wal_level showing the actual WAL level being used?
> >
>
> I am not so sure about how we want to communicate this to the user,
> but I guess to start with, this is a good idea.

I recently had a discussion with Ashtosh at PGConf.dev regarding an
alternative approach: introducing a new command syntax such as "ALTER
SYSTEM UPDATE wal_level TO 'logical'". In his presentation[1], he
outlined this proposed command as a means to modify specific GUC
parameters synchronously. The backend executing this command would
manage the transition, allowing users to interrupt the process via
Ctrl-C if necessary. In the specific context of wal_level change, this
command could be designed to reject operations like "ALTER SYSTEM
UPDATE wal_level TO 'minimal'" with an error, effectively preventing
undesirable wal_level transitions to or from 'minimal'. While this
approach shares similarities with our previous proposal of
implementing a dedicated SQL function for WAL level modifications, it
offers a more standardized interface for users.

Though I find merit in this proposal, I remain uncertain about its
implementation details and whether it represents the optimal solution
for online wal_level changes, particularly given that our current
approach of automatic WAL level adjustment appears viable. Ashtosh
plans to initiate a separate discussion thread where we can explore
these considerations in greater detail.

Regards,

[1] https://www.pgevents.ca/events/pgconfdev2025/schedule/session/286-changing-shared_buffers-on-the-fly/

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, May 21, 2025 at 12:45 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Mon, May 19, 2025 at 2:05 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Sun, May 18, 2025 at 1:09 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Sat, May 10, 2025 at 7:08 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > >
> > > >
> > > > Can we have a parameter like immediately_reserve in
> > > > create_logical_slot API, similar to what we have for physical slots?
> > > > We need to work out the details, but that should address the kind of
> > > > use case you are worried about, unless I am missing something.
> > >
> > > Interesting idea. One concern in my mind is that in the use case I
> > > mentioned above, users would need to carefully manage the extra
> > > logical slot to keep the logical decoding active. The logical decoding
> > > is deactivated on the standby as soon as users drop all logical slots
> > > on the primary.
> > >
> >
> > Yes, but the same is true for a physical slot in the case of physical
> > replication used via primary_slot_name parameter.
>
> Could you elaborate on this?
>

I am trying to correlate with the case where standby no longer needs
physical slot due to some reason like the standby machine failure, or
say someone uses pg_createsubscriber on standby to make it subscriber,
etc. In such a case, user needs to manually remove the physical slot
on primary. There is difference in both cases but the point is one may
need to manage physical slot as well.

>
> I recently had a discussion with Ashtosh at PGConf.dev regarding an
> alternative approach: introducing a new command syntax such as "ALTER
> SYSTEM UPDATE wal_level TO 'logical'". In his presentation[1], he
> outlined this proposed command as a means to modify specific GUC
> parameters synchronously. The backend executing this command would
> manage the transition, allowing users to interrupt the process via
> Ctrl-C if necessary. In the specific context of wal_level change, this
> command could be designed to reject operations like "ALTER SYSTEM
> UPDATE wal_level TO 'minimal'" with an error, effectively preventing
> undesirable wal_level transitions to or from 'minimal'. While this
> approach shares similarities with our previous proposal of
> implementing a dedicated SQL function for WAL level modifications, it
> offers a more standardized interface for users.
>
> Though I find merit in this proposal, I remain uncertain about its
> implementation details and whether it represents the optimal solution
> for online wal_level changes, particularly given that our current
> approach of automatic WAL level adjustment appears viable.
>

Yeah, I find the idea that the presence of a logical slot will allow
the user to enable logical decoding/replication more appealing than
this new alternative, leaving aside the challenges of realizing it.

--
With Regards,
Amit Kapila.

On Tue, May 20, 2025 at 9:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Wed, May 21, 2025 at 12:45 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Mon, May 19, 2025 at 2:05 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > On Sun, May 18, 2025 at 1:09 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > On Sat, May 10, 2025 at 7:08 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > >
> > > > >
> > > > > Can we have a parameter like immediately_reserve in
> > > > > create_logical_slot API, similar to what we have for physical slots?
> > > > > We need to work out the details, but that should address the kind of
> > > > > use case you are worried about, unless I am missing something.
> > > >
> > > > Interesting idea. One concern in my mind is that in the use case I
> > > > mentioned above, users would need to carefully manage the extra
> > > > logical slot to keep the logical decoding active. The logical decoding
> > > > is deactivated on the standby as soon as users drop all logical slots
> > > > on the primary.
> > > >
> > >
> > > Yes, but the same is true for a physical slot in the case of physical
> > > replication used via primary_slot_name parameter.
> >
> > Could you elaborate on this?
> >
>
> I am trying to correlate with the case where standby no longer needs
> physical slot due to some reason like the standby machine failure, or
> say someone uses pg_createsubscriber on standby to make it subscriber,
> etc. In such a case, user needs to manually remove the physical slot
> on primary. There is difference in both cases but the point is one may
> need to manage physical slot as well.

Thank you for clarifying this. I see your point.

> >
> > I recently had a discussion with Ashtosh at PGConf.dev regarding an
> > alternative approach: introducing a new command syntax such as "ALTER
> > SYSTEM UPDATE wal_level TO 'logical'". In his presentation[1], he
> > outlined this proposed command as a means to modify specific GUC
> > parameters synchronously. The backend executing this command would
> > manage the transition, allowing users to interrupt the process via
> > Ctrl-C if necessary. In the specific context of wal_level change, this
> > command could be designed to reject operations like "ALTER SYSTEM
> > UPDATE wal_level TO 'minimal'" with an error, effectively preventing
> > undesirable wal_level transitions to or from 'minimal'. While this
> > approach shares similarities with our previous proposal of
> > implementing a dedicated SQL function for WAL level modifications, it
> > offers a more standardized interface for users.
> >
> > Though I find merit in this proposal, I remain uncertain about its
> > implementation details and whether it represents the optimal solution
> > for online wal_level changes, particularly given that our current
> > approach of automatic WAL level adjustment appears viable.
> >
>
> Yeah, I find the idea that the presence of a logical slot will allow
> the user to enable logical decoding/replication more appealing than
> this new alternative, leaving aside the challenges of realizing it.

I've drafted this idea. Here are summary for attached two patches:

0001 patch allows us to create a logical slot without WAL reservation.

0002 patch is the main patch for dynamically enabling/disabling
logical decoding when wal_level is 'replica'. It's in PoC state and
has a lot of XXX comments. One thing I think we need to consider is
that since disabling the logical decoding needs to write a WAL record
for standbys and happens when dropping the last logical slot which
needs to write a WAL record for standbys, it's possible that we write
a WAL record in a process shutdown during the process exit (e.g.,
ReplicationSlotRelease() and ReplicationSlotCleanup() are called by
ReplicationSlotShmemExit()). It might be safe as long as we do that
during calling before_shmem_exit callback but I'm not sure there is a
chance to do that during calling on_shmem_exit callbacks. It would be
better to somehow lazily disable the logical decoding.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, Jun 4, 2025 at 6:41 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Tue, May 20, 2025 at 9:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > Yeah, I find the idea that the presence of a logical slot will allow
> > the user to enable logical decoding/replication more appealing than
> > this new alternative, leaving aside the challenges of realizing it.

+1. This idea appears more user-friendly and easier to understand
compared to other approaches, such as having multiple GUCs or using
ALTER SYSTEM.

> I've drafted this idea. Here are summary for attached two patches:
>
> 0001 patch allows us to create a logical slot without WAL reservation.
>
> 0002 patch is the main patch for dynamically enabling/disabling
> logical decoding when wal_level is 'replica'.

Thank You for the patches. I have done some initial testing, it seems
to be working well. I will do more testing and review and will share
further feedback.

thanks
Shveta

On Wed, Jun 4, 2025 at 3:40 PM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
>
> On Wed, Jun 4, 2025 at 6:41 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Tue, May 20, 2025 at 9:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > Yeah, I find the idea that the presence of a logical slot will allow
> > > the user to enable logical decoding/replication more appealing than
> > > this new alternative, leaving aside the challenges of realizing it.
>
> +1. This idea appears more user-friendly and easier to understand
> compared to other approaches, such as having multiple GUCs or using
> ALTER SYSTEM.
>
> > I've drafted this idea. Here are summary for attached two patches:
> >
> > 0001 patch allows us to create a logical slot without WAL reservation.
> >
> > 0002 patch is the main patch for dynamically enabling/disabling
> > logical decoding when wal_level is 'replica'.
>
> Thank You for the patches. I have done some initial testing, it seems
> to be working well. I will do more testing and review and will share
> further feedback.

I reviewed further and had few concerns:

1)
We now invalidate slots on standby if the primary (with
wal_level=replica) has dropped the last logical slot and internally
reverted its runtime (effective) wal_level back to replica. Consider
the following scenario involving a cascaded logical replication setup:

a) The publisher is configured with wal_level = replica and has
created a publication (pub1).
b) A subscriber server creates a subscription (sub1) to pub1. As part
of the slot creation for sub1, the publisher's effective wal_level is
switched to logical.
c) The publisher also has a physical standby, which in turn has its
own logical subscriber, named standby_sub1.

At this point, everything works as expected i.e. changes from the
publisher flow through the physical standby and are replicated to
standby_sub1. Now if the user drops sub1, the replication slot on the
primary is also dropped. Since this was the last logical slot, the
primary automatically switches its effective wal_level back to
replica. This change propagates to the standby, causing it to
invalidate the slot for standby_sub1. As a result, the standby logs
the following error:

STATEMENT: START_REPLICATION SLOT "standby_sub1" LOGICAL 0/0 (...)
ERROR: logical decoding needs to be enabled on the primary

Even if we manually recreate a logical slot on the primary afterward,
the standby_sub1 subscriber is not able to proceed:
ERROR: can no longer access replication slot "standby_sub1"
DETAIL: This replication slot has been invalidated due to
"wal_level_insufficient".

So the removal of the logical subscriber for the publisher has somehow
restricted the logical subscriber of standby to work. Is this
behaviour acceptable?

Without this feature, if I manually switch back wal_level to replica
on primary, then it will fail to start. This makes the issue obvious
and prevents misconfiguration.
FATAL: logical replication slot "sub2" exists, but "wal_level" < "logical"
HINT: Change "wal_level" to be "logical" or higher.

But the current behaviour is harder to diagnose, as the problem is
effectively hidden behind subscription/slot creation/deletion.

2)
'show effective_wal_level' shows output as 'logical' if a slot exists
on primary. But on physical standby, it still shows it as 'replica'
even in the presence of slots. Is this intentional?

3)
I haven’t tested this yet, but I’d like to discuss what the expected
behavior should be if a slot exists on the primary but is marked as
invalidated. Will an invalidated slot still cause the effective
wal_level to remain at logical, or will invalidating the only logical
slot trigger a switch back to replica?
There is a chance that a slot with un-reserved wal may be invalidated
due to time-out.

thanks
Shveta

On Fri, Jun 6, 2025 at 3:02 AM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
>
> On Wed, Jun 4, 2025 at 3:40 PM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
> >
> > On Wed, Jun 4, 2025 at 6:41 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Tue, May 20, 2025 at 9:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > >
> > > > Yeah, I find the idea that the presence of a logical slot will allow
> > > > the user to enable logical decoding/replication more appealing than
> > > > this new alternative, leaving aside the challenges of realizing it.
> >
> > +1. This idea appears more user-friendly and easier to understand
> > compared to other approaches, such as having multiple GUCs or using
> > ALTER SYSTEM.
> >
> > > I've drafted this idea. Here are summary for attached two patches:
> > >
> > > 0001 patch allows us to create a logical slot without WAL reservation.
> > >
> > > 0002 patch is the main patch for dynamically enabling/disabling
> > > logical decoding when wal_level is 'replica'.
> >
> > Thank You for the patches. I have done some initial testing, it seems
> > to be working well. I will do more testing and review and will share
> > further feedback.
>
> I reviewed further and had few concerns:

Thank you for reviewing this feature!

>
> 1)
> We now invalidate slots on standby if the primary (with
> wal_level=replica) has dropped the last logical slot and internally
> reverted its runtime (effective) wal_level back to replica. Consider
> the following scenario involving a cascaded logical replication setup:
>
> a) The publisher is configured with wal_level = replica and has
> created a publication (pub1).
> b) A subscriber server creates a subscription (sub1) to pub1. As part
> of the slot creation for sub1, the publisher's effective wal_level is
> switched to logical.
> c) The publisher also has a physical standby, which in turn has its
> own logical subscriber, named standby_sub1.
>
> At this point, everything works as expected i.e. changes from the
> publisher flow through the physical standby and are replicated to
> standby_sub1. Now if the user drops sub1, the replication slot on the
> primary is also dropped. Since this was the last logical slot, the
> primary automatically switches its effective wal_level back to
> replica. This change propagates to the standby, causing it to
> invalidate the slot for standby_sub1. As a result, the standby logs
> the following error:
>
> STATEMENT: START_REPLICATION SLOT "standby_sub1" LOGICAL 0/0 (...)
> ERROR: logical decoding needs to be enabled on the primary
>
> Even if we manually recreate a logical slot on the primary afterward,
> the standby_sub1 subscriber is not able to proceed:
> ERROR: can no longer access replication slot "standby_sub1"
> DETAIL: This replication slot has been invalidated due to
> "wal_level_insufficient".
>
> So the removal of the logical subscriber for the publisher has somehow
> restricted the logical subscriber of standby to work. Is this
> behaviour acceptable?
>
> Without this feature, if I manually switch back wal_level to replica
> on primary, then it will fail to start. This makes the issue obvious
> and prevents misconfiguration.
> FATAL: logical replication slot "sub2" exists, but "wal_level" < "logical"
> HINT: Change "wal_level" to be "logical" or higher.
>
> But the current behaviour is harder to diagnose, as the problem is
> effectively hidden behind subscription/slot creation/deletion.

The most upstream server in replication configuration would carefully
need to keep having at least one logical slot. One way to keep
effective_wal_level 'logical' on the publisher where wal_level =
'replica' is to have a logical slot without WAL reservation that is
not relevant with any subscriptions. It could require an extra logical
slot but seems workable. Does it resolve this concern?

> 2)
> 'show effective_wal_level' shows output as 'logical' if a slot exists
> on primary. But on physical standby, it still shows it as 'replica'
> even in the presence of slots. Is this intentional?

Yes. I think we should disallow the standbys to create a logical slot
as long as they use wal_level = 'replica', because otherwise the
standby would need to invalidate the logical slot at a promotion.
Which could cause a large down time in a failover case.

> 3)
> I haven’t tested this yet, but I’d like to discuss what the expected
> behavior should be if a slot exists on the primary but is marked as
> invalidated. Will an invalidated slot still cause the effective
> wal_level to remain at logical, or will invalidating the only logical
> slot trigger a switch back to replica?
> There is a chance that a slot with un-reserved wal may be invalidated
> due to time-out.

Good point. I think we don't need to decrease the effective_wal_level
to 'replica' even if we invalidate all logical slots. We need neither
WAL reservation nor dead tuple retention in order to set
effective_wal_level to 'logical' so I think it's straightforward that
effective_wal_level value depends on only the presence of logical
slots. If dle_replication_slot_timeout affects also logical slots
created with immeidately_reserve=false, we might want to exclude them
to avoid confusion.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Sat, Jun 7, 2025 at 2:44 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Fri, Jun 6, 2025 at 3:02 AM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
> >
> > On Wed, Jun 4, 2025 at 3:40 PM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
> > >
> > > On Wed, Jun 4, 2025 at 6:41 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > On Tue, May 20, 2025 at 9:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > >
> > > > > Yeah, I find the idea that the presence of a logical slot will allow
> > > > > the user to enable logical decoding/replication more appealing than
> > > > > this new alternative, leaving aside the challenges of realizing it.
> > >
> > > +1. This idea appears more user-friendly and easier to understand
> > > compared to other approaches, such as having multiple GUCs or using
> > > ALTER SYSTEM.
> > >
> > > > I've drafted this idea. Here are summary for attached two patches:
> > > >
> > > > 0001 patch allows us to create a logical slot without WAL reservation.
> > > >
> > > > 0002 patch is the main patch for dynamically enabling/disabling
> > > > logical decoding when wal_level is 'replica'.
> > >
> > > Thank You for the patches. I have done some initial testing, it seems
> > > to be working well. I will do more testing and review and will share
> > > further feedback.
> >
> > I reviewed further and had few concerns:
>
> Thank you for reviewing this feature!
>
> >
> > 1)
> > We now invalidate slots on standby if the primary (with
> > wal_level=replica) has dropped the last logical slot and internally
> > reverted its runtime (effective) wal_level back to replica. Consider
> > the following scenario involving a cascaded logical replication setup:
> >
> > a) The publisher is configured with wal_level = replica and has
> > created a publication (pub1).
> > b) A subscriber server creates a subscription (sub1) to pub1. As part
> > of the slot creation for sub1, the publisher's effective wal_level is
> > switched to logical.
> > c) The publisher also has a physical standby, which in turn has its
> > own logical subscriber, named standby_sub1.
> >
> > At this point, everything works as expected i.e. changes from the
> > publisher flow through the physical standby and are replicated to
> > standby_sub1. Now if the user drops sub1, the replication slot on the
> > primary is also dropped. Since this was the last logical slot, the
> > primary automatically switches its effective wal_level back to
> > replica. This change propagates to the standby, causing it to
> > invalidate the slot for standby_sub1. As a result, the standby logs
> > the following error:
> >
> > STATEMENT: START_REPLICATION SLOT "standby_sub1" LOGICAL 0/0 (...)
> > ERROR: logical decoding needs to be enabled on the primary
> >
> > Even if we manually recreate a logical slot on the primary afterward,
> > the standby_sub1 subscriber is not able to proceed:
> > ERROR: can no longer access replication slot "standby_sub1"
> > DETAIL: This replication slot has been invalidated due to
> > "wal_level_insufficient".
> >
> > So the removal of the logical subscriber for the publisher has somehow
> > restricted the logical subscriber of standby to work. Is this
> > behaviour acceptable?
> >
> > Without this feature, if I manually switch back wal_level to replica
> > on primary, then it will fail to start. This makes the issue obvious
> > and prevents misconfiguration.
> > FATAL: logical replication slot "sub2" exists, but "wal_level" < "logical"
> > HINT: Change "wal_level" to be "logical" or higher.
> >
> > But the current behaviour is harder to diagnose, as the problem is
> > effectively hidden behind subscription/slot creation/deletion.
>
> The most upstream server in replication configuration would carefully
> need to keep having at least one logical slot. One way to keep
> effective_wal_level 'logical' on the publisher where wal_level =
> 'replica' is to have a logical slot without WAL reservation that is
> not relevant with any subscriptions. It could require an extra logical
> slot but seems workable. Does it resolve this concern?
>

Yes, I agree that publishers should have a separate slot (not related
with any subscription) without WAL reservation to retain
effective_wal_level as logical when wal_level is replica. But the
question is how can that be ensured? Will it be user's responsibility
to always create that slot? If user has already some subscriptions
subscribing to most upstream server, then while setting up logical
replication on physical standby at a later stage, user will not even
encounter the error:
ERROR: logical decoding needs to be enabled on the primary,
HINT: Set wal_level >= logical or create at least one logical slot on
the primary.

And in lack of such error, users may always end up in the above
explained situation.

> > 2)
> > 'show effective_wal_level' shows output as 'logical' if a slot exists
> > on primary. But on physical standby, it still shows it as 'replica'
> > even in the presence of slots. Is this intentional?
>
> Yes. I think we should disallow the standbys to create a logical slot
> as long as they use wal_level = 'replica', because otherwise the
> standby would need to invalidate the logical slot at a promotion.
> Which could cause a large down time in a failover case.

Do you mean even if primary is running on effective_wal_level=logical,
we shall disallow slot-creation on standby if standby has
wal_level=replica? It means the $subject's enhancement is only valid
on primary?

Or the other way could be that we can have 2 trigger points for
enabling effective_wal_level to logical on primary:
1) One is when a logical slot is created on primary.
2) Another is when a logical slot is created on any of its physical standby.

We need to maintain these 2 separately as drop of last primary's slot
should not toggle it back to replica when any of its physical
standbys still need it. But if a publisher has multiple physical
standbys, then it will need extra handling i.e. last logical-slot drop
on standby1 should not end up toggling effective_wal_level to replica
when standby2 still has some logical slots. I am somehow trying to
think of a way where we have that extra slot without the user's
intervention.

>
> > 3)
> > I haven’t tested this yet, but I’d like to discuss what the expected
> > behavior should be if a slot exists on the primary but is marked as
> > invalidated. Will an invalidated slot still cause the effective
> > wal_level to remain at logical, or will invalidating the only logical
> > slot trigger a switch back to replica?
> > There is a chance that a slot with un-reserved wal may be invalidated
> > due to time-out.
>
> Good point. I think we don't need to decrease the effective_wal_level
> to 'replica' even if we invalidate all logical slots. We need neither
> WAL reservation nor dead tuple retention in order to set
> effective_wal_level to 'logical' so I think it's straightforward that
> effective_wal_level value depends on only the presence of logical
> slots. If dle_replication_slot_timeout affects also logical slots
> created with immeidately_reserve=false, we might want to exclude them
> to avoid confusion.
>

Yes, we shall exclude such slot from timeout based invalidation. As
there are chances that if a slot is invalidated, user may drop it
anytime.

thanks
Shveta

On Sun, Jun 8, 2025 at 8:57 PM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
>
> On Sat, Jun 7, 2025 at 2:44 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Fri, Jun 6, 2025 at 3:02 AM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
> > >
> > > On Wed, Jun 4, 2025 at 3:40 PM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
> > > >
> > > > On Wed, Jun 4, 2025 at 6:41 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > > >
> > > > > On Tue, May 20, 2025 at 9:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > > >
> > > > > > Yeah, I find the idea that the presence of a logical slot will allow
> > > > > > the user to enable logical decoding/replication more appealing than
> > > > > > this new alternative, leaving aside the challenges of realizing it.
> > > >
> > > > +1. This idea appears more user-friendly and easier to understand
> > > > compared to other approaches, such as having multiple GUCs or using
> > > > ALTER SYSTEM.
> > > >
> > > > > I've drafted this idea. Here are summary for attached two patches:
> > > > >
> > > > > 0001 patch allows us to create a logical slot without WAL reservation.
> > > > >
> > > > > 0002 patch is the main patch for dynamically enabling/disabling
> > > > > logical decoding when wal_level is 'replica'.
> > > >
> > > > Thank You for the patches. I have done some initial testing, it seems
> > > > to be working well. I will do more testing and review and will share
> > > > further feedback.
> > >
> > > I reviewed further and had few concerns:
> >
> > Thank you for reviewing this feature!
> >
> > >
> > > 1)
> > > We now invalidate slots on standby if the primary (with
> > > wal_level=replica) has dropped the last logical slot and internally
> > > reverted its runtime (effective) wal_level back to replica. Consider
> > > the following scenario involving a cascaded logical replication setup:
> > >
> > > a) The publisher is configured with wal_level = replica and has
> > > created a publication (pub1).
> > > b) A subscriber server creates a subscription (sub1) to pub1. As part
> > > of the slot creation for sub1, the publisher's effective wal_level is
> > > switched to logical.
> > > c) The publisher also has a physical standby, which in turn has its
> > > own logical subscriber, named standby_sub1.
> > >
> > > At this point, everything works as expected i.e. changes from the
> > > publisher flow through the physical standby and are replicated to
> > > standby_sub1. Now if the user drops sub1, the replication slot on the
> > > primary is also dropped. Since this was the last logical slot, the
> > > primary automatically switches its effective wal_level back to
> > > replica. This change propagates to the standby, causing it to
> > > invalidate the slot for standby_sub1. As a result, the standby logs
> > > the following error:
> > >
> > > STATEMENT: START_REPLICATION SLOT "standby_sub1" LOGICAL 0/0 (...)
> > > ERROR: logical decoding needs to be enabled on the primary
> > >
> > > Even if we manually recreate a logical slot on the primary afterward,
> > > the standby_sub1 subscriber is not able to proceed:
> > > ERROR: can no longer access replication slot "standby_sub1"
> > > DETAIL: This replication slot has been invalidated due to
> > > "wal_level_insufficient".
> > >
> > > So the removal of the logical subscriber for the publisher has somehow
> > > restricted the logical subscriber of standby to work. Is this
> > > behaviour acceptable?
> > >
> > > Without this feature, if I manually switch back wal_level to replica
> > > on primary, then it will fail to start. This makes the issue obvious
> > > and prevents misconfiguration.
> > > FATAL: logical replication slot "sub2" exists, but "wal_level" < "logical"
> > > HINT: Change "wal_level" to be "logical" or higher.
> > >
> > > But the current behaviour is harder to diagnose, as the problem is
> > > effectively hidden behind subscription/slot creation/deletion.
> >
> > The most upstream server in replication configuration would carefully
> > need to keep having at least one logical slot. One way to keep
> > effective_wal_level 'logical' on the publisher where wal_level =
> > 'replica' is to have a logical slot without WAL reservation that is
> > not relevant with any subscriptions. It could require an extra logical
> > slot but seems workable. Does it resolve this concern?
> >
>
> Yes, I agree that publishers should have a separate slot (not related
> with any subscription) without WAL reservation to retain
> effective_wal_level as logical when wal_level is replica. But the
> question is how can that be ensured? Will it be user's responsibility
> to always create that slot? If user has already some subscriptions
> subscribing to most upstream server, then while setting up logical
> replication on physical standby at a later stage, user will not even
> encounter the error:
> ERROR: logical decoding needs to be enabled on the primary,
> HINT: Set wal_level >= logical or create at least one logical slot on
> the primary.
>
> And in lack of such error, users may always end up in the above
> explained situation.

I think it's the user's responsibility to keep at least one logical
slot. It seems that setting wal_level to 'logical' would be the most
reliable solution for this case. We might want to provide a way to
keep 'logical' WAL level somehow but I don't have a good idea for now.

> > > 2)
> > > 'show effective_wal_level' shows output as 'logical' if a slot exists
> > > on primary. But on physical standby, it still shows it as 'replica'
> > > even in the presence of slots. Is this intentional?
> >
> > Yes. I think we should disallow the standbys to create a logical slot
> > as long as they use wal_level = 'replica', because otherwise the
> > standby would need to invalidate the logical slot at a promotion.
> > Which could cause a large down time in a failover case.
>
> Do you mean even if primary is running on effective_wal_level=logical,
> we shall disallow slot-creation on standby if standby has
> wal_level=replica? It means the $subject's enhancement is only valid
> on primary?

Thank you for pointing it out, my assumption was wrong. Even if the
standby sets wal_level='replica', it should be able to create logical
slots if the primary enables the logical decoding, and it should be
able to continue using it even after the promotion. I"ve updated the
patch accordingly.

>
> Or the other way could be that we can have 2 trigger points for
> enabling effective_wal_level to logical on primary:
> 1) One is when a logical slot is created on primary.
> 2) Another is when a logical slot is created on any of its physical standby.
>
> We need to maintain these 2 separately as drop of last primary's slot
> should not toggle it back to replica when any of its physical
> standbys still need it. But if a publisher has multiple physical
> standbys, then it will need extra handling i.e. last logical-slot drop
> on standby1 should not end up toggling effective_wal_level to replica
> when standby2 still has some logical slots. I am somehow trying to
> think of a way where we have that extra slot without the user's
> intervention.

Considering cascading replication cases too, 2) could be tricky as
cascaded standbys need to propagate the information of logical slot
creation up to the most upstream server.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, Jun 11, 2025 at 2:31 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> I think it's the user's responsibility to keep at least one logical
> slot. It seems that setting wal_level to 'logical' would be the most
> reliable solution for this case. We might want to provide a way to
> keep 'logical' WAL level somehow but I don't have a good idea for now.
>

Okay, Let me think more on this.

>
> Considering cascading replication cases too, 2) could be tricky as
> cascaded standbys need to propagate the information of logical slot
> creation up to the most upstream server.
>

Yes, I understand the challenges here.

Thanks for the v2 patches, few concerns:

1)
Now when the slot on standby is invalidated due to effective_wal_level
switched back to replica and if we restart standby, it fails to
restart even if wal_level is explicitly changed to logical in conf
file.

FATAL: logical replication slot "slot_st" exists, but logical
decoding is not enabled
HINT: Change "wal_level" to be "replica" or higher.

2)
I see that when primary switches back its effective wal_level to
replica while standby has wal_level=logical in conf file, then standby
has this status:

postgres=# show wal_level;
wal_level
-----------
logical

postgres=# show effective_wal_level;
effective_wal_level
---------------------
replica

Is this correct? Can effective_wal_level be < wal_level anytime? I
feel it can be greater but never lesser.

3)
When standby invalidate obsolete slots due to effective_wal_level on
primary changed to replica, it dumps below:
LOG: invalidating obsolete replication slot "slot_st2"
DETAIL: Logical decoding on standby requires "wal_level" >= "logical"
on the primary server

Shall we update this message as well to convey about slot-presence on primary.
DETAIL: Logical decoding on standby requires "wal_level" >= "logical"
or presence of logical slot on the primary server.

4)
I see that the slotsync worker is running all the time now as against
the previous behaviour where it will not start if wal_level is less
than logical or switched to '< logical' anytime. Even with wal_level
and effective_wal_level set to replica, slot-sync keeps on attempting
synchronization. This does not look correct. I think we need to find a
way to stop sot-sync worker when effective_wal_level is switched to
replica from logical.

5)
Can you please help me understand the changes at [1].

a) Why is it needed when we have code logic at [2]
b) in [1], why do we check n_inuse_logical_slots on standby and then
make decisions? Why not to disable logical-decoding directly just like
[2]

[1]:
+ if (xlrec.wal_level == WAL_LEVEL_LOGICAL)
+ {
+ /*
+ * If the primary increase WAL level to 'logical', we can
+ * unconditionally enable the logical decoding on the standby.
+ */
+ UpdateLogicalDecodingStatus(true);
+ }
+ else if (xlrec.wal_level == WAL_LEVEL_REPLICA &&
+ pg_atomic_read_u32(&ReplicationSlotCtl->n_inuse_logical_slots) == 0)
+ {
+ /*
+ * Disable the logical decoding if there is no in-use logical slot
+ * on the standby.
+ */
+ UpdateLogicalDecodingStatus(false);
+ }

[2]:
+ else if (info == XLOG_LOGICAL_DECODING_STATUS_CHANGE)
+ {
+ bool logical_decoding;
+
+ memcpy(&logical_decoding, XLogRecGetData(record), sizeof(bool));
+ UpdateLogicalDecodingStatus(logical_decoding);
+
+ /*
+ * Invalidate logical slots if we are in hot standby and the primary
+ * disabled the logical decoding.
+ */
+ if (!logical_decoding && InRecovery && InHotStandby)
+ InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_LEVEL,
+ 0, InvalidOid,
+ InvalidTransactionId);
+
+ LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
+ ControlFile->logicalDecodingEnabled = logical_decoding;
+ UpdateControlFile();
+ LWLockRelease(ControlFileLock);
+ }

thanks
Shveta

On Mon, Jun 16, 2025 at 11:48 PM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
>
> On Wed, Jun 11, 2025 at 2:31 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > I think it's the user's responsibility to keep at least one logical
> > slot. It seems that setting wal_level to 'logical' would be the most
> > reliable solution for this case. We might want to provide a way to
> > keep 'logical' WAL level somehow but I don't have a good idea for now.
> >
>
> Okay, Let me think more on this.
>
> >
> > Considering cascading replication cases too, 2) could be tricky as
> > cascaded standbys need to propagate the information of logical slot
> > creation up to the most upstream server.
> >
>
> Yes, I understand the challenges here.
>
> Thanks for the v2 patches, few concerns:

Thank you for the comments!

> 1)
> Now when the slot on standby is invalidated due to effective_wal_level
> switched back to replica and if we restart standby, it fails to
> restart even if wal_level is explicitly changed to logical in conf
> file.
>
> FATAL: logical replication slot "slot_st" exists, but logical
> decoding is not enabled
> HINT: Change "wal_level" to be "replica" or higher.

Good catch, we should fix it.
>
> 2)
> I see that when primary switches back its effective wal_level to
> replica while standby has wal_level=logical in conf file, then standby
> has this status:
>
> postgres=# show wal_level;
> wal_level
> -----------
> logical
>
> postgres=# show effective_wal_level;
> effective_wal_level
> ---------------------
> replica
>
> Is this correct? Can effective_wal_level be < wal_level anytime? I
> feel it can be greater but never lesser.

Hmm, I think we need to define what value we should show in
effective_wal_level on standbys because the standbys actually are not
writing any WALs and whether or not the logical decoding is enabled on
the standbys depends on the primary.

In the previous version patch, the standby's effective_wal_level value
depended solely on the standby's wal_level value. However, it was
confusing in a sense because it's possible that the logical decoding
could be available even though effective_wal_level is 'replica' if the
primary already enables it. One idea is that given that the logical
decoding availability and effective_wal_level value are independent in
principle, it's better to provide a SQL function to get the logical
decoding status so that users can check the logical decoding
availability without checking effective_wal_level. With that function,
it might make sense to revert back the behavior to the previous one.
That is, on the primary the effective_wal_level value is always
greater than or equal to wal_level whereas on the standbys it's always
the same as wal_level, and users would be able to check the logical
decoding availability using the SQL function. Or it might also be
worth considering to show effective_wal_level as NULL on standbys.

>
> 3)
> When standby invalidate obsolete slots due to effective_wal_level on
> primary changed to replica, it dumps below:
> LOG: invalidating obsolete replication slot "slot_st2"
> DETAIL: Logical decoding on standby requires "wal_level" >= "logical"
> on the primary server
>
> Shall we update this message as well to convey about slot-presence on primary.
> DETAIL: Logical decoding on standby requires "wal_level" >= "logical"
> or presence of logical slot on the primary server.

Will fix.

> 4)
> I see that the slotsync worker is running all the time now as against
> the previous behaviour where it will not start if wal_level is less
> than logical or switched to '< logical' anytime. Even with wal_level
> and effective_wal_level set to replica, slot-sync keeps on attempting
> synchronization. This does not look correct. I think we need to find a
> way to stop sot-sync worker when effective_wal_level is switched to
> replica from logical.

Right, will fix.

> 5)
> Can you please help me understand the changes at [1].
>
> a) Why is it needed when we have code logic at [2]

This is because we use XLOG_LOGICAL_DECODING_STATUS_CHANGE record only
for changing the logical decoding status online (i.e., without
restarting the server). So I think we still these part of code in
cases where we enable/disable the logical decoding by changing the
wal_level value with restarting the server

Suppose that both the primary and the standby set wal_level='replica',
the logical decoding is not available on both sides. If the primary
restarts with wal_level='logical', it doesn't write an
XLOG_LOGICAL_DECODING_STATUS_CHANGE record.

Another case is that suppose that the primary sets wal_level='logical'
and the standby sets wal_level='replica', the logical decoding is
available on both sides. If the primary restarts with
wal_level='replica' we need to somehow tell the standby the fact that
the logical decoding gets disabled. (BTW I realized we need to
invalidate the logical slots in this case too).

> b) in [1], why do we check n_inuse_logical_slots on standby and then
> make decisions? Why not to disable logical-decoding directly just like
> [2]

It seems the code is incorrect. We should disable the logical decoding
anyway if the primary disables it. Will fix.