Getting psql to redisplay command after \e

The attached patch teaches psql to redisplay any not-yet-executed
query text after editing with \e. The fact that you don't get to
see what you're about to execute has been complained of before,
most recently at bug #16034 [1]. In that thread I complained that
we needed some probably-not-very-portable readline functionality
to make this work. However, after experimenting with trying to
shove text back into readline's buffer, I realized that there's
not really any need to do that: we just need to print the waiting
text and then collect another line. (As a bonus, it works the
same even if you turned off readline with -n.)

It also seems like to make this not confusing, we need to regurgitate
a prompt before the query text. As an example, if I do

regression=# \e

and then put this into the edited file:

select 1,
2,
3

what I see after exiting the editor is now

regression=# select 1,
2,
3
regression-#

Without the initial prompt it looks (to me anyway) like output
from the command, rather than something I've sort of automagically
typed.

In the cited bug, Pavlo argued that we should also print any
completed commands that get sent to the backend immediately
after \e. It'd be possible to do that by extending this patch
(basically, dump about-to-be-executed commands if need_redisplay
is still true), but on the whole I think that that would be overly
chatty, so I didn't do it.

This could stand some review and testing (e.g. does it interact
badly with any other psql features), so I'll add it to the
upcoming CF.

regards, tom lane

[1] https://www.postgresql.org/message-id/flat/16034-a7ebf0622970a1dd%40postgresql.org

Hello Tom,

> The attached patch teaches psql to redisplay any not-yet-executed
> query text after editing with \e.
>
> [...]

I've tested this patch. Although I agree that it is an improvement, I'm a
little at odd with the feature as is:

psql=> \e
# select 1...

then:

psql=> select 1...
psql-> <prompt>

I cannot move back with readline to edit further, I'm stuck there, which
is strange. I would prefer a simpler:

psql=> select 1...<prompt>

that would also be readline-aware, so that I know I'm there and ready to
nl but also to edit directly if I want that.

That would suggest to remove the ending newline rather than appending it,
and possibly to discuss a little bit with readline as well so that the
display line is also the current line for its point of view, so that it
can be edited further?

--
Fabien.

Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> writes:
>> The attached patch teaches psql to redisplay any not-yet-executed
>> query text after editing with \e.

> I've tested this patch. Although I agree that it is an improvement, I'm a
> little at odd with the feature as is:

> psql=> \e
> # select 1...

> then:

> psql=> select 1...
> psql-> <prompt>

> I cannot move back with readline to edit further, I'm stuck there, which
> is strange.

I don't follow. readline doesn't allow you to edit already-entered lines
today, that is, after typing "select 1<return>" you see

regression=# select 1
regression-#

and there isn't any way to move back and edit the already-entered line
within readline. I agree it might be nicer if you could do that, but
that's *far* beyond the scope of this patch. It would take entirely
fundamental rethinking of our use of libreadline, if indeed it's possible
at all. I also don't see how we could have syntax-aware per-line prompts
if we were allowing readline to treat the whole query as one line.

In the larger picture, tinkering with how that works would affect
every psql user at the level of "muscle memory" editing habits,
and I suspect that their reactions would not be uniformly positive.
What I propose here doesn't affect anyone who doesn't use \e at all.
Even for \e users it doesn't have any effect on what you need to type.

regards, tom lane

On Mon, 2019-10-28 at 23:00 -0400, Tom Lane wrote:
> The attached patch teaches psql to redisplay any not-yet-executed
> query text after editing with \e. The fact that you don't get to
> see what you're about to execute has been complained of before,
> most recently at bug #16034 [1]. In that thread I complained that
> we needed some probably-not-very-portable readline functionality
> to make this work. However, after experimenting with trying to
> shove text back into readline's buffer, I realized that there's
> not really any need to do that: we just need to print the waiting
> text and then collect another line. (As a bonus, it works the
> same even if you turned off readline with -n.)

This is a nice improvement.

I tried to torture it with a hex editor, but couldn't get it to break.

There were some weird carriage returns in the patch, but after I
removed them, it applied fine.

Yours,
Laurenz Albe

Hello Tom,

>> psql=> select 1...
>> psql-> <prompt>
>
>> I cannot move back with readline to edit further, I'm stuck there, which
>> is strange.
>
> I don't follow. readline doesn't allow you to edit already-entered lines
> today, that is, after typing "select 1<return>" you see
>
> regression=# select 1
> regression-#
>
> and there isn't any way to move back and edit the already-entered line
> within readline.

Yep.

My point is to possibly not implicitely <return> at the end of \e, but to
behave as if we were moving in history, which allows editing the lines, so
that you would get

psql=> select 1<cursor>

Instead of the above.

> I agree it might be nicer if you could do that, but that's *far* beyond
> the scope of this patch. It would take entirely fundamental rethinking
> of our use of libreadline, if indeed it's possible at all. I also don't
> see how we could have syntax-aware per-line prompts if we were allowing
> readline to treat the whole query as one line.

I was suggesting something much simpler than rethinking readline handling.
Does not mean that it is a good idea, but while testing the patch I would
have liked the unfinished line to be in the current editing buffer,
basically as if I had not typed <nl>.

ISTM more natural that \e behaves like history when coming back from
editing, i.e. the \e-edited line is set as the current buffer for
readline.

> In the larger picture, tinkering with how that works would affect
> every psql user at the level of "muscle memory" editing habits,
> and I suspect that their reactions would not be uniformly positive.
> What I propose here doesn't affect anyone who doesn't use \e at all.
> Even for \e users it doesn't have any effect on what you need to type.

--
Fabien.

Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> writes:
> My point is to possibly not implicitely <return> at the end of \e, but to
> behave as if we were moving in history, which allows editing the lines, so
> that you would get
> psql=> select 1<cursor>
> Instead of the above.

>> I agree it might be nicer if you could do that, but that's *far* beyond
>> the scope of this patch. It would take entirely fundamental rethinking
>> of our use of libreadline, if indeed it's possible at all. I also don't
>> see how we could have syntax-aware per-line prompts if we were allowing
>> readline to treat the whole query as one line.

> I was suggesting something much simpler than rethinking readline handling.
> Does not mean that it is a good idea, but while testing the patch I would
> have liked the unfinished line to be in the current editing buffer,
> basically as if I had not typed <nl>.

I did experiment with trying to do that, but I couldn't get it to work,
even with the single version of libreadline I had at hand. It appears
to me that readline() starts by clearing the internal buffer. Even if
we could persuade it to work in a particular readline version, I think
the odds of making it portable across all the libreadline and libedit
versions that are out there aren't very good. And there's definitely
no chance of being remotely compatible with that behavior when using the
bare tty drivers (psql -n).

In practice, if you decide that you don't like what you're looking at,
you're probably going to go back into the editor to fix it, ie issue
another \e. So I'm not sure that it's worth such pushups to get the
data into readline's buffer.

regards, tom lane

Hello Tom,

>> I was suggesting something much simpler than rethinking readline handling.
>> Does not mean that it is a good idea, but while testing the patch I would
>> have liked the unfinished line to be in the current editing buffer,
>> basically as if I had not typed <nl>.
>
> I did experiment with trying to do that, but I couldn't get it to work,
> even with the single version of libreadline I had at hand. It appears
> to me that readline() starts by clearing the internal buffer. Even if
> we could persuade it to work in a particular readline version, I think
> the odds of making it portable across all the libreadline and libedit
> versions that are out there aren't very good. And there's definitely no
> chance of being remotely compatible with that behavior when using the
> bare tty drivers (psql -n).

Argh, too bad.

This suggests that readline cannot be used to edit simply a known string?
:-( "rl_insert_text" looked promising, although probably not portable, and
I tried to make it work without much success anyway. Maybe I'll try to
investigate more deeply later.

Note that replacing the current buffer is exactly what history does. So
maybe that could be exploited by appending the edited line into history
(add_history) and tell readline to move there (could not find how to do
that automatically, though)? Or some other history handling…

> In practice, if you decide that you don't like what you're looking at,
> you're probably going to go back into the editor to fix it, ie issue
> another \e. So I'm not sure that it's worth such pushups to get the
> data into readline's buffer.

For me \e should mean edit, not edit-and-execute, so I should be back to
prompt, which is the crux of my unease with how the feature behaves,
because it combines two functions that IMO shouldn't.

Anyway the submitted patch is an improvement to the current status.

--
Fabien.

ne 3. 11. 2019 v 20:58 odesílatel Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr>
napsal:

>
> Hello Tom,
>
> >> I was suggesting something much simpler than rethinking readline
> handling.
> >> Does not mean that it is a good idea, but while testing the patch I
> would
> >> have liked the unfinished line to be in the current editing buffer,
> >> basically as if I had not typed <nl>.
> >
> > I did experiment with trying to do that, but I couldn't get it to work,
> > even with the single version of libreadline I had at hand. It appears
> > to me that readline() starts by clearing the internal buffer. Even if
> > we could persuade it to work in a particular readline version, I think
> > the odds of making it portable across all the libreadline and libedit
> > versions that are out there aren't very good. And there's definitely no
> > chance of being remotely compatible with that behavior when using the
> > bare tty drivers (psql -n).
>
> Argh, too bad.
>
> This suggests that readline cannot be used to edit simply a known string?
> :-( "rl_insert_text" looked promising, although probably not portable, and
> I tried to make it work without much success anyway. Maybe I'll try to
> investigate more deeply later.
>

pspg uses rl_insert_text

https://github.com/okbob/pspg/blob/59d115cd55926ab1886fc0dedbbc6ce0577b0cb3/src/pspg.c#L2522

Pavel

> Note that replacing the current buffer is exactly what history does. So
> maybe that could be exploited by appending the edited line into history
> (add_history) and tell readline to move there (could not find how to do
> that automatically, though)? Or some other history handling…
>
> > In practice, if you decide that you don't like what you're looking at,
> > you're probably going to go back into the editor to fix it, ie issue
> > another \e. So I'm not sure that it's worth such pushups to get the
> > data into readline's buffer.
>
> For me \e should mean edit, not edit-and-execute, so I should be back to
> prompt, which is the crux of my unease with how the feature behaves,
> because it combines two functions that IMO shouldn't.
>
> Anyway the submitted patch is an improvement to the current status.
>
> --
> Fabien.

Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> writes:
>> I did experiment with trying to do that, but I couldn't get it to work,
>> even with the single version of libreadline I had at hand. It appears
>> to me that readline() starts by clearing the internal buffer. Even if
>> we could persuade it to work in a particular readline version, I think
>> the odds of making it portable across all the libreadline and libedit
>> versions that are out there aren't very good. And there's definitely no
>> chance of being remotely compatible with that behavior when using the
>> bare tty drivers (psql -n).

> This suggests that readline cannot be used to edit simply a known string?
> :-( "rl_insert_text" looked promising, although probably not portable, and
> I tried to make it work without much success anyway. Maybe I'll try to
> investigate more deeply later.

I think that rl_insert_text and friends can probably only be used from
readline callback functions. So in principle maybe you could make it
work by having an rl_startup_hook that injects text if there is any
to inject. There would remain the issues of (a) is it portable across
a wide range of readline and libedit versions, (b) will the prompting
behavior be nice, and (c) do we really want this to work fundamentally
differently when readline is turned off?

(Pavel's code cited nearby seems to me to be a fine example of what
we do *not* want to do. Getting in bed with libreadline to that
extent is inevitably going to misbehave in some places.)

>> In practice, if you decide that you don't like what you're looking at,
>> you're probably going to go back into the editor to fix it, ie issue
>> another \e. So I'm not sure that it's worth such pushups to get the
>> data into readline's buffer.

> For me \e should mean edit, not edit-and-execute, so I should be back to
> prompt, which is the crux of my unease with how the feature behaves,
> because it combines two functions that IMO shouldn't.

I don't understand that complaint at all. My proposed patch does not
change the behavior to force execution, and it does display a prompt ---
one that reflects whether you've given a complete command.

regards, tom lane

I wrote:
> Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> writes:
>> This suggests that readline cannot be used to edit simply a known string?
>> :-( "rl_insert_text" looked promising, although probably not portable, and
>> I tried to make it work without much success anyway. Maybe I'll try to
>> investigate more deeply later.

> I think that rl_insert_text and friends can probably only be used from
> readline callback functions. So in principle maybe you could make it
> work by having an rl_startup_hook that injects text if there is any
> to inject. There would remain the issues of (a) is it portable across
> a wide range of readline and libedit versions, (b) will the prompting
> behavior be nice, and (c) do we really want this to work fundamentally
> differently when readline is turned off?

I thought maybe you were going to work on this right away, but since
you haven't, I went ahead and pushed what I had. There's certainly
plenty of time to reconsider if you find a better answer.

regards, tom lane

>> I think that rl_insert_text and friends can probably only be used from
>> readline callback functions. So in principle maybe you could make it
>> work by having an rl_startup_hook that injects text if there is any
>> to inject. There would remain the issues of (a) is it portable across
>> a wide range of readline and libedit versions, (b) will the prompting
>> behavior be nice, and (c) do we really want this to work fundamentally
>> differently when readline is turned off?
>
> I thought maybe you were going to work on this right away, but since
> you haven't, I went ahead and pushed what I had. There's certainly
> plenty of time to reconsider if you find a better answer.

Indeeed.

I started to play with the startup hook, it kind of worked but not exactly
as I wished, and I do not have much time available this round.

--
Fabien.