Unicode handling of warnings appears broken in 5.18.2 and 5.24.0

[p5pRT]

Migrated from rt.perl.org#128701 (status was 'resolved')

Searchable as RT128701$

[p5pRT]

From [email protected]

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line 4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line 4.

Or at least an error about the use of an invalid punctuation character as a
delimiter (perhaps "what do you think this is, Perl 6?!"). But definitely
not munged latin-1...

Also tried this under 5.24.0 and got the same output.

--
Aaron Sherman, M.​:
P​: 617-440-4332 Google Talk, Email and Google Plus​: ajs@​ajs.com
Toolsmith, developer, gamer and life-long student.

[p5pRT]

From @cpansprout

On Fri Jul 22 10​:36​:25 2016, ajs wrote​:

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line 4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line 4.

Or at least an error about the use of an invalid punctuation character as a
delimiter (perhaps "what do you think this is, Perl 6?!"). But definitely
not munged latin-1...

Also tried this under 5.24.0 and got the same output.

The same bug occurs with the ‘(Might be a runaway multi-line...)’ hint that sometimes accompanies a syntax error.

In a utf-8 terminal, with 5.12.4​:

$ perl -CS -e 'use utf8; q Ω' -e 'Ω >'
syntax error at -e line 2, at EOF
  (Might be a runaway multi-line ?????????????????????????? string starting on line 1)
Execution of -e aborted due to compilation errors.

And 5.25.3​:

$ ./perl -Ilib -CS -e 'use utf8; q Ω' -e 'Ω >'
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
syntax error at -e line 2, at EOF
  (Might be a runaway multi-line ?????????????????????????? string starting on line 1)
Execution of -e aborted due to compilation errors.

Ow!

(Another thing I might note is that the runaway quote hint should include the operator if the delimiter is non-standard, because this is unhelpful​:

$ perl -e 'q x' -e 'x >'
syntax error at -e line 2, at EOF
  (Might be a runaway multi-line xx string starting on line 1)
Execution of -e aborted due to compilation errors.
)
--

Father Chrysostomos

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @cpansprout

On Tue Jul 26 00​:04​:19 2016, sprout wrote​:

On Fri Jul 22 10​:36​:25 2016, ajs wrote​:

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line 4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line 4.

This is fixed in cb65013. But there are problems with heredoc terminators (handled by the same code, more or less), so there is still some more fixing to do​:

$ ./perl -Ilib -CS -e 'use utf8; <<"«"'
Can't find string terminator "«" anywhere before EOF at -e line 1.

The same bug occurs with the ‘(Might be a runaway multi-line...)’ hint
that sometimes accompanies a syntax error.

In a utf-8 terminal, with 5.12.4​:

$ perl -CS -e 'use utf8; q Ω' -e 'Ω >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line ?????????????????????????? string
starting on line 1)
Execution of -e aborted due to compilation errors.

That is not yet fixed.

(Another thing I might note is that the runaway quote hint should
include the operator if the delimiter is non-standard, because this is
unhelpful​:

$ perl -e 'q x' -e 'x >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line xx string starting on line 1)
Execution of -e aborted due to compilation errors.
)

Neither is that.

--

Father Chrysostomos

[p5pRT]

From @cpansprout

On Tue Jul 26 01​:55​:10 2016, sprout wrote​:

On Tue Jul 26 00​:04​:19 2016, sprout wrote​:

On Fri Jul 22 10​:36​:25 2016, ajs wrote​:

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line
4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line
4.

This is fixed in cb65013. But there are problems with heredoc
terminators (handled by the same code, more or less), so there is
still some more fixing to do​:

$ ./perl -Ilib -CS -e 'use utf8; <<"«"'
Can't find string terminator "«" anywhere before EOF at -e line 1.

That one is fixed now in commit 7f7f7d0.

The same bug occurs with the ‘(Might be a runaway multi-line...)’
hint
that sometimes accompanies a syntax error.

In a utf-8 terminal, with 5.12.4​:

$ perl -CS -e 'use utf8; q Ω' -e 'Ω >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line ?????????????????????????? string
starting on line 1)
Execution of -e aborted due to compilation errors.

That is not yet fixed.

Actually, cb65013 *did* fix it. I didn’t realise that Perl_sv_catpvf supports %c with numbers > 255, but apparently it does. However, it only goes up to 32 bits (at least on a Mac), because %c takes an int (which is 32 bits).

That means you get garbled output for heredoc delimiters above \x{7fffffff}, but those are not valid Unicode characters anyway, and hence cannot display in a terminal. So I’m not going to worry about it. (I started to write a patch, but it adds to much complexity to the code.)

(Another thing I might note is that the runaway quote hint should
include the operator if the delimiter is non-standard, because this
is
unhelpful​:

$ perl -e 'q x' -e 'x >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line xx string starting on line 1)
Execution of -e aborted due to compilation errors.
)

Neither is that.

I have decided to leave this alone. It’s not easy to fix, and nobody has complained about it.

--

Father Chrysostomos

[p5pRT]

@cpansprout - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.26.0, this and 210 other issues have been
resolved.

Perl 5.26.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.26.0

If you find that the problem persists, feel free to reopen this ticket.

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'