mb_convert_encoding "\" (backslash) and "~" (tilde) convert failed to Shift_JIS

[youkidearitai]

Description

Backslash(\) and tilde(~) is convert to Shift_JIS (SJIS) using mb_convrert_encoding, But converted word is wrong word. Please see below code and 3v4l https://3v4l.org/nSVPB. Reproduced only PHP 8.1.

The following code:

<?php
var_dump(mb_convert_encoding(mb_convert_encoding("\\", "SJIS", "UTF-8"), "UTF-8", "SJIS"));
var_dump(mb_convert_encoding(mb_convert_encoding("~", "SJIS", "UTF-8"), "UTF-8", "SJIS"));

Resulted in this output:

string(3) "＼"
string(3) "〜"

But I expected this output instead:

string(1) "\"
string(1) "~"

PHP Version

PHP 8.1.4

Operating System

No response

[nikic]

cc @alexdowad

[alexdowad]

Dear @youkidearitai, thanks very much for this report. I will explain the behavior that you are observing below. However, if you have other information or references which may help to make these matters more clear, that is welcome.

The Wikipedia article on Shift-JIS is a good reference to start with. As it explains, single-byte characters in Shift-JIS are not in the ASCII character set, but rather in the JIS X 0201 character set. In JIS X 0201, 0x5C represents a Yen sign (¥, U+00A5). Also, in JIS X 0201, 0x7E represents an overline or overbar (‾, U+203E).

This is different from ASCII, where 0x5C represents a backslash (\) and 0x7E represents a tilde (~).

This means that when converting UTF-8 to Shift-JIS, we cannot correctly convert ASCII 0x5C to a Shift-JIS 0x5C byte, as that would change its meaning. However, Shift-JIS can also represent characters in the JIS X 0208 character set, using 2 bytes per character. Fortunately, JIS X 0208 kuten code 0x2141 is a "wave dash", which is similar to a tilde. JIS X 0208 kuten code 0x2140 is a backslash. So we can convert U+005C to the JIS X 0208 backslash, and U+007E to the JIS X 0208 wave dash.

The problem comes when converting in the reverse direction. ASCII has (halfwidth) backslash and tilde. Unicode has both halfwidth and fullwidth backslashes and tildes. However, JIS X 0208 has only one backslash character, which is generally treated as fullwidth, and one "wave dash" character, which is also treated as fullwidth. (JIS X 0201 has neither.)

The upshot of all this is that converting from Unicode → JISX 0201/0208 → Unicode is not a lossless conversion. If we convert JIS X 0208 0x2141 to the halfwidth tilde, then you may be happy, but others who were expecting to get a fullwidth tilde will not be. Likewise if we convert JIS X 0208 backslash to the halfwidth backslash.

You might wonder why you are only seeing this behavior on PHP 8.1. In short, it is because mbstring was buggy before that; it would convert U+005C to JIS X 0201 0x5C, which, as mentioned, is a different character. Likewise for U+007E and JIS X 0201 0x7E.

Generally, converting text back and forth between different legacy text encodings and expecting to get back what you started with is problematic. Since Unicode was designed to be a superset of all previous text encodings, it is generally best to do all processing in Unicode if possible. If that's not possible, then the best thing to do depends on the situation. If you must receive text in a legacy encoding and output it in the same legacy encoding, it may be best to do all the processing in the same encoding rather than converting to and from Unicode. On the other hand, if you receive text in legacy encodings but do not need to output it, then it would be better to convert to Unicode immediately when ingesting the text, and never convert it back.

(Note that what you are doing in the sample code is something which is almost never necessary or advisable: taking nice Unicode text, converting it to a legacy format, and then back to Unicode again. Converting from legacy → Unicode → legacy might sometimes make sense; converting from Unicode → legacy → Unicode almost never does.)

Please feel free to share any clarifying remarks, and thanks again.

[youkidearitai]

Dear, @alexdoward. thank you very much for reply.

I know ASCII backslash(0x5C) and tilde(0x7E) is different in JIS X 0201.But most Japanese users 0x5C and 0x7E is not using strict convert to 0x5C to U+00A5 and 0x7E to U+203E.

At least, it can be said that there are very few cases where the convert to backslash and tilde is multibyte.

Japanese Wikipedia Yen sign problem section in "現実的解決" (Realistic solution) explain Realistic solution.

この問題に対する現実的解決として、ほとんどの環境では日本の円記号はUnicodeのバックスラッシュ (U+005C) に変換される。
(As a practical solution to this problem, Japanese yen symbols are converted to Unicode backslashes (U+005C) in most environments.)

Some Japanese see to Yen sign, but code is almost 0x5C.

From this, Japanese language is through in general case ASCII and JIS X 0201 to 0x5C and 0x7E as it is.

Reference: プログラマのための文字コード入門 (ISBN978-4-7741-4164-0) Page 316 - 318

[alexdowad]

@youkidearitai, Thanks for those references. This was particularly interesting:

多くの日本語JISキーボードでは円記号とバックスラッシュのキーが別々に存在しているが、どちらを入力しても005Cが入力されるようになっている。

That is completely insane. Whoever came up with that idea deserves an award for bad design.

Thanks also for the reference to the book プログラマのための文字コード入門, though it doesn't seem there is any way I can access a copy right now. I will be in Japan in a few months (COVID-19 situation allowing) and could look it up then, but it would be nice to conclude this issue faster than that. 😆

As we consider this matter further, I think it would also help if you could explain: What is your use case here? How would treating SJIS 0x5C as a halfwidth backslash (U+005C) rather than a Yen sign (U+00A5) affect the software which you are developing or maintaining?

What is the typical source of SJIS text data for PHP-based software that you are working on, or are aware of? Text files uploaded by users? Direct text inputs in entry fields on a web site, from users whose OS supports SJIS input? Are we talking about newly created data, or legacy data which has been around for a long time, but you still need to process?

[youkidearitai]

@alexdowad , Thanks for reply. I'll answer questions.

Shift_JIS is still used in active use, and a common use case is importing CSV into Excel. Since Excel could only import CSV with Shift_JIS for a long time, there may still be cases where it is converted with SJIS.

In other words, mb_convert_encoding is done when downloading and uploading CSV.
Also, Windows uses \0x5C for the directory separator. If convert this, it will not function as a directory separator.

This conversion of \0x5C and \0x7E can be confusing. In most Japanese character code implementations, it seems customary to convert \0x5C and \0x7E untouched.

Japanese PHP users may have other opportunities to use Shift_JIS, so I was very interested in talking about this problem.

[alexdowad]

Shift_JIS is still used in active use, and a common use case is importing CSV into Excel. Since Excel could only import CSV with Shift_JIS for a long time, there may still be cases where it is converted with SJIS.

In other words, mb_convert_encoding is done when downloading and uploading CSV.

OK. So you have users uploading CSVs which are Shift-JIS encoded, and you also export Shift-JIS-encoded CSVs for download?
Are there reasons why you typically need to convert the text in those CSVs to Unicode?

Is it correct to say that when your users upload Shift-JIS-encoded CSVs, those may include 0x5C bytes, and the users expect those to be treated as backslashes?
How about when you export Shift-JIS-encoded CSVs for download? Is it typical that you need to include halfwidth backslashes in them?

Do you offer options for the encoding of these files? Or you export in Shift-JIS encoding by default, because that is what works for the greatest number of your users?

Also, Windows uses \0x5C for the directory separator. If convert this, it will not function as a directory separator.

That's true; but remembering the context here, we are looking at use cases for conversion between Shift-JIS and Unicode, and trying to determine whether there are more cases where interpreting 0x5C/0x7E according to spec is preferable, or more cases where interpreting them as ASCII is preferable. The discussion is not general, but specific to mb_convert_encoding and the needs of its users.

Does Shift-JIS text that PHP-based applications receive from users typically include Windows pathnames? After converting to ASCII/UTF-8/etc., would your PHP code typically take those pathnames and interpolate them into a Windows shell command, or pass them to fopen? If so, is there a reason to convert to Unicode, or could the 'raw' Shift-JIS text be used?

This conversion of \0x5C and \0x7E can be confusing. In most Japanese character code implementations, it seems customary to convert \0x5C and \0x7E untouched.

It is certainly confusing. My preference is to follow published specifications when possible; I believe this tends to reduce confusion in the long term. However, if there are strong practical reasons to deviate from specifications, that can certainly be done.

However, we do not want to flip-flop back and forth. To avoid flip-flopping, we need to thoroughly understand all the implications of either following the spec or deviating from it. After all factors are considered, and as many interested parties as possible are consulted, if the final decision is to change, then we should document the reason for the decision and stick to it.

One of the great challenges involved in working on open-source, and especially popular projects like PHP, is that users only speak up when something is not working well for them. With proprietary, in-house software, you generally know who all the users are and can survey them to see what they think about proposed changes. With open source, you often only discover later how your users were impacted by some change. This is a good reason to carefully think changes through and gather as much information as possible before deciding.

You mentioned that it seems customary to treat SJIS 0x5C and 0x7E as ASCII; if you can share as many specific examples as possible of existing software which does or doesn't do this, that would be appreciated. What is 'customary' is definitely one important factor to consider, since it shapes people's expectations.

[youkidearitai]

There are tons of cases where Unicode and Shift_JIS conversions are involved in CSV uploads and downloads, and it can be difficult to find out how much they are.
Similarly, it's hard to find out how much a Windows path doesn't work.

However, many users will find it very difficult for 0x5C and 0x7E to have "strict" conversions the moment they upgrade to PHP 8.1. This is good enough for Japanese users to hesitate to upgrade to PHP 8.1.

This is because it is perceived by the Japanese as being converted to a different character.

It is certainly confusing. My preference is to follow published specifications when possible; I believe this tends to reduce confusion in the long term. However, if there are strong practical reasons to deviate from specifications, that can certainly be done.
However, we do not want to flip-flop back and forth. To avoid flip-flopping, we need to thoroughly understand all the implications of either following the spec or deviating from it. After all factors are considered, and as many interested parties as possible are consulted, if the final decision is to change, then we should document the reason for the decision and stick to it.

As you said, I think it is correct to follow the published specifications. However, this is a change that breaks backwards compatibility, and if so, I feel that this change should be discussed in PHP RFCs and so on.

As about for the convention(customary), at least in Python 3, even if 0x5C or 0x7E is converted to Shift_JIS, it is converted as it is.

$ python3
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> '\\'.encode("SJIS")
b'\\'
>>> '\\~'.encode("SJIS")
b'\\~'
>>> 'あ'.encode("SJIS")
b'\x82\xa0'
>>>

I tried it with Ruby 2.7. After all it converted as it is.

$ ruby --version
ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-linux-gnu]
$ ruby -e 'puts "あ".encode("SJIS").encode("UTF-8")'
あ
$ ruby -e 'puts "\\".encode("SJIS")'
\
$ ruby -e 'puts "~".encode("SJIS")'
~

[cmb69]

Apparently, there are several variations of Shift JIS in use, of which some are already supported by MBString. In this case @youkidearitai is likely looking for Windows CP 932 ("CP932"), which behaves as expected wrt. ~ and \, see https://3v4l.org/ZjiBS.

I think this is just something that we should document better.