AppleIDN: fails IDN tests 1034, 1035

[vszakats]

I did this

Run tests with a curl built with AppleIDN:

FAIL 165: [HTTP over proxy with IDN host name] HTTP HTTP GET HTTP proxy IDN
FAIL 1034: [HTTP over proxy with malformatted IDN host name] HTTP HTTP GET HTTP proxy IDN FAILURE config file
FAIL 1035: [HTTP over proxy with too long IDN host name] HTTP HTTP GET HTTP proxy IDN FAILURE

TESTFAIL: These test cases failed: 165 1034 1035

Ref: https://github.com/curl/curl/actions/runs/9919852332/job/27406122714

For 165, it seems to be caused by AppleIDN converting ß to ss instead of punycode ß expected by the test (and generated by libidn2 with en_US.UTF-8).

/cc @MonkeybreadSoftware

edit: test 165 fixed via #14179.

I expected the following

Tests passing

curl/libcurl version

master 3765d75

operating system

Apple

[MonkeybreadSoftware]

Yes, seems like these tests or the library may need updating. I'll check

I found a FAQ entry from DENIC:

Most applications support the characters which have been permitted since the Internet standard 2003 became effective. The situation is different for "ß", which can only be used since the IDN standard was revised in August 2010. Up to that date, "ß" was always converted into "ss" (normalized).

https://www.denic.de/en/faqs/faqs-about-idns-ss#code-365

[MonkeybreadSoftware]

to solve the große problem:

uidna_openUTS46 is called with flags, currently UIDNA_CHECK_BIDI.
Change that to

UIDNA_CHECK_BIDI|UIDNA_NONTRANSITIONAL_TO_UNICODE|UIDNA_NONTRANSITIONAL_TO_ASCII

for both directions, so it does the ß differently.

[MonkeybreadSoftware]

For test 1034, it seems like Apple's unicode function here doesn't do any UTF-8 input validation. The invalid character is just passed through. Not sure whether you like to do an UTF-8 validation yourself in the code.
Also no idea whether passing on the invalid UTF-8 to a DNS server later causes trouble.

For 1035, there is an RFC defining 63 characters, but the unicode functions don't enforce this. I thought domains could be longer, but well if you like to enforce this, you could add a check.

In both cases I wonder if CURL should enforce this at all. If the name has invalid characters or is too long, the DNS resolver will not find anything.

[MonkeybreadSoftware]

I just made the change for the flags:
#14179

[dfandrich]

For 1035, there is an RFC defining 63 characters, but the unicode functions don't enforce this. I thought domains could be longer, but well if you like to enforce this, you could add a check.

IIRC, DNS has a limit of 63 character on hostnames, but names resolved outside DNS have no such limit. So, preventing longer names here seems unnecessary.

[bagder]

Also no idea whether passing on the invalid UTF-8 to a DNS server later causes trouble.

It does not and cannot. DNS works with plain ASCII and it does not even know what UTF-8 or unicode is.

In both cases I wonder if CURL should enforce this at all. If the name has invalid characters or is too long, the DNS resolver will not find anything.

1. Then the resolver function returns that information to us
2. name resolving is not limited to DNS

[vszakats]

FWIW WinIDN has the same "grosse" problem:
https://github.com/curl/curl/actions/runs/9931630439/job/27431789481#step:13:2148