OpenSSL::PKey::EC.check_key is mostly useless when linked against OpenSSL 3

[bannable]

When calling #check_key against an OpenSSL::PKey::EC instance representing an invalid point for the group, the method always returns true.

I believe this is because OpenSSL 3 deprecated EC_KEY_check_key, and the underlying call is swapped out for EVP_PKEY_public_check in the 3.x branch.

However, the EVP_PKEY_public_check does not serve the same purpose as EC_KEY_check_key.

EVP_PKEY_public_check validates only the resulting public component, and does not validate the private component. I believe EVP_PKEY_pairwise_check is closer to the behavior of EC_KEY_check_key, where both the public and private components are validated, though it seems to assume the presence of a private component.

Reproducer

# check.rb
ver = ARGV[0]
gem 'openssl', ver
require 'openssl'

# ECDSA secp384r1 encoded key where the point is not on the curve
pem = <<~INVALID_KEY
-----BEGIN EC PRIVATE KEY-----
MIGkAgEBBDDA1Tm0m7YhkfeVpFuarAJYVlHp2tQj+1fOBiLa10t9E8TiQO/hVfxB
vGaVEQwOheWgBwYFK4EEACKhZANiAASyGqmryZGqdpsq5gEDIfNvgC3AwSJxiBCL
XKHBTFRp+tCezLDOK/6V8KK/vVGBJlGFW6/I7ahyXprxS7xs7hPA9iz5YiuqXlu+
lbrIpZOz7b73hyQQCkvbBO/Avg+hPAk=
-----END EC PRIVATE KEY-----
INVALID_KEY

begin
  result = OpenSSL::PKey::EC.new(pem).check_key
rescue =e
  result = e.message
end

puts format('%25s: %s','RUBY_VERSION', RUBY_VERSION)
puts format('%25s: %s','OPENSSL_LIBRARY_VERSION', OpenSSL::OPENSSL_LIBRARY_VERSION)
puts format('%25s: %s','OPENSSL_VERSION', OpenSSL::VERSION)
puts format('%25s: %s','result', result)

OpenSSL 1.1.1

$ rvm 2.7 do ruby check.rb 2.1.2
             RUBY_VERSION: 2.7.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 2.1.2
                   result: EC_KEY_check_key: invalid private key
$ rvm 2.7 do ruby check.rb 3.0.1
             RUBY_VERSION: 2.7.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 3.0.1
                   result: EVP_PKEY_public_check: invalid private key

$ rvm 3.1.2 do ruby check.rb
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 3.0.1
                   result: EVP_PKEY_public_check: invalid private key

OpenSSL 3.0.2

$ ruby check.rb
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 3.0.2 15 Mar 2022
          OPENSSL_VERSION: 3.0.1
                   result: true
$ ruby check.rb 3.0.0
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 3.0.2 15 Mar 2022
          OPENSSL_VERSION: 3.0.0
                   result: true

[rhenium]

#565 is superseded by #580. This is now fixed in the maint-3.0 branch, to be included in openssl gem v3.0.2/v3.1.0.

Thank you for the report.