Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wolfSSL X509_STORE use after free due to CA caching #14278

Closed
alexsn opened this issue Jul 26, 2024 · 12 comments
Closed

wolfSSL X509_STORE use after free due to CA caching #14278

alexsn opened this issue Jul 26, 2024 · 12 comments

Comments

@alexsn
Copy link

alexsn commented Jul 26, 2024 •
edited
Loading

I did this

Tried upgrading curl to 8.9.0 which triggered a use-after-free bug in wolfssl X509_STORE caching code (added in 80aa519):

stderr:
=================================================================
==3959875==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d000005850 at pc 0x0000029b33e4 bp 0x7ffdd7d77d50 sp 0x7ffdd7d77d48
READ of size 4 at 0x61d000005850 thread T0
    #0 0x29b33e3 in wolfSSL_X509_STORE_free fbsource/src/x509_str.c:800:33
    #1 0x275e59f in wssl_x509_share_free arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c:461:5
    #2 0x264cd74 in hash_element_dtor arvr/third-party/curl/curl-8.9.0/lib/hash.c:44:7
    #3 0x269f125 in Curl_llist_remove arvr/third-party/curl/curl-8.9.0/lib/llist.c:146:5
    #4 0x269f380 in Curl_llist_destroy arvr/third-party/curl/curl-8.9.0/lib/llist.c:154:7
    #5 0x264d512 in Curl_hash_destroy arvr/third-party/curl/curl-8.9.0/lib/hash.c:219:7
    #6 0x26b10c5 in curl_multi_cleanup arvr/third-party/curl/curl-8.9.0/lib/multi.c:2847:5
    #7 0x2708a81 in Curl_close arvr/third-party/curl/curl-8.9.0/lib/url.c:251:7
    #8 0x25e6f9c in curl_easy_cleanup arvr/third-party/curl/curl-8.9.0/lib/easy.c:813:5
0x61d000005850 is located 2000 bytes inside of 2304-byte region [0x61d000005080,0x61d000005980)
freed by thread T0 here:
    #0 0x21dc157 in __interceptor_free.part.0 crtstuff.c
    #1 0x2ab6438 in wolfSSL_Free third-party/wolfssl/wolfcrypt/src/memory.c:437:9
    #2 0x28df33d in FreeSSL_Ctx third-party/wolfssl/src/internal.c:2757:9
    #3 0x29631fd in wolfSSL_CTX_free third-party/wolfssl/src/ssl.c:1401:9
    #4 0x275c729 in wolfssl_close arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c:1472:5
    #5 0x275895d in cf_close arvr/third-party/curl/curl-8.9.0/lib/vtls/vtls.c:1544:5
    #6 0x2753862 in ssl_cf_close arvr/third-party/curl/curl-8.9.0/lib/vtls/vtls.c:1656:3
    #7 0x260af41 in cf_setup_close arvr/third-party/curl/curl-8.9.0/lib/connect.c:1382:5
    #8 0x267e1f3 in cf_h2_close arvr/third-party/curl/curl-8.9.0/lib/http2.c:2450:5
    #9 0x261bc68 in cf_hc_close arvr/third-party/curl/curl-8.9.0/lib/cf-https-connect.c:459:5
    #10 0x25fb52c in Curl_conn_close arvr/third-party/curl/curl-8.9.0/lib/cfilters.c:176:5
    #11 0x260f5b3 in connc_disconnect arvr/third-party/curl/curl-8.9.0/lib/conncache.c:992:3
    #12 0x2612e35 in connc_shutdown_discard_all arvr/third-party/curl/curl-8.9.0/lib/conncache.c:570:5
    #13 0x260ffee in connc_shutdown_all arvr/third-party/curl/curl-8.9.0/lib/conncache.c:1134:3
    #14 0x260ffee in connc_close_all arvr/third-party/curl/curl-8.9.0/lib/conncache.c:611:3
    #15 0x2612771 in Curl_conncache_multi_close_all arvr/third-party/curl/curl-8.9.0/lib/conncache.c:1058:3
    #16 0x26b10ad in curl_multi_cleanup arvr/third-party/curl/curl-8.9.0/lib/multi.c:2844:5
    #17 0x2708a81 in Curl_close arvr/third-party/curl/curl-8.9.0/lib/url.c:251:7
    #18 0x25e6f9c in curl_easy_cleanup arvr/third-party/curl/curl-8.9.0/lib/easy.c:813:5
previously allocated by thread T0 here:
    #0 0x21dd31f in malloc (...) 
    #1 0x2ab6418 in wolfSSL_Malloc third-party/wolfssl/wolfcrypt/src/memory.c:352:15
    #2 0x2962c41 in wolfSSL_CTX_new_ex third-party/wolfssl/src/ssl.c:1304:25
    #3 0x298a48a in wolfSSL_CTX_new third-party/wolfssl/src/ssl.c:1367:12
    #4 0x2760322 in wolfssl_connect_step1 arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c:697:18
    #5 0x2760322 in wolfssl_connect_common arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c:1608:14
    #6 0x275c580 in wolfssl_connect_nonblocking arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c:1689:10
    #7 0x2758c0f in ssl_connect_nonblocking arvr/third-party/curl/curl-8.9.0/lib/vtls/vtls.c:512:10
    #8 0x27531cf in ssl_cf_connect arvr/third-party/curl/curl-8.9.0/lib/vtls/vtls.c:1697:14
    #9 0x25fd232 in Curl_conn_cf_connect arvr/third-party/curl/curl-8.9.0/lib/cfilters.c:371:12
    #10 0x2609859 in cf_setup_connect arvr/third-party/curl/curl-8.9.0/lib/connect.c:1284:14
    #11 0x25fd232 in Curl_conn_cf_connect arvr/third-party/curl/curl-8.9.0/lib/cfilters.c:371:12
    #12 0x261e05d in cf_hc_baller_connect arvr/third-party/curl/curl-8.9.0/lib/cf-https-connect.c:136:15
    #13 0x261a6d8 in cf_hc_connect arvr/third-party/curl/curl-8.9.0/lib/cf-https-connect.c:288:16
    #14 0x25fd66d in Curl_conn_connect arvr/third-party/curl/curl-8.9.0/lib/cfilters.c:419:14
    #15 0x26b88aa in multi_runsingle arvr/third-party/curl/curl-8.9.0/lib/multi.c:2107:16
    #16 0x26b6cc0 in curl_multi_perform arvr/third-party/curl/curl-8.9.0/lib/multi.c:2742:16
    #17 0x25e6b0b in easy_transfer arvr/third-party/curl/curl-8.9.0/lib/easy.c:676:15
    #18 0x25e6b0b in easy_perform arvr/third-party/curl/curl-8.9.0/lib/easy.c:770:42
    #19 0x25e6b0b in curl_easy_perform arvr/third-party/curl/curl-8.9.0/lib/easy.c:789:10
SUMMARY: AddressSanitizer: heap-use-after-free fbsource/src/x509_str.c:800:33 in wolfSSL_X509_STORE_free
Shadow bytes around the buggy address:
  0x0c3a7fff8ab0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a7fff8ac0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a7fff8ad0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a7fff8ae0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a7fff8af0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c3a7fff8b00: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x0c3a7fff8b10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a7fff8b20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a7fff8b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a7fff8b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a7fff8b50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3959875==ABORTING

The issue here being that X509_STORE in question is actually embedded in wolfSSL_CTX here as we get it from wolfSSL_CTX_get_cert_store here.

The point here is that we need to check whether WOLFSSL_X509_STORE->isDynamic == TRUE before we can do this (isDynamic is defined here but is under an ifdef). In any case, is it possible to disable this CA caching thing via some API call?

I expected the following

No response

curl/libcurl version

curl 8.9.0 with wolfSSL

operating system

Linux
@icing
Copy link
Contributor

icing commented Jul 26, 2024

If I read the wolfSSL code corectly, this depends on OPENSSL_EXTRA defined in the wolfSSL build? Our CI jobs seem to build wolfSSL always with --enable-all and might therefore not catch this.

Have you built wolfSSL yourself? Can you check if --enable-all fixes this? If so, we need to find a way to detect this difference either in our configure or in the code.

@alexsn
Copy link
Author

alexsn commented Jul 26, 2024

This patch fixes the issue:

diff --git a/arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c b/arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c
--- a/arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c
+++ b/arvr/third-party/curl/curl-8.9.0/lib/vtls/wolfssl.c
@@ -701,6 +701,15 @@
     return CURLE_OUT_OF_MEMORY;
   }
 
+  {
+    X509_STORE *store = wolfSSL_X509_STORE_new();
+    if (!store) {
+      failf(data, "SSL: could not create a X509 store");
+      return CURLE_OUT_OF_MEMORY;
+    }
+    wolfSSL_CTX_set_cert_store(backend->ctx, store);
+  }
+

@bagder - should I make a PR?

@alexsn
Copy link
Author

alexsn commented Jul 26, 2024

If I read the wolfSSL code corectly, this depends on OPENSSL_EXTRA defined in the wolfSSL build? Our CI jobs seem to build wolfSSL always with --enable-all and might therefore not catch this.

Have you built wolfSSL yourself? Can you check if --enable-all fixes this? If so, we need to find a way to detect this difference either in our configure or in the code.

I'm not building it from configure / make, in any case this can be resolved by creating a X509_STORE right after creating SSL_CTX so the store is always dynamic, alternatively we can do this right before calling set_cached_x509_store

@icing
Copy link
Contributor

icing commented Jul 26, 2024 •
edited
Loading

I think a PR would be great, if it preserves the sharing.

If you are not sure, I can have a go at it later.

icing added a commit to icing/curl that referenced this issue Jul 26, 2024
@icing
wolfssl, x509 store share fix

Verified

This commit was signed with the committer’s verified signature.
vszakats Viktor Szakats
GPG key ID: B5ABD165E2AEF201
Verified
Learn about vigilant mode
f91d66f 
When sharing the x509 store in wolfSSL, always use an explicitly
constructed one, as the SSLCTX might have "only" an internal one
which is not obeying reference count lifetimes.

- refs curl#14278
@icing icing mentioned this issue Jul 26, 2024
Closed
@icing
Copy link
Contributor

icing commented Jul 26, 2024

Please see #14279 for my proposed fix. Would be nice if you could verify. Thanks.

@alexsn
Copy link
Author

alexsn commented Jul 26, 2024

Please see #14279 for my proposed fix. Would be nice if you could verify. Thanks.

Yeah, that works

@alexsn
Copy link
Author

alexsn commented Jul 26, 2024

QQ: How does this suppose to work in the presence of CURLOPT_SSL_CTX_FUNCTION as this callback can modify the X509_STORE that's attached to the context which can potentially be shared no? Is the user expected to set CURLOPT_CAINFO to an empty string in order to prevent caching? Does CURLOPT_SSL_CTX_FUNCTION disables caching altogether?

@icing
Copy link
Contributor

icing commented Jul 26, 2024 •
edited
Loading

Good point. I think this is currently not really defined and needs some thought.

First of all, I think my wolfssl PR needs to trigger the x509 setup if a CURLOPT_SSL_CTX_FUNCTION is set. The openssl backend does the same. This has been missing.
Update: done.

Second, some users might not want to manipulate the store, but other things in the SSLCTX. Other users might want to override the store for just this one transfer, so sharing should not happen.

The safe thing is to auto-disable sharing when CURLOPT_SSL_CTX_FUNCTION is set, I believe. OTOH, overriding things in the callback that can be set via CURLOPT_* does not seem a good idea, and unexpected behaviour is then maybe what you get.

@icing
Copy link
Contributor

icing commented Jul 26, 2024

@bagder, feel free to chime in here when you return from your vacation.

@alexsn
Copy link
Author

alexsn commented Jul 26, 2024 •
edited
Loading

It is enough to just set CURLOPT_CAINFO to trigger this bug, the ssl ctx thing is just a possible issue I thought of.

But yeah, I agree that ssl ctx callback should disable caching as we have no control of what the callback will do or add a CURLOPT_SSL_CTX_CALLBACK_WILL_NOT_MODIFY_X509_STORE

@bagder
Copy link
Member

bagder commented Jul 26, 2024

I agree that ssl ctx callback should disable caching

I think this makes sense. Users using the ctx callback sort of take more charge over the handshake and we should then probably not complicate those further by doing caching as well.

@alexsn
Copy link
Author

alexsn commented Jul 27, 2024 •
edited
Loading

First of all, I think my wolfssl PR needs to trigger the x509 setup if a CURLOPT_SSL_CTX_FUNCTION is set. The openssl backend does the same. This has been missing. Update: done.

Yeah, this thing also got broken in 8.9.0 as the ssl ctx callback I was setting wasn't getting called (it's a separate issue I got in a different program - we use libcurl in a bunch of apps).

@icing - In your diff stack we also require:

  • Adding !data->set.ssl.fsslctx check when setting the value of cache_criteria_met.
  • It would be nice if we can avoid calling get_cached_x509_store if cache_criteria_met == FALSE, something like: cached_store = cache_criteria_met ? get_cached_x509_store(cf, data) : NULL; right after setting the value of cache_criteria_met.

@bagder bagder closed this as completed in 0f2876b Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@icing @bagder @alexsn