Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add portable way to unblock/alert blocking curl_multi_wait from another thread #4418

Closed
pps83 opened this issue Sep 25, 2019 · 6 comments
Closed

Add portable way to unblock/alert blocking curl_multi_wait from another thread #4418

pps83 opened this issue Sep 25, 2019 · 6 comments
Labels
feature-request libcurl API

Comments

@pps83
Copy link
Contributor

pps83 commented Sep 25, 2019

I'm very happy with curl :) but the way I implemented http code in our project is by using 1ms busy wait worker thread that doesn't even use curl_multi_wait. It simply sleeps(1ms) and then does curl_multi_perform. One of the reasons is because I wasn't able to (or rather didn't want to) implement portable code that would be able to "unblock" curl_multi_wait call if I needed to. On non-windows systems generally it's a simple task: an anonymous pipe can be used to alert curl_multi_wait. On windows however, this won't work as select on windows requires socket to be passed. It would be helpful if libcurl instead provided through some api a way to unblock curl_multi_wait call from another thread (perhaps internally it already has something like that?).
On non-windows systems it may implement it as anonymous pipe. On windows it may use self-connected udp socket (this way nobody on localhost would be able to send data to it).
@bagder
Copy link
Member

bagder commented Sep 25, 2019

Yes it would be useful and I'm sure appreciated by other users too. No, libcurl doesn't have anything like that internally already.

@pps83
Copy link
Contributor Author

pps83 commented Sep 25, 2019 •
edited
Loading

Somewhat related quick question: I use CURLOPT_CONNECT_ONLY (for websockets) and when I curl_easy_cleanup the handle (after it was remove from multi etc) tcp connection doesn't seem to get closed (according to wireshark). In my case app runs for longer time and I enabled TCP keepalives and a minute after I performed all closing I see keepalive messages in wireshark (1122 is websocket server in nodejs that replies with websocket-close and closes connection):

image

Is it curl bug, or I need to dome something else to close connection?

@pps83
Copy link
Contributor Author

pps83 commented Sep 25, 2019 •
edited
Loading

I tried to set CURLOPT_FORBID_REUSE and CURLOPT_CONNECT_ONLY, but in this case nothing works: after connect succeeds I cannot make any sends, curl_easy_send weirdly returns CURLE_UNSUPPORTED_PROTOCOL. I use 7.64.1

(when we tried to switch to 7.65.3 we had catastrophic TLS 5x-10x overload on our load balancers and it was not related to connection max age and wasn't related to that bug that divided max age time twice. We had to roll back as an emergency for now)

@bagder
Copy link
Member

bagder commented Sep 25, 2019

That sounds like a totally different issue than "unblocking" curl_multi_wait from another thread. If this is a bug, please file this as a separate issue with details to allow us to reproduce.

@bagder
Copy link
Member

bagder commented Oct 10, 2019

I've jotted down some ideas for curl_multi_unblock in the wiki. I'm interested in feedback and comments!

@jay
Copy link
Member

jay commented Oct 10, 2019 •
edited
Loading

I think an interesting issue is whether a socket always receiving data with received data like in curl_multi_perform would cause an unblock, like can curl_multi_perform be always out (I think I asked about this before my read of the code was unclear on it) and if there is data pending should it return. So say you want to add some URLs to the multi, in your loop you've got a lock free list or something that is checked each time multi perform returns, well you want those transfers added right away, so you in the other thread you add them then you can call unblock?

@bagder bagder closed this as completed in cba52e2 Nov 8, 2019
@ngg ngg mentioned this issue Nov 17, 2019
Closed
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
@ngg
add curl_multi_unblock()
bd14068 
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 25, 2019
@ngg
multi: add curl_multi_wakeup()
e9e03f1 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
bagder pushed a commit that referenced this issue Nov 25, 2019
@ngg @bagder
multi: add curl_multi_wakeup()
f3c35e3 
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes #4418
Closes #4608
@lock lock bot locked as resolved and limited conversation to collaborators Feb 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
feature-request libcurl API
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

multi: add curl_multi_wakeup() tresorit/curl
3 participants
@bagder @jay @pps83