Trying Jay's PostUpload example I noted there were 3202 calls to:

WSAIoctl(sockfd, SIO_IDEAL_SEND_BACKLOG_QUERY ...);
setsockopt (784, SOL_SOCKET, SO_SNDBUF, ...);

The Ideal Send Backlog value started at 64kB, a few at 130kB and the remaining at 256 kB. Simply disabling the win_update_buffer_size() call, the transfer speed increased a bit. Somewhat related?

But I got around 9.3 MBit/s which IMHO is very good speed. In Google Chrome it reports 10.2 MBit/s blank to the same Frankfurt, DE server at https://testmy.net/upload. Using a 50 MByte test file for each test.

The Ideal Send Backlog value started at 64kB, a few at 130kB and the remaining at 256 kB. Simply disabling the win_update_buffer_size() call, the transfer speed increased a bit. Somewhat related?

I see the opposite, performance suffers without it.

17f58c8 (parent of d2a7d7c) with win_update_buffer_size (build default)
Transfer rate: 3886 KB/sec (52429310 bytes in 13 seconds)
Transfer rate: 3182 KB/sec (52429310 bytes in 16 seconds)
Transfer rate: 4103 KB/sec (52429310 bytes in 12 seconds)

17f58c8 (parent of d2a7d7c) without win_update_buffer_size
Transfer rate: 1871 KB/sec (52429310 bytes in 27 seconds)
Transfer rate: 1934 KB/sec (52429310 bytes in 26 seconds)
Transfer rate: 1908 KB/sec (52429310 bytes in 27 seconds)

d2a7d7c with win_update_buffer_size (build default)
Transfer rate: 197 KB/sec (52429310 bytes in 260 seconds)
Transfer rate: 502 KB/sec (52429310 bytes in 102 seconds)
Transfer rate: 255 KB/sec (52429310 bytes in 201 seconds)

d2a7d7c without win_update_buffer_size
Transfer rate: 63 KB/sec (52429310 bytes in 815 seconds)
Transfer rate: 63 KB/sec (52429310 bytes in 813 seconds)
Transfer rate: 63 KB/sec (52429310 bytes in 816 seconds)

curl 7.73.0-DEV (i386-pc-win32) libcurl/7.73.1-DEV OpenSSL/1.0.2t nghttp2/1.40.0

Edit: When I run the 50MB test via the browser I get mbps results similar to what I get from 17f58c8, which is expected.

If the 1 second delay comes from the timeout, the main question should be: why does it run into that timeout?

@jay would you mind performing the same tests for the follow up commits to multi.c which should actually have already improved the situation?

1. 3334ee6
2. 003e81e
3. 1060955

Especially the code section regarding already_writable was replaced after the commits you identified.

All of them are slow. I think we should work from master for further testing.

If the 1 second delay comes from the timeout, the main question should be: why does it run into that timeout?

That is the question alright and I don't know the answer. I assume the event is supposed to be signaled at that point since there is data, which would mean no wait. One strange thing I've noticed is in some of the results the speed is sometimes doubled or halved. For example, usually 100 seconds but sometimes 50 or 200.

• 3334ee6

Transfer rate: 489 KB/sec (52429310 bytes in 105 seconds)
Transfer rate: 503 KB/sec (52429310 bytes in 102 seconds)
Transfer rate: 503 KB/sec (52429310 bytes in 102 seconds)

• 003e81e

Transfer rate: 255 KB/sec (52429310 bytes in 201 seconds)
Transfer rate: 503 KB/sec (52429310 bytes in 102 seconds)
Transfer rate: 968 KB/sec (52429310 bytes in 53 seconds)

• 1060955

Transfer rate: 255 KB/sec (52429310 bytes in 201 seconds)
Transfer rate: 503 KB/sec (52429310 bytes in 102 seconds)
Transfer rate: 503 KB/sec (52429310 bytes in 102 seconds)

5c8849c (master 2020-11-22)

Transfer rate: 497 KB/sec (52429310 bytes in 103 seconds)
Transfer rate: 503 KB/sec (52429310 bytes in 102 seconds)
Transfer rate: 968 KB/sec (52429310 bytes in 53 seconds)

I had one idea for an alternative implementation of this waiting in multi.c for Windows which could potentially improve the situation again, but avoid the fake socket pairs being created:

1. revert back to using Curl_poll also on Windows, but just use it for polling with no timeout and exclude the wakeup socket here.
2. check if any of the sockets in Curl_poll are already ready and skip the next part if that is the case.
3. if nothing is ready yet, use the new implementation to wait on the sockets including the new wakeup event.

This way we should be able to reduce again the overall number of syscalls, by doing one Curl_poll as readiness pre-check with the old logic and then just perform the waiting with the new logic including the new wakeup event instead of the old wakeup socket pair. This could potentially also reduce the number of ifdef's in the code again.

How does that reduce the number of syscalls? It'd either be poll or objectwait, right?

More importantly: that logic doesn't work if there's a timeout:

In state (1) it only waits for socket and let's say we have a 1000ms timeout. During this period the wakeup event isn't checked and thus doesn't work. We delay the wakeup with this time.

As I said, step 1 (" with no timeout") wouldn't use a timeout, it would just poll with a timeout of 0 as a pre-check. At the moment a pre-check is done with Curl_socket_check per socket in the wait setup loop, but we could also just use the final poll structure (which is currently disabled for Windows) to do the pre-check without caring for the wakeup socket/event. The wakeup event would only be checked in case of actually waiting. To summarize:

1. poll with timeout 0 as pre-check on sockets
2. if nothing ready yet: objectwait with timeout on sockets and wakeup event

We're only 6 days from release now. Can we revert this regression and work on a proper fix long-term? The only PR is still draft...

If that WSAWaitForMultipleEvents line is commented out then the transfer completes in the expected time (but then of course eats CPU in the interim since there's no poll wait).

One comment about this... In the old days of Win32 programming, you could Sleep(0) to give up the remainder of your time slice. A time slice was 30 milliseconds (2 slices) on a client, and 180 milliseconds (12 slices) on a server.

It looks like the behavior has carried through to modern versions of Windows. See Sleep function (synchapi.h).

If you would like to play nicely without a Wait function, then maybe try a Sleep(0). Sleep(0) will yield to other threads. It may work well enough to get you through without spinning on the cpu until you have time to work up a proper fix.

Also be aware the Sleep function yields the remainder of the time slice. SleepEx actually puts the process in a Wait state, so be careful of SleepEx. SleepEx may be just as bad as a Wait function.

Russinovich, Solomon and Ionescu discuss time slices in the Windows Internal books. Find the section on Quantums under Processes and Threads.

So I know nothing about networking and nothing about usermode programming.

But cold reading the code and the fine documentation

• At the beginning the function is being used to poll the events (timeout of zero). This is kind of weird but I guess it is expected
• The next interesting this is that when there is data there the function still says "timeout". So I'm wondering whether the IO completion fires an APC back to this thread in case maybe the final parameter (fAlertable) simply need to be true.
• Of course you need to be pretty happy that you are safe against any aynchrony that might happen as a result (a big deal in K-mode). If that was the case then you'd need to do the wait in a different thread and get that thread to signal another event and do a non interruptable wait with infinite timeout.

But this is about 10 minutes speculation based on zero insight either into cUrl or WSA.

This may not be on the right track for why the timeout is being hit because I know nothing about the internals of curl.

The WSAWaitForMultipleEvents is being used to wait until the socket is able to receive writes again correct? But, the FD_WRITE is only triggered if there has been a write attempt that was blocked. So, if the buffer sends without ever blocking wouldn't the WSAWaitForMultipleEvents always wait the timeout_ms duration for the next loop iteration in easy_transfer?

I set a breakpoint in Curl_send_plain and I'm never hitting a case where the bytes_written is -1 so I don't think the FD_WRITE would ever be signaled for my scenario.

This SO answer links to the relevant section about the behavior of FD_WRITE:
https://stackoverflow.com/questions/16479106/how-network-event-fd-write-is-generated-when-using-event-driven-sockets

This information is gold, thank you @tpodom. I will try to incorporate this into the next attempt, potentially via #6245.

@mback2k awesome, just tag me if there’s anything you’d like me to test!

Actually the pre-checking with Curl_poll/select should make it never check/wait for FD_WRITE on a socket via WSAWaitForMultipleEvents since select on Windows should emulate standard behavior and signal readiness for writing every time a send call would not block. So I would appreciate your help with testing #6245 which I just updated. cc @jay

I have a question, as I use libcurl 7.73.0 for our product on Windows.
I see the report, and the suggested fix or code revert applies to multi.c source file and to the function Curl_multi_wait.
If I use only the easy interface, will I also be effected by this issue, or is it relevant to the multi interface solely

@Eliyahu-Machluf this issue is already fixed in later releases.

It doesn't matter which public API you use to do transfers, they all use the same unified multi code underneath.

@bagder, thanks for the quick response.

Some more questions:

1. Our product uses libcurl as http client to upload files to hadoop (using its webhdfs interface).
   If I want to check if the product is effected from this issue, which scenario should I run? simply upload large files (using PUT http verb) and compare performance to built kit with older curl version (7.54)? it is related to specific http verb? to other specific parameters?

2. I saw also curl 7.73.0 for Windows upload performance is extremely bad when server only supports 64KB TCP windows. #6276 which is marked as duplicate of this issue. Does this issue relates to specific tcp windows size? The Windows default is 64KB, so I should see this issue on default Windows configuration, right? We use Windows-2016-Server and Windows-10 machines.

Thanks.

Since this issue is already fixed, I will not spend time on it anymore.

OK. Thanks.

@Eliyahu-Machluf if your product is using libcurl 7.73.0 on Windows then it is affected. The performance slowdown is not constant, it depends on network conditions and even then is somewhat arbitrary. I would skip the metrics and upgrade as soon as you can. Sorry for the inconvenience.

@mback2k - Will I be able to just take multi.c changes for fixing this performance issue on my libcurl 7.73 source branch and rebuild? It will be too risky now to take 7.74 release into our code. Are there special instructions to enable the performance changes in this module

You can just try to apply this PR, yes.

ok - 6146 is the issue #, where is the associated PR? Iam seeing a lot of threads for related issues in this page

So 7.74 and 7.75 have been fixed by reverting the problematic commits. And #6245 is the next attempt to implement the changes correctly, so you would have to apply all commits from that PR except the first one which reverts the reverting from 7.74.

There's likely a much smaller risk to upgrade to the next release rather than to have you custom-patch code with patches you don't seem to have full control or understanding of. I would strongly advice against doing that. Use released versions instead.