-
Notifications
You must be signed in to change notification settings - Fork 17.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: io, net: implement WriterTo for pipes #34624
Comments
Implements the WriterTo interface for io pipes to avoid intermediate buffers when copying with `io.Copy`. Updates golang#34624
Implements the WriterTo interface for net pipes to avoid intermediate buffers when copying with `io.Copy`. Fixes golang#34624
Change https://golang.org/cl/177977 mentions this issue: |
This seems fine in principle but I'm pretty scared by the big comment about recovering panics in the CL. Is that really necessary? Why? |
This discussion seems to be waiting for an answer to my query from Nov 27, namely why is that CL so complex and what does recovering panics have to do with it? |
I understand the panic now, having looked more closely at the code. But I don't really understand why this is an important optimization. The extra code is very subtle. It avoids the intermediate copy, but how often does io.Pipe get used in performance-sensitive contexts? It's weird, because io.Pipe already is zero-copy between the read and write ends. |
I apologize for being slow to respond. This hasn't been high on my priority list lately (not a good reason to leave you hanging).
Interfaces and abstractions. For example, let's say I have some form of abstract "router" where I can ask this router to open a stream to some subsystem. If the subsystem is within the same process, it can give both sides of the connection an As you say, writes will already be zero copy with respect to reads. However, the other side will still need an intermediate buffer if they want to forward the data anywhere. With this optimization, I can proxy the local "connection" (e.g., to an HTTP response, local file, or some other subsystem) without having to copy. On the other hand, this definitely isn't a critical optimization. I filed the PR because it seemed like an easy win but that was before the issues around panicing became apparent. |
It sounds like there is not a compelling reason to add this. Having extra interfaces and abstractions that amount to no-ops is not a great thing to do. In this case, instead of making the extra abstraction zero-copy, users should probably reach for removing an io.Pipe, as I mentioned above. This seems like a likely decline. |
I agree with closing this, however:
When possible and/or convenient. The motivation here was abstracting away the difference between remote and local endpoints without paying for this abstraction with additional overhead. The alternatives are to either pay for the abstraction with a copy (the current approach) or simply not abstract over this difference (leading to duplicate code for handling both cases). |
Sorry, I got confused by the closed proposal and set the mode wrong. We'll leave this open for one more week just to make sure it ends up in the minutes. But thanks for closing. |
No change in consensus, so declined. |
What version of Go are you using (
go version
)?Description
Currently,
io.Copy(pipew, piper)
requires an intermediate buffer and an intermediate copy. ImplementingWriteTo
on pipes would allow writes to be forwarded directly to the writer with no intermediate copies.Implementation: #32125
The text was updated successfully, but these errors were encountered: