Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: repeated run pkg@version calls hit the proxy for no apparent reason #72944

Closed
mvdan opened this issue Mar 19, 2025 · 9 comments
Closed
Labels
BugReport Issues describing a possible bug in the Go implementation. GoCommand cmd/go NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Mar 19, 2025

Go version

go version go1.24.1 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/mvdan/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/mvdan/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build878653294=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/mvdan/go/pkg/mod'
GONOPROXY='github.com/cue-unity'
GONOSUMDB='github.com/cue-unity'
GOOS='linux'
GOPATH='/home/mvdan/go'
GOPRIVATE='github.com/cue-unity'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/mvdan/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Ran go run -x honnef.co/go/tools/cmd/[email protected] -version a number of times.

What did you see happen?

$ go run -x honnef.co/go/tools/cmd/[email protected] -version
WORK=/tmp/go-build3841505877
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.043s)
/home/mvdan/.cache/go-build/ce/cefc08a55ed89b9e6ccbf96cc8155a15c7b548b63404c0988cc5b9998be24db2-d/staticcheck -version
staticcheck 2024.1.1 (0.5.1)
$ go run -x honnef.co/go/tools/cmd/[email protected] -version
WORK=/tmp/go-build1725445701
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.044s)
/home/mvdan/.cache/go-build/ce/cefc08a55ed89b9e6ccbf96cc8155a15c7b548b63404c0988cc5b9998be24db2-d/staticcheck -version
staticcheck 2024.1.1 (0.5.1)
$ go run -x honnef.co/go/tools/cmd/[email protected] -version
WORK=/tmp/go-build1793441905
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.049s)
/home/mvdan/.cache/go-build/ce/cefc08a55ed89b9e6ccbf96cc8155a15c7b548b63404c0988cc5b9998be24db2-d/staticcheck -version
staticcheck 2024.1.1 (0.5.1)

What did you expect to see?

I would expect the repeated runs to not hit GOPROXY at all; the module version is fully cached locally via GOMODCACHE.

Note that this is separate from go run caching binaries now. Even without caching of binaries, I would expect the local module cache to be sufficient so that the build can happen without hitting GOPROXY.

@gabyhelp gabyhelp added the BugReport Issues describing a possible bug in the Go implementation. label Mar 19, 2025
@rogpeppe
Copy link
Contributor

Here's another interesting observation: how much the network is accessed depends a lot on whether GOPRIVATE is set or not (apparently regardless of the actual value of GOPRIVATE).

Here's a testscript reproducer:

env GOCACHE=$WORK/.cache
go env
# without GOPRIVATE
go run -x honnef.co/go/tools/cmd/[email protected] --version

# without GOPRIVATE but with cache
go run -x honnef.co/go/tools/cmd/[email protected] --version

# without GOPRIVATE but with cache (again, to be sure)
go run -x honnef.co/go/tools/cmd/[email protected] --version

# with GOPRIVATE
env GOPRIVATE=something.unrelated
go run -x honnef.co/go/tools/cmd/[email protected] --version

# with GOPRIVATE with cache
go run -x honnef.co/go/tools/cmd/[email protected] --version

I see this output from testscript -v, with some lengthy output removed for brevity.

> env GOCACHE=$WORK/.cache
> go env
[stdout]
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='$WORK/.cache'
GOCACHEPROG=''
GODEBUG=''
GOENV='/no-home/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=$WORK/.tmp/go-build1245423453=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='$WORK/.gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='$WORK/.gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/rogpeppe/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/no-home/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/rogpeppe/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

# without GOPRIVATE (8.745s)
> go run -x honnef.co/go/tools/cmd/[email protected] --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
go: downloading honnef.co/go/tools v0.5.1
WORK=$WORK/.tmp/go-build3168116774
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info
....
[7618 lines elided]
....
cp $WORK/b001/exe/staticcheck $WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck # internal
$WORK/b001/exe/staticcheck --version

# without GOPRIVATE but with cache (0.239s)
> go run -x honnef.co/go/tools/cmd/[email protected] --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build3961093243
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info: 404 Not Found (0.034s)
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info: 404 Not Found (0.034s)
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info: 404 Not Found (0.035s)
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info: 404 Not Found (0.035s)
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.013s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

# without GOPRIVATE but with cache (again, to be sure) (0.224s)
> go run -x honnef.co/go/tools/cmd/[email protected] --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build2190875445
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info: 404 Not Found (0.027s)
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info: 404 Not Found (0.028s)
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info: 404 Not Found (0.028s)
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info: 404 Not Found (0.028s)
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.013s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

# with GOPRIVATE (0.185s)
> env GOPRIVATE=something.unrelated
> go run -x honnef.co/go/tools/cmd/[email protected] --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build2012329811
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.082s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

# with GOPRIVATE with cache (0.173s)
> go run -x honnef.co/go/tools/cmd/[email protected] --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build3915661201
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.074s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

PASS

I see the same behaviour under Go tip as of go1.25-24b395119b

@seankhliao
Copy link
Member

maybe because of #59230

@rogpeppe
Copy link
Contributor

@seankhliao that would explain the hit on @latest, but not difference with respect to GOPRIVATE AFAICS.

@matloob
Copy link
Contributor

matloob commented Mar 19, 2025

So the reason we're hitting the proxy is to get the deprecations.

The GOPRIVATE behavior is a bit harder to explain but I think it's still correct. So what's going on is that when GOPRIVATE is set GOPROXY implicitly has a first element that will lookup only the GOPRIVATE matching paths directly before trying other proxies with the GOPRIVATE matching paths disabled.

When we're trying to figure out which module honnef.co/go/tools/cmd/[email protected] belongs to, we try all the prefixes to see which ones exist. In the non-GOPRIVATE case, we directly send all of them to the module proxy. That's the normal behavior you see. But in the GOPRIVATE case, we first try to only resolve the GOPRIVATE matching paths directly. I think the thing that introduces a bit of confusion here is that we're always allowed to use the module cache regardless of the setting of GOPRIVATE. So essentially, with GOPRIVATE set to something that doesn't match any of the staticcheck prefixes, we try to look them all up using the module cache first. The honnef.co/go/tools lookup succeeds so the operation of finding the modules that exist succeeds and doesn't fail over to the first real element of GOPROXY.

@cherrymui
Copy link
Member

Sounds like this works as intended? Is there any suggestion for improvement? Otherwise perhaps we can close this. Thanks.

@cherrymui cherrymui added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. GoCommand cmd/go labels Mar 20, 2025
@cherrymui cherrymui added this to the Backlog milestone Mar 20, 2025
@matloob
Copy link
Contributor

matloob commented Apr 2, 2025

Yes I think this is working as intended. In the discussion offline, @seankhliao suggested the proposal #43646 as a potential option to skip some of this work. That's probably the way we want to go.

@cherrymui
Copy link
Member

Thanks. Sounds like we can close this, and use #43646 (or new proposals) for potential improvements.

@cherrymui cherrymui closed this as not planned Won't fix, can't repro, duplicate, stale Apr 2, 2025
@rogpeppe
Copy link
Contributor

rogpeppe commented Apr 6, 2025

Sorry for the delayed response.

Yes I think this is working as intended. In the discussion offline, @seankhliao suggested the proposal #43646 as a potential option to skip some of this work. That's probably the way we want to go.

I don't think that GOPROXY=cache per se would really fit my use case. We have go run pkg@version statements in shell scripts and the like, and when the script is run, we don't know whether anything is installed yet. So to take advantage of #43646, we'd need to first run some command to pre-populate the cache, which seems like it would require knowing in advance (or scanning for) all the go run occurrences that might or might happen, because otherwise the command will fail when the content isn't available.

I guess you're going to say "just set GOPROXY=cache,https://proxy.golang.org,direct" and I guess that's a possibility, but I don't want to have to remember to set it for just this use case and I wouldn't want it to synthesise results for @latest.

So what's going on is that when GOPRIVATE is set GOPROXY implicitly has a first element that will lookup only the GOPRIVATE matching paths directly before trying other proxies with the GOPRIVATE matching paths disabled.

This feels to me like an implementation detail not a reason that it should work this way. AFAICS nothing in the module docs say anything about GOPRIVATE acting like an implicit first element. Instead, GOPRIVATE is always phrased as if it were orthogonal to the order of lookup (e.g. "GOPRIVATE — list of glob patterns of module path prefixes that should be considered private.").

In general, I at least do have a workaround for now, but explaining it to people will be odd: "set GOPRIVATE to something, anything, and go run will run faster".

In fact, given that the behaviour this behaviour is considered OK when GOPRIVATE has no relationship to any module that's being used, why not always just behave as if there's an implicit first element in GOPROXY that allows us to check the cache before doing anything else? Is there any down side to doing that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BugReport Issues describing a possible bug in the Go implementation. GoCommand cmd/go NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests

6 participants