x/tools/gopls: macOS kernel panic after heavy file IO

[richtera]

Go version

go version go1.22.12 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN='/Users/andy/.local/share/mise/installs/go/1.22.12/bin'
GOCACHE='/Users/andy/Library/Caches/go-build'
GOENV='/Users/andy/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/andy/gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/andy/gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/andy/.local/share/mise/installs/go/1.22.12'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/andy/.local/share/mise/installs/go/1.22.12/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.12'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/andy/Development/go.mod'
GOWORK='/Users/andy/Development/my-conciliate-app/go.work'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/vp/5m13m6sx52g2y7zn2f006jdh0000gn/T/go-build3793407225=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Started up VSCode.

What did you see happen?

My computer panics 5 times now back to back. I couldn't really figure out what was going on at first, but then I narrowed it down to "too many open files" and then killing the go/gopls processes manually it stopped crashing. Then I reconfigured my vscode go setup using the following settings.json

{
  "go.toolsEnvVars": {
    "GOOS": "js",
    "GOARCH": "wasm"
  },
  "go.inferGopath": false,
  "go.useLanguageServer": true,
  "go.languageServerFlags": ["-rpc.trace"],
  "go.buildFlags": ["-tags=js wasm"],
  "go.delveConfig": {
    "dlvLoadConfig": {
      "followPointers": false,
      "maxVariableRecurse": 1,
      "maxStringLen": 120,
      "maxArrayValues": 25,
      "maxStructFields": 5
    }
  },
  "go.toolsManagement.autoUpdate": false,
  "go.diagnostic.enableAny": false,
  "go.lintOnSave": "off",
  "go.vetOnSave": "off",
  "go.modReferences": false,
  "go.survey.prompt": false,
  "[go]": {
    "editor.defaultFormatter": "golang.go"
  },
  "files.watcherExclude": {
    "**/.git/objects/**": true,
    "**/.git/subtree-cache/**": true,
    "**/node_modules/**": true,
    "**/vendor/**": true,
    "**/hardhat/**": true,
    "**/.next/**": true,
    "**/dist/**": true,
    "**/go/pkg/**": true,
    "**/*.wasm": true,
    "**/*.js": true,
    "**/*.ts": true,
    "**/*.tsx": true,
    "assets/**": true,
    "backup/**": true,
    "bin/**": true,
    "cache/**": true,
    "images/**": true,
    "logs/**": true,
    "system/**": true,
    "tmp/**": true,
    "webserver-configs/**": true
  }
}

and now it seems to no longer reboot.

What did you expect to see?

I think gopls should limit itself how many files it attempts to open. To me it feels that un-configured projects should not panic the computer. It took a few hours to figure out why it was panicing.

[adonovan]

My computer panics

Do you mean you got a kernel panic? Could you elaborate?

Could you provide the results of these commands? Thanks.

$ ulimit -n
$ sysctl kern.maxfilesperproc
$ sysctl kern.maxfiles
$ gopls stats

[richtera]

Yes I got multiple kernel panic on MacOS and my UI would just blick out and the computer would reboot. I already have my kernel limits set for normal nodejs development, but runing gopls pushed it to just crash the computer when vscode started up with a project containing one template go.mod and one pretty much empty go file (I was getting ready to write a npm compiled with tinygo as wasm) but then spent over an hour trying to figure out why computer was crashing finally figuring out that changing the vscode go config prevented it from crashing again. My current settings are

❯ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       5333
-n: file descriptors                66304
❯ gdu --inodes -s
99621   .
❯ find . -name "go.*" -o -name "*.go"
./go.work
./packages/lilypad-wrapper/go/go.mod
./packages/lilypad-wrapper/go/mymodule.go

[adonovan]

Thanks. Did you try the three other commands?

[richtera]

Sorry forgot to copy all of them.

❯ ulimit -n
1048575

❯ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 122880

❯ sysctl kern.maxfiles
kern.maxfiles: 524288

❯ gopls stats
Initializing workspace...     done (186.614417ms)
Gathering bug reports...      done (6.306583ms)
Querying memstats...          done (1.481875ms)
Querying workspace stats...   done (73.959µs)
Collecting directory info...  done (772.244875ms)
{
  "BugReports": [],
  "CacheDir": "/Users/andy/Library/Caches/gopls/b4908c84",
  "DirStats": {
    "Files": 87676,
    "TestdataFiles": 10,
    "GoFiles": 1,
    "ModFiles": 1,
    "Dirs": 12160
  },
  "GOARCH": "arm64",
  "GOOS": "darwin",
  "GOPACKAGESDRIVER": "",
  "GOPLSCACHE": "",
  "GoVersion": "go1.23.8",
  "GoplsVersion": "v0.18.1",
  "InitialWorkspaceLoadDuration": "186.614417ms",
  "MemStats": {
    "HeapAlloc": 101833920,
    "HeapInUse": 104095744,
    "TotalAlloc": 104967808
  },
  "WorkspaceStats": {
    "Files": {
      "Total": 4,
      "Largest": 572,
      "Errs": 0
    },
    "Views": [
      {
        "GoCommandVersion": "go1.22.12",
        "AllPackages": {
          "Packages": 1,
          "LargestPackage": 1,
          "CompiledGoFiles": 1,
          "Modules": 1
        },
        "WorkspacePackages": {
          "Packages": 1,
          "LargestPackage": 1,
          "CompiledGoFiles": 1,
          "Modules": 1
        },
        "Diagnostics": 2
      }
    ]
  }
}

[findleyr]

Gopls should never need to hold files open for a long period of time: we open project files and indexes concurrently, but (IIRC) both with fixed parallelism.

If the error reporting is accurate, this indicates a bug in gopls, or the go command.

Can you investigate with, for example, lsof -c gopls? Running that now, I see some shared object files, local telemetry files, and logs, but not much else. Can you check, and summarize what you see? (you may not want to share specific file names, but anything like "I see a ton of telemetry files", or "a lot of .go files that are opened" would help us investigate.

[richtera]

NOTE: Without the above configuration inside of vscode settings the computer only stays up for about 10 minutes before panicing, blanking the screen and then rebooting. So this list is with the reconfiguration in place.
I noticed the too many files, because Apple Mail also crashed and my shell was unable to even open starship to generate the next prompt or even able to load lsof for example.

❯  lsof -c gopls
COMMAND   PID USER   FD     TYPE             DEVICE SIZE/OFF                NODE NAME
gopls   46797 andy  cwd      DIR               1,18     1120           261473744 /Users/andy/Development/my-conciliate-app
gopls   46797 andy  txt      REG               1,18 34031346           293889054 /Users/andy/.local/share/mise/installs/go/1.22.12/bin/gopls
gopls   46797 andy  txt      REG               1,18  2288832 1152921500312522782 /usr/lib/dyld
gopls   46797 andy  txt      REG               1,18    16384           295896870 /Users/andy/Library/Application Support/go/telemetry/local/[email protected]
gopls   46797 andy    0u    unix 0xb29d2c57956854d6      0t0                     ->0xb593d01cae990c22
gopls   46797 andy    1u    unix 0xc048b87049aeeee7      0t0                     ->0x898f8cc976603c62
gopls   46797 andy    2u    unix 0xb951fecf85673239      0t0                     ->0x92e199e475028814
gopls   46797 andy    3u     REG               1,18    16384           295896870 /Users/andy/Library/Application Support/go/telemetry/local/[email protected]
gopls   46797 andy    4     PIPE 0x3b5f857069ac489e    16384                     ->0xbc145413e536f0d5
gopls   46797 andy    5     PIPE 0x3b5f857069ac489e    16384                     ->0xbc145413e536f0d5
gopls   46797 andy    6u  KQUEUE                                                 count=0, state=0xa
gopls   46798 andy  cwd      DIR               1,18     1760            72882933 /Users/andy/Library/Application Support/go/telemetry/local
gopls   46798 andy  txt      REG               1,18 34031346           293889054 /Users/andy/.local/share/mise/installs/go/1.22.12/bin/gopls
gopls   46798 andy  txt      REG               1,18    16384           295896870 /Users/andy/Library/Application Support/go/telemetry/local/[email protected]
gopls   46798 andy  txt      REG               1,18  2288832 1152921500312522782 /usr/lib/dyld
gopls   46798 andy    0     PIPE 0xbc145413e536f0d5    16384                     ->0x3b5f857069ac489e
gopls   46798 andy    1w     CHR                3,2      0t0                 336 /dev/null
gopls   46798 andy    2w     CHR                3,2      0t0                 336 /dev/null
gopls   46798 andy    3u     REG               1,18    16384           295896870 /Users/andy/Library/Application Support/go/telemetry/local/[email protected]

FYI since I reconfigured vscode...
❯ uptime 
10:38  up 1 day, 19 hrs, 1 user, load averages: 2.50 2.59 2.49

[findleyr]

@richtera sorry, I'd missed that the configuration fixes the problem.

So the issue is file watchers (lsof on the code process could confirm). For what it's worth, file watching is done by VS Code, not gopls (but gopls requests glob patterns to watch).

Need to think if there's anything we can do from gopls.

[richtera]

Hmmm, the reason I was blaming gopls and go was that those were the processes at 60% and 1000% CPU usage (yea crazy enough it literally was over 1000% which was weird enough to begin with.) But it's quite possible that vscode called it it a weird arg that pushed it off of a cliff :)

[adonovan]

1000% CPU means there are 10x as many runnable threads (not goroutines) as CPUs. Generally in gopls we try to avoid scheduling more than GOMAXPROCS goroutines of pure computation at once, and in any case I would not expect the process load average to climb much above 100% since GOMAXPROCS defaults to the number of cores. If you see gopls using that much CPU again, could you abort it (killall -ABRT gopls) and report the traceback that it prints (which can usually be found in VS Code's "Output > gopls (server)" pane)? Thanks. (BTW: have you set GOMAXPROCS in your environment?)

If this happens to the go command, it's harder to access the traceback, but we would also be interested to see it.