pkg/util/goschedstats/goschedstats_test_/goschedstats_test: pkg failed

[cockroach-teamcity]

pkg/util/goschedstats/goschedstats_test_/goschedstats_test.pkg failed on master @ 8cd9b03b04c3007dc40c3bdf6b10bbb1d15df430:

Parameters:

• attempt=1
• deadlock=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-48104

[srosenberg]

=== RUN   TestSchedStatsTickerShortPeriodOverride
-- Test timed out at 2025-02-25 09:21:02 UTC --
Execution result: https://tanzanite.cluster.engflow.com/actions/executions/ChCWcq5hxO9D9ILbVwFqIXHFEgdkZWZhdWx0GiUKIHqMdlHT1nzql3nkQRcZ9n4qn5spx-6vEuot6jXf9yoTEN8C

[srosenberg]

SIGSEGV after stressing via ./dev test -v pkg/util/goschedstats --stress --count 1000,

metamorphic: use COCKROACH_RANDOM_SEED=7509133097050672513 for reproduction
initialized metamorphic constant "parse-json-impl" with value 1
initialized metamorphic constant "span-reuse-rate" with value 76
=== RUN   TestNumRunnableGoroutines
--- PASS: TestNumRunnableGoroutines (0.00s)
=== RUN   TestSchedStatsTicker
--- PASS: TestSchedStatsTicker (1.56s)
=== RUN   TestSchedStatsTickerShortPeriodOverride
SIGSEGV: segmentation violation
PC=0x4574cb m=8 sigcode=2 addr=0xc0002f

Coincidentally, last failure of this test was during an attempt to upgrade to Go 1.23 [1].

[1] #132497 (comment)

[RaduBerinde]

I can't find an explanation for why the code doesn't work, when there is code that does similar things in runtime. Perhaps the fact that it's inside our program and not inside the runtime is making a difference. I will try to move the implementation to our modified go and just call it. This will be much easier to maintain anyway.

[RaduBerinde]

Ok, I found something interesting. Even if I stub out the numRunnableGoroutines function like below, I can still repro the failure with dev test pkg/util/goschedstats -f 'TestSchedStatsTicker$$' --stress --count 10000

func numRunnableGoroutines() (numRunnable int, numProcs int) {
	return 0, runtime.GOMAXPROCS(0)
}

[RaduBerinde]

Even more strange, one of the stacks show that we are running an init() function:

goroutine 1 gp=0x140000021c0 m=0 mp=0x10632f3c0 [running, locked to thread]:
runtime.systemstack_switch()
        src/runtime/asm_arm64.s:201 +0x8 fp=0x1400039f9e0 sp=0x1400039f9d0 pc=0x104d07a58
runtime.gcStart({0x10632f3c0?, 0x14000480000?, 0x100000?})
        GOROOT/src/runtime/mgc.go:791 +0x530 fp=0x1400039fac0 sp=0x1400039f9e0 pc=0x104ca5900
runtime.mallocgc(0x100000, 0x105912440, 0x1)
        GOROOT/src/runtime/malloc.go:1304 +0x96c fp=0x1400039fb60 sp=0x1400039fac0 pc=0x104cfbd3c
runtime.makeslice(0x1400039fbb8?, 0x104d2bbb4?, 0x1400039fc30?)
        GOROOT/src/runtime/slice.go:116 +0x50 fp=0x1400039fb90 sp=0x1400039fb60 pc=0x104d038a0
github.com/elastic/gosigar.kernProcargs(0xc826, 0x1400039fd28, 0x0, 0x0)
        com_github_cockroachdb_cockroach/external/com_github_elastic_gosigar/sigar_common_darwin.go:372 +0x5c fp=0x1400039fd00 sp=0x1400039fb90 pc=0x1052493cc
github.com/elastic/gosigar.(*ProcExe).Get(0x104d6ec60?, 0x0?)
        com_github_cockroachdb_cockroach/external/com_github_elastic_gosigar/sigar_common_darwin.go:354 +0x3c fp=0x1400039fd40 sp=0x1400039fd00 pc=0x1052492cc
github.com/cockroachdb/cockroach/pkg/util/debugutil.init.0.func1(0x0?)
        pkg/util/debugutil/debugutil.go:31 +0x44 fp=0x1400039fdf0 sp=0x1400039fd40 pc=0x105249d94
github.com/cockroachdb/cockroach/pkg/util/debugutil.init.0()
        pkg/util/debugutil/debugutil.go:45 +0x20 fp=0x1400039fe10 sp=0x1400039fdf0 pc=0x105249d10
runtime.doInit1(0x1062bdf80)
        GOROOT/src/runtime/proc.go:7321 +0xd4 fp=0x1400039ff40 sp=0x1400039fe10 pc=0x104cd8aa4
runtime.doInit(...)
        GOROOT/src/runtime/proc.go:7288
runtime.main()
        GOROOT/src/runtime/proc.go:254 +0x344 fp=0x1400039ffd0 sp=0x1400039ff40 pc=0x104cc8fe4
runtime.goexit({})

[cockroach-teamcity]

pkg/util/goschedstats/goschedstats_test_/goschedstats_test.pkg failed on master @ c44b3134d23d6037e1d5d5e12461a6914aa588e3:

Parameters:

• attempt=1
• deadlock=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[RaduBerinde]

I commented out the entire _test.go file, the init() function and added a dummy test, now it repros immediately with a segmentation fault.

I narrowed it down to this:

//go:linkname sched runtime.sched
var sched schedt

No longer repros after commenting out this (otherwise unused) declaration. It looks like the old method of using go:linkname and a prefix of the struct defintion doesn't work (though I don't understand the details).

[RaduBerinde]

Adding _ [2000]byte at the end of our schedt fixes it, so there's a problem with the hack of having a prefix of the struct. I believe we are using this in other places, we should do an audit.

In any case, the good solution here is to just move this code to the Go runtime, since it's only functional with our patched Go anyway.