Implicit jobserver token is yielded by rayon #67385
Labels
A-parallel-compiler
Area: parallel compiler
C-bug
Category: This is a bug.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
Comments
lnicola
pushed a commit
to lnicola/rust-analyzer
that referenced
this issue
May 1, 2025
Add a jobserver proxy to ensure at least one token is always held This adds a jobserver proxy to ensure at least one token is always held by `rustc`. Currently with `-Z threads` `rustc` can temporarily give up all its tokens, causing `cargo` to spawn additional `rustc` instances beyond the job limit. The current behavior causes an issue with `cargo fix` which has a global lock preventing concurrent `rustc` instances, but it also holds a jobserver token, causing a deadlock when `rustc` gives up its token. That is fixed by this PR. Fixes rust-lang/rust#67385. Fixes rust-lang/rust#133873. Fixes rust-lang/rust#140093.
github-actions bot
pushed a commit
to rust-lang/miri
that referenced
this issue
May 3, 2025
Add a jobserver proxy to ensure at least one token is always held This adds a jobserver proxy to ensure at least one token is always held by `rustc`. Currently with `-Z threads` `rustc` can temporarily give up all its tokens, causing `cargo` to spawn additional `rustc` instances beyond the job limit. The current behavior causes an issue with `cargo fix` which has a global lock preventing concurrent `rustc` instances, but it also holds a jobserver token, causing a deadlock when `rustc` gives up its token. That is fixed by this PR. Fixes rust-lang/rust#67385. Fixes rust-lang/rust#133873. Fixes rust-lang/rust#140093.
github-actions bot
pushed a commit
to rust-lang/rustc-dev-guide
that referenced
this issue
May 5, 2025
Add a jobserver proxy to ensure at least one token is always held This adds a jobserver proxy to ensure at least one token is always held by `rustc`. Currently with `-Z threads` `rustc` can temporarily give up all its tokens, causing `cargo` to spawn additional `rustc` instances beyond the job limit. The current behavior causes an issue with `cargo fix` which has a global lock preventing concurrent `rustc` instances, but it also holds a jobserver token, causing a deadlock when `rustc` gives up its token. That is fixed by this PR. Fixes rust-lang/rust#67385. Fixes rust-lang/rust#133873. Fixes rust-lang/rust#140093.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Rayon is currently yielding the implicit token "owned" by the process which means we can end up in a situation where e.g. a
-j1build has more than 1 rustc running (even though only one of them is actively running, but they're all consuming memory and other resources, so this is a bug).The current intended fix is to stop yielding the implicit token by keeping track of the amount of tokens we've acquired/yielded and just skipping the yield (and eventual reacquire) for that implicit token.
cc #64750
The text was updated successfully, but these errors were encountered: