--dist=worksteal + high core counts leads to hangs

[alex]

I'm able to consistently reproduce with https://github.com/pyca/cryptography on an M1 MBP, which has 10 cores (8 performance, 2 efficiency).

Specifically py.test -n 10 --dist worksteal --last-failed tests/x509/ consistently hangs.

It consistently works if I remove the --dist worksteal flag. Using lower core counts (e.g. -n 8) makes the hangs more intermittent, but the hangs still occur.

The hangs happen at the end of the test suite -- I believe after all the tests have run.

If there's any additional information I can provide, pleaese let me know.

[amezin]

So the problem is:

1. A worker has executed all of its tests, except the last one. Worker's run_one_test() is blocked on self.torun.get().
2. The same worker gets a steal request - scheduler's view of the queue is lagging behind, so it thinks there are still pending tests for this worker. But it's expected
3. What's not expected, however, is that steal() replaces the torun queue - while torun.get() is still blocked. So run_one_test() stays blocked on the old queue, and all shutdown requests are pushed to the new queue. Hm.