7.10.0 regression: `FAILED tests/test_process.py::ProcessTest::test_save_signal_usr1 - assert -9 == 137`

[mgorny]

Describe the bug
Starting with 7.10.0, I'm seeing the following test failure on Gentoo Linux amd64:

$ tox -e py313
py313: pip-25.1.1-py3-none-any.whl already present in /home/mgorny/.local/share/virtualenv/wheel/3.13/embed/3/pip.json
py313: install_deps> python -m pip install -U -r requirements/pip.pip -r requirements/pytest.pip
.pkg: install_requires> python -I -m pip install setuptools
.pkg: _optional_hooks> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py313: install_package_deps> python -m pip install -U 'tomli; python_full_version <= "3.11.0a6"'
py313: install_package> python -m pip install -U --force-reinstall --no-deps .tox/.tmp/package/1/coverage-7.10.1a0.dev1.tar.gz
py313: commands[0]> python igor.py zip_mods
py313: commands[1]> python igor.py remove_extension
py313: commands[2]> python igor.py test_with_core pytrace
=== CPython 3.13.5 (main Jun 12 2025 03:41:01) (gil) with Python tracer (.tox/py313/bin/python) ===
bringing up nodes...
............................................................................................................................... [  8%]
............................................................................................................................s.. [ 16%]
....................................................s.sss......s...s.ssss...................................................... [ 25%]
...........s..ss...s........................................................................................................... [ 33%]
............................................................................................................................... [ 42%]
............................................................................................................................... [ 50%]
..................................................................................s..................s......................... [ 59%]
................................................................................................................ss.s..s........ [ 67%]
...s....................................................................................................s.ssss.sssss..ssssss... [ 76%]
sssss...............s....................sF..................................s.............s................................... [ 84%]
............................................................................................................................... [ 93%]
.............................................s...................s.................................ss....                       [100%]
============================================================== FAILURES ===============================================================
__________________________________________________ ProcessTest.test_save_signal_usr1 __________________________________________________
[gw9] linux -- Python 3.13.5 /tmp/coveragepy/.tox/py313/bin/python

self = <tests.test_process.ProcessTest object at 0x7f8debaf5c20>

    @pytest.mark.skipif(env.WINDOWS, reason="This test is not for Windows")
    def test_save_signal_usr1(self) -> None:
        self.assert_doesnt_exist(".coverage")
        self.make_file("dummy_hello.py", """\
            import os
            import signal
    
            print(f"Sending SIGUSR1 to myself")
            os.kill(os.getpid(), signal.SIGUSR1)
            os.kill(os.getpid(), signal.SIGKILL)
    
            print("Done and goodbye")
            """)
>       out = self.run_command(
            "coverage run --save-signal=USR1 dummy_hello.py",
            status=-signal.SIGKILL,
        )

/tmp/coveragepy/tests/test_process.py:708: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.test_process.ProcessTest object at 0x7f8debaf5c20>, cmd = 'coverage run --save-signal=USR1 dummy_hello.py'

    def run_command(self, cmd: str, *, status: int = 0) -> str:
        """Run the command-line `cmd` in a subprocess.
    
        `cmd` is the command line to invoke in a subprocess. Returns the
        combined content of `stdout` and `stderr` output streams from the
        subprocess.
    
        Asserts that the exit status is `status` (default 0).
    
        See `run_command_status` for complete semantics.
    
        Use this when you need to test the process behavior of coverage.
    
        Compare with `command_line`.
    
        """
        if status < 0 and env.LINUX:
            # Mac properly returns -signal as the exit status. Linux returns 128 + signal.
            status = 128 - status
        actual_status, output = self.run_command_status(cmd)
>       assert actual_status == status
E       assert -9 == 137

/tmp/coveragepy/tests/coveragetest.py:424: AssertionError
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
Sending SIGUSR1 to myself
Saving coverage data...

======================================================= short test summary info =======================================================
FAILED tests/test_process.py::ProcessTest::test_save_signal_usr1 - assert -9 == 137
1 failed, 1450 passed, 51 skipped in 24.08s
py313: exit 1 (24.64 seconds) /tmp/coveragepy> python igor.py test_with_core pytrace pid=581799
  py313: FAIL code 1 (44.84=setup[19.88]+cmd[0.18,0.13,24.64] seconds)
  evaluation failed :( (45.78 seconds)

I've reproduced it with CPython 3.11.13, 3.13.5 and PyPy3.11 7.3.20 — and effectively both with C and Python tracers. Both with 7.10.0 release and git HEAD.

To Reproduce

tox :-)

$ git rev-parse HEAD
0f00d49a594c2dc3576714a0669fbdbcc4dd4c7e
$ coverage debug sys
-- sys -------------------------------------------------------
               coverage_version: 7.10.1a0.dev1
                coverage_module: /tmp/coveragepy/.tox/py313/lib/python3.13/site-packages/coverage/__init__.py
                           core: -none-
                        CTracer: unavailable
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: /tmp/coveragepy/.coveragerc
                                 /tmp/coveragepy/setup.cfg
                                 /tmp/coveragepy/tox.ini
                                 /tmp/coveragepy/pyproject.toml
                   configs_read: /tmp/coveragepy/tox.ini
                                 /tmp/coveragepy/pyproject.toml
                    config_file: None
                config_contents: -none-
                      data_file: -none-
                         python: 3.13.5 (main, Jun 12 2025, 03:41:01) [GCC 14.3.0]
                       platform: Linux-6.15.8-gentoo-dist-x86_64-AMD_Ryzen_5_3600_6-Core_Processor-with-glibc2.41
                 implementation: CPython
                          build: main
                                 Jun 12 2025 03:41:01
                    gil_enabled: True
                     executable: /tmp/coveragepy/.tox/py313/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 603694
                            cwd: /tmp/coveragepy
                           path: /tmp/coveragepy/.tox/py313/bin
                                 /usr/lib/python313.zip
                                 /usr/lib/python3.13
                                 /usr/lib/python3.13/lib-dynload
                                 /tmp/coveragepy/.tox/py313/lib/python3.13/site-packages
                    environment: HOME = /home/mgorny
                                 LIBXCB_ALLOW_SLOPPY_LOCK = 1
                   command_line: /tmp/coveragepy/.tox/py313/bin/coverage debug sys
         sqlite3_sqlite_version: 3.50.2
             sqlite3_temp_store: 0
        sqlite3_compile_options: ATOMIC_INTRINSICS=1, COMPILER=gcc-14.3.0, DEFAULT_AUTOVACUUM,
                                 DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
                                 DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
                                 DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
                                 DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
                                 DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
                                 DEFAULT_WORKER_THREADS=0, DIRECT_OVERFLOW_READ, ENABLE_API_ARMOR,
                                 ENABLE_BYTECODE_VTAB, ENABLE_COLUMN_METADATA, ENABLE_DBPAGE_VTAB,
                                 ENABLE_DBSTAT_VTAB, ENABLE_EXPLAIN_COMMENTS, ENABLE_FTS3,
                                 ENABLE_FTS3_PARENTHESIS, ENABLE_FTS4, ENABLE_FTS5, ENABLE_GEOPOLY,
                                 ENABLE_HIDDEN_COLUMNS, ENABLE_ICU, ENABLE_MATH_FUNCTIONS, ENABLE_MEMSYS5,
                                 ENABLE_NORMALIZE, ENABLE_OFFSET_SQL_FUNC, ENABLE_PREUPDATE_HOOK,
                                 ENABLE_RBU, ENABLE_RTREE, ENABLE_SESSION, ENABLE_STMTVTAB,
                                 ENABLE_STMT_SCANSTATUS, ENABLE_UNKNOWN_SQL_FUNCTION, ENABLE_UNLOCK_NOTIFY,
                                 ENABLE_UPDATE_DELETE_LIMIT, HAVE_ISNAN, MALLOC_SOFT_LIMIT=1024,
                                 MAX_ATTACHED=10, MAX_COLUMN=2000, MAX_COMPOUND_SELECT=500,
                                 MAX_DEFAULT_PAGE_SIZE=8192, MAX_EXPR_DEPTH=1000, MAX_FUNCTION_ARG=1000,
                                 MAX_LENGTH=1000000000, MAX_LIKE_PATTERN_LENGTH=50000,
                                 MAX_MMAP_SIZE=0x7fff0000, MAX_PAGE_COUNT=0xfffffffe, MAX_PAGE_SIZE=65536,
                                 MAX_SQL_LENGTH=1000000000, MAX_TRIGGER_DEPTH=1000,
                                 MAX_VARIABLE_NUMBER=32766, MAX_VDBE_OP=250000000, MAX_WORKER_THREADS=8,
                                 MUTEX_PTHREADS, SOUNDEX, SYSTEM_MALLOC, TEMP_STORE=1, THREADSAFE=1, USE_URI
$ pip freeze
attrs==25.3.0
colorama==0.4.6
coverage @ file:///tmp/coveragepy/.tox/.tmp/package/1/coverage-7.10.1a0.dev1.tar.gz#sha256=ecf266bcce96c61ce565d7444d7415eefaa80fdd2e270689b49ec43ebea66870
distlib==0.4.0
exceptiongroup==1.3.0
execnet==2.1.1
filelock==3.18.0
flaky==3.8.1
hypothesis==6.136.3
iniconfig==2.1.0
packaging==25.0
platformdirs==4.3.8
pluggy==1.6.0
Pygments==2.19.2
pytest==8.4.1
pytest-xdist==3.8.0
setuptools==80.9.0
sortedcontainers==2.4.0
tomli==2.2.1
typing_extensions==4.14.1
virtualenv==20.32.0

Expected behavior
Passing tests :-).

Additional context
This is on glibc 2.41, if that could be relevant.

[mgorny]

I'm curious why we're seeing this kind of difference in behavior, but I guess the simplest solution would be to use something like:

        assert actual_status in (status, 128 - status)

[thesamesam]

The test is from 17fa2ad (#1998).

cc @ark-g

[nedbat]

I'm curious why we're seeing this kind of difference in behavior, but I guess the simplest solution would be to use something like:

    assert actual_status in (status, 128 - status)

I am also curious. Documentation about expected status code in this case is murky. I think your proposed change is a good idea.

[mgorny]

My another idea was bash version, but downgrading all the way back to 3.2 still yields -9.

[nedbat]

Fixed in commit a4300a7.

[mgorny]

Well, in case anyone's wondering, I'm still digging.

What's clear is that shell's $? behaves the way described here:

$ sh -c 'kill -KILL $$' ; echo $?
Killed                     sh -c 'kill -KILL $$'
137

However, I can't get Python to actually behave the same way: FWICS at least back to 3.11, subprocess has been using os.waitstatus_to_exitcode(), and even if I nest shells, exit status does not seem to be altered. The only way to actually get 137 here would be to explicitly make shell exit with $?:

>>> import subprocess
>>> subprocess.run("sh -c 'kill -KILL $$'", shell=True)
CompletedProcess(args="sh -c 'kill -KILL $$'", returncode=-9)
>>> subprocess.run("sh -c 'kill -KILL $$'; exit $?", shell=True)
Killed
CompletedProcess(args="sh -c 'kill -KILL $$'; exit $?", returncode=137)

[zackw]

Some shells (I'm sure bash does this, and empirically zsh also seems to) have an optimization where they will exec the last command in a script rather than forking for it.

A shell that does this, effectively erases the nesting that's textually present in subprocess.run("sh -c 'kill -KILL $$'", shell=True); the process that was killed will report an exit status s satisfying the equation WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL directly to Python.

On the other hand, a shell that does not do this optimization will munge the exit status through $? and report WIFEXITED(s) && WEXITSTATUS(s) == 128 + SIGKILL to Python. I do not have any such shell on this computer, but @nedbat are you perhaps on Debian or a derivative? Because I wouldn't be surprised if dash, which is usually /bin/sh on those distributions, doesn't do this optimization.

(If this is the correct explanation for the phenomenon, assert actual_status in (status, 128 - status) seems to me like a sensible way to make the test suite not care which kind of shell is involved.)

[mgorny]

I'm using dash, and at least as of 0.5.12 it gave -SIGKILL. To be honest, my first suspicion is that dash is working differently than bash, but that wasn't it.