5.0: Coverage report generation failure

[tyacbovi]

After the current release of 5.0a3, the coverage generation is failing.

I am using the following packages for running my tests:
Python 2.7.15, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
plugins: xdist-1.23.2, forked-0.2, cov-2.6.0

The failure stack trace:

.......................................................INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/hooks.py", line 258, in call
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 61, in
INTERNALERROR> firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR> gen.send(outcome)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/plugin.py", line 228, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/engine.py", line 297, in finish
INTERNALERROR> self.cov.save()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 527, in save
INTERNALERROR> data = self.get_data()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 581, in get_data
INTERNALERROR> if self._collector and self._collector.flush_data():
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/collector.py", line 425, in flush_data
INTERNALERROR> self.covdata.add_lines(abs_file_dict(self.data))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 231, in add_lines
INTERNALERROR> self._choose_lines_or_arcs(lines=True)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 277, in _choose_lines_or_arcs
INTERNALERROR> con.execute("update meta set has_lines = ?, has_arcs = ?", (lines, arcs))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 534, in execute
INTERNALERROR> raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
INTERNALERROR> CoverageException: Couldn't use data file '/home/circleci/project/.coverage.905ab52c4304.2293.575964': no such table: meta
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/hooks.py", line 258, in call
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 61, in
INTERNALERROR> firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR> gen.send(outcome)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/plugin.py", line 228, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/engine.py", line 297, in finish
INTERNALERROR> self.cov.save()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 527, in save
INTERNALERROR> data = self.get_data()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 581, in get_data
INTERNALERROR> if self._collector and self._collector.flush_data():
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/collector.py", line 425, in flush_data
INTERNALERROR> self.covdata.add_lines(abs_file_dict(self.data))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 231, in add_lines
INTERNALERROR> self._choose_lines_or_arcs(lines=True)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 277, in _choose_lines_or_arcs
INTERNALERROR> con.execute("update meta set has_lines = ?, has_arcs = ?", (lines, arcs))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 534, in execute
INTERNALERROR> raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
INTERNALERROR> CoverageException: Couldn't use data file '/home/circleci/project/.coverage.905ab52c4304.2291.551230': no such table: meta
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/hooks.py", line 258, in call
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 61, in
INTERNALERROR> firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR> gen.send(outcome)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/plugin.py", line 228, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/engine.py", line 297, in finish
INTERNALERROR> self.cov.save()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 527, in save
INTERNALERROR> data = self.get_data()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 581, in get_data
INTERNALERROR> if self._collector and self._collector.flush_data():
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/collector.py", line 425, in flush_data
INTERNALERROR> self.covdata.add_lines(abs_file_dict(self.data))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 231, in add_lines
INTERNALERROR> self._choose_lines_or_arcs(lines=True)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 277, in _choose_lines_or_arcs
INTERNALERROR> con.execute("update meta set has_lines = ?, has_arcs = ?", (lines, arcs))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 534, in execute
INTERNALERROR> raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
INTERNALERROR> CoverageException: Couldn't use data file '/home/circleci/project/.coverage.905ab52c4304.2288.788971': no such table: meta
... [100%]INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/hooks.py", line 258, in call
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/manager.py", line 61, in
INTERNALERROR> firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR> gen.send(outcome)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/plugin.py", line 228, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/pytest_cov/engine.py", line 249, in finish
INTERNALERROR> self.cov.stop()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 527, in save
INTERNALERROR> data = self.get_data()
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/control.py", line 581, in get_data
INTERNALERROR> if self._collector and self._collector.flush_data():
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/collector.py", line 425, in flush_data
INTERNALERROR> self.covdata.add_lines(abs_file_dict(self.data))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 231, in add_lines
INTERNALERROR> self._choose_lines_or_arcs(lines=True)
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 277, in _choose_lines_or_arcs
INTERNALERROR> con.execute("update meta set has_lines = ?, has_arcs = ?", (lines, arcs))
INTERNALERROR> File "/home/circleci/project/.tox/py27_script_plugin/lib/python2.7/site-packages/coverage/sqldata.py", line 534, in execute
INTERNALERROR> raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
INTERNALERROR> CoverageException: Couldn't use data file '/home/circleci/project/.coverage': no such table: meta

[nedbat]

Can you give me instructions for reproducing this problem?

[tyacbovi]

The relevant code is in this repo,
and this is the tox.ini.
Also a link to the circleci build

[tyacbovi]

Did not mention, but the pytest with coverage runs successfully for the 5.0a2 version.

[krisrogers]

We are observing this error in one repo, and a similar one (shown below) in another repo. Both were working with 5.02a.

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/coverage/sqldata.py", line 532, in execute
INTERNALERROR> return self.con.execute(sql, parameters)
INTERNALERROR> sqlite3.IntegrityError: UNIQUE constraint failed: file.path
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/pluggy/hooks.py", line 258, in call
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/pluggy/manager.py", line 61, in
INTERNALERROR> firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR> gen.send(outcome)
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/pytest_cov/plugin.py", line 228, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/pytest_cov/engine.py", line 167, in finish
INTERNALERROR> self.cov.stop()
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/coverage/control.py", line 527, in save
INTERNALERROR> data = self.get_data()
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/coverage/control.py", line 581, in get_data
INTERNALERROR> if self._collector and self._collector.flush_data():
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/coverage/collector.py", line 425, in flush_data
INTERNALERROR> self.covdata.add_lines(abs_file_dict(self.data))
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/coverage/sqldata.py", line 236, in add_lines
INTERNALERROR> file_id = self._file_id(filename, add=True)
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/coverage/sqldata.py", line 189, in _file_id
INTERNALERROR> cur = con.execute("insert into file (path) values (?)", (filename,))
INTERNALERROR> File "/home/kris/dev/kapiche/chrysalis/.tox/py35/lib/python3.5/site-packages/coverage/sqldata.py", line 534, in execute
INTERNALERROR> raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
INTERNALERROR> coverage.misc.CoverageException: Couldn't use data file '/home/kris/dev/kapiche/chrysalis/.coverage': UNIQUE constraint failed: file.path

[nedbat]

@krisrogers if you could provide reproduction instructions, it would be very helpful, thanks :)

[minrk]

I'm hitting this in my test suites, and I think I've narrowed it down to tests run with pytest-cov which launch subprocesses. Take the self-contained module mymodule.py:

# mymodule.py
from subprocess import Popen
import sys


def foo():
    pass


def bar():
    pass


def test_foo():
    foo()


def test_subprocess():
    """this test triggers coveragepy#716"""
    p = Popen([sys.executable, __file__])
    p.communicate()


if __name__ == "__main__":
    bar()

test_foo has no trouble, but running test_subprocess produces this error:

(test-coverage) benjaminrk[13:53]/tmp $ pytest --cov mymodule -vk subprocess mymodule.py
======================================================================================================================================================================= test session starts =======================================================================================================================================================================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /tmp/test-coverage/bin/python3
cachedir: .pytest_cache
rootdir: /private/tmp, inifile:
plugins: cov-2.6.0
collected 2 items / 1 deselected

mymodule.py::test_subprocess PASSED                                                                                                                                                                                                                                                                                                                         [100%]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/coverage/sqldata.py", line 532, in execute
INTERNALERROR>     return self.con.execute(sql, parameters)
INTERNALERROR> sqlite3.IntegrityError: UNIQUE constraint failed: file.path
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/pluggy/hooks.py", line 258, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/pytest_cov/plugin.py", line 228, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/pytest_cov/engine.py", line 167, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/coverage/control.py", line 527, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/coverage/control.py", line 581, in get_data
INTERNALERROR>     if self._collector and self._collector.flush_data():
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/coverage/collector.py", line 425, in flush_data
INTERNALERROR>     self.covdata.add_lines(abs_file_dict(self.data))
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/coverage/sqldata.py", line 236, in add_lines
INTERNALERROR>     file_id = self._file_id(filename, add=True)
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/coverage/sqldata.py", line 189, in _file_id
INTERNALERROR>     cur = con.execute("insert into file (path) values (?)", (filename,))
INTERNALERROR>   File "/tmp/test-coverage/lib/python3.6/site-packages/coverage/sqldata.py", line 534, in execute
INTERNALERROR>     raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
INTERNALERROR> coverage.misc.CoverageException: Couldn't use data file '/private/tmp/.coverage': UNIQUE constraint failed: file.path

============================================================================================================================================================= 1 passed, 1 deselected in 0.15 seconds ==============================================================================================================================================================

pip freeze:

atomicwrites==1.2.1
attrs==18.2.0
coverage==5.0a3
more-itertools==4.3.0
pluggy==0.7.1
py==1.6.0
pytest==3.8.2
pytest-cov==2.6.0
six==1.11.0

I don't know enough to say if this is an issue with coverage.py or something that should be updated in pytest-cov, but it seems like there's a race to create the lines corresponding to files touched in both processes. I don't know how touching the coverage database is coordinated between processes, but there is definitely a race where rows created by other processes between the initialization of _file_map() in _open_db and the addition of new lines in _file_id(), where _file_id assumes that _file_map is up-to-date.

[minrk]

#718 fixes this issue for me by using insert or ignore to resolve the race. I don't know if there are other issues that may be indicated by a stale cache of data from the database.

[krisrogers]

@nedbat Sorry, can't provide reproduction details; I'm observing this in a commercial code base; Here is the tox file though, if it helps

tox.ini.txt

[nedbat]

For the "UNIQUE constraint failed: file.path" problem: you need to use [run] parallel=True in a .coveragerc file to make things behave properly. Once you have that setting, the results are correct.