test_all_releases fails when run in isolation

[bowlofeggs]

bodhi/tests/server/test_models.py::TestRelease::test_all_releases fails if you run it by itself, but passes when run with other tests:

 bowlofeggs  (e) bodhi  ~  devel  bodhi  time vagrant ssh -c "py.test-3 --no-cov -vx bodhi/tests/server/test_models.py -k test_all_releases"
/usr/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:6: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/usr/lib/python3.7/site-packages/more_itertools/more.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Counter, defaultdict, deque, Sequence
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.7.3, pytest-3.6.4, py-1.5.4, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/vagrant/bodhi, inifile: setup.cfg
plugins: cov-2.5.1
collected 315 items / 311 deselected                                                                                                                                                                                                         

bodhi/tests/server/test_models.py::TestRelease::test_all_releases FAILED                                                                                                                                                               [ 25%]

================================================================================================================== FAILURES ==================================================================================================================
_______________________________________________________________________________________________________ TestRelease.test_all_releases ________________________________________________________________________________________________________

self = <bodhi.tests.server.test_models.TestRelease testMethod=test_all_releases>

    def test_all_releases(self):
        releases = model.Release.all_releases()
    
        state = ReleaseState.from_string(list(releases.keys())[0])
>       self.assertIn('long_name', releases[state.value][0], releases)
E       IndexError: list index out of range

bodhi/tests/server/test_models.py:732: IndexError
WARNING: Coverage disabled via --no-cov switch!
============================================================================================================== warnings summary ==============================================================================================================
bodhi/tests/server/test_models.py::TestRelease::test_all_releases
  /usr/local/lib/python3.7/site-packages/pyramid_debugtoolbar/tbtools.py:35: DeprecationWarning: Flags not at the start of the expression b'^(.*?)$(?m)'
    _line_re = re.compile(br'^(.*?)$(?m)')
  /usr/lib64/python3.7/site-packages/markupsafe/__init__.py:13: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping
  /usr/lib64/python3.7/site-packages/sqlalchemy/orm/scoping.py:107: SAWarning: At least one scoped session is already present.  configure() can not affect sessions that have already been created.
    "At least one scoped session is already present. "
  /usr/lib/python3.7/site-packages/defusedxml/ElementTree.py:68: DeprecationWarning: The html argument of XMLParser() is deprecated
    _XMLParser.__init__(self, html, target, encoding)
  /usr/lib/python3.7/site-packages/dogpile/cache/util.py:31: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
    args = inspect.getargspec(fn)

<undetermined location>
  Coverage disabled via --no-cov switch!

-- Docs: http://doc.pytest.org/en/latest/warnings.html
============================================================================================ 1 failed, 311 deselected, 6 warnings in 3.10 seconds ============================================================================================
Connection to 192.168.122.222 closed.

real	0m4.780s
user	0m0.614s
sys	0m0.083s

This means it is depending on a change from some other test, and it means that other tests aren't cleaning up their state changes when they finish.

[sebwoj]

The same with TestReleaseModular, TestReleaseContainer and TestReleaseFlatpak.

The problem is when we run first TestCase(or only one TestCase), where we initialise TestApp:

bodhi/bodhi/tests/server/base.py

Line 184 in 639e140

_app = TestApp(main({}, testing='guest', **self.app_settings))

TestApp calls:

bodhi/bodhi/server/__init__.py

Line 198 in 639e140

def main(global_config, testing=None, session=None, **settings):

where call to:

bodhi/bodhi/server/__init__.py

Line 327 in 639e140

models.Release.all_releases()

takes place.

In case of TestRelease*.test_all_releases tests, new Release in setUp method is created after call to
TestApp init, so this new Release object is not catched by Release.all_releases().
releases variable in test is empty, this is why we get:
IndexError: list index out of range

Similar problem: #2177