Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cythonize the call loop #263

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

goodboy
Copy link
Contributor

@goodboy goodboy commented Jun 4, 2020

Resolves #104 and is something I wrote a few years ago after a discussion with @RonnyPfannschmidt.

I haven't integrated the _c_multicall() into the benchmark tests yet (starting to think I shouldn't have removed the multiple loops from #147 🙄) but this is at least a draft that builds the faster function.

Update: Above is all complete.

Interested to see what peeps think!

TODO:

@goodboy goodboy changed the title Cythonize call loop Cythonize the call loop Jun 4, 2020
@goodboy
Copy link
Contributor Author

goodboy commented Jun 4, 2020

Ahh yeah so there's an import cycle in this..

I need to break out the _Result into it's own module, any preferences on a name?
I was thinking _result.py ?

It's that or we move the _c_multicall() import further down the file?

@RonnyPfannschmidt
Copy link
Member

sounds fair

@goodboy
Copy link
Contributor Author

goodboy commented Jun 4, 2020

One other idea: given the .pyx code isn't yet truly written as native cython code we could just breakout _multicall() to a new module and then copy and cythonize it on build instead of having duplicate code in both files?

Might just make sense to actually play with writing properly tuned cython code to see if there's any benefit before jumping on that wagon tho.

@bluetech
Copy link
Member

bluetech commented Jun 4, 2020

I need to break out the _Result into it's own module, any preferences on a name? I was thinking _result.py ?

I proposed this before in #227 so 👍 from me

@goodboy
Copy link
Contributor Author

goodboy commented Jun 4, 2020

I proposed this before in #227 so +1 from me

@bluetech I might add it in in a new PR just so that it can come in before this one and because we're touching stuff for the 1.0 release that's importing it.

@goodboy
Copy link
Contributor Author

goodboy commented Jun 4, 2020

Got the benchmarks running with both.
See the results from my terminal:

screenshot-2020-06-04_14-58-44

@bluetech
Copy link
Member

bluetech commented Jun 4, 2020

I have been working a bit on pytest performance lately, so let me know once you believe that the implementation is semantically correct, and I'll do some profiling on pytest (when I get the chance).

@goodboy
Copy link
Contributor Author

goodboy commented Jun 4, 2020

@bluetech sounds good!

I'm just gonna put up a PR to split out the _Result and friends and then this will basically be ready once I rebase on that.

@goodboy
Copy link
Contributor Author

goodboy commented Jun 4, 2020

I rebased this onto #268 so once that's merged I'll rebase onto master again to factor out those commits.

try: # run impl and wrapper setup functions in a loop
teardowns = []
try:
for hook_impl in reversed(hook_impls):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder if there is a way to line up implementations better so we cna do less work and less work on calls,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RonnyPfannschmidt oh there definitely is. I actually have some ideas but it's going to require some rejigging of how implementations are stored/managed in the manger iirc and I think this may have something to do with the way we couple tracing to that management (see #262 and #217). I also think we should let @bluetech play with it in the context of pytest before we get too ahead of ourselves - micro-benchmarks shouldn't be our guiding light IMO.

I also thought about this a while back with ideas for the rework of internals.. let's see if I can find some notes.

@RonnyPfannschmidt another question I have is how to implement both of these implementations (cython and python) without too much duplicate work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@goodboy as cython has a python mode, where a annotated file is next to the implementation file, i beleive we can do it better these days (by implementing it in python, and then cythonizing those files

Copy link
Contributor Author

@goodboy goodboy Aug 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So then it is as simple as running cython on the original code?
Haven't looked at cython in ages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im going to sprint on this next week, i hope to get a sync with simon from datasette about async use of pluggy before so we can enable that as well

Tyler Goodlet and others added 10 commits June 5, 2020 16:10
Cythonize `pluggy.callers._multicall` which gives around a 2x speedup
according to the benchmark suite. Transform the `pluggy.callers`
module in a new package and move utils and the legacy call loop into
separate modules.
When cython is installed always rebuild the C sources, when
not installed use whatever C sources were included in the sdist.
This reverts commit 72948af since we
need this parametrization for testing the cythonized version of
`_multicall` aka `pluggy.caller.cythonized._c_multicall()`.
@bluetech
Copy link
Member

bluetech commented Jun 6, 2020

Running pytest's test suite against this branch (+ switching to _c_multicall) gives two failures:

Details
================================================================================== FAILURES ==================================================================================
________________________________________________________________ TestTracebackCutting.test_traceback_argsetup ________________________________________________________________

self = <collect.TestTracebackCutting object at 0x7f98a766a790>, testdir = <Testdir local('/tmp/pytest-of-ran/pytest-9/test_traceback_argsetup0')>

    def test_traceback_argsetup(self, testdir):
        testdir.makeconftest(
            """
            import pytest
    
            @pytest.fixture
            def hello(request):
                raise ValueError("xyz")
        """
        )
        p = testdir.makepyfile("def test(hello): pass")
        result = testdir.runpytest(p)
        assert result.ret != 0
        out = result.stdout.str()
        assert "xyz" in out
        assert "conftest.py:5: ValueError" in out
        numentries = out.count("_ _ _")  # separator for traceback entries
>       assert numentries == 0
E       assert 26 == 0

/home/ran/src/pytest/testing/python/collect.py:955: AssertionError
---------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-5.4.1.dev459+g10bfcf590.d20200606, py-1.8.1, pluggy-1.0.0.dev17+g05b57e7
rootdir: /tmp/pytest-of-ran/pytest-9/test_traceback_argsetup0
collected 1 item

test_traceback_argsetup.py E                                             [100%]

==================================== ERRORS ====================================
____________________________ ERROR at setup of test ____________________________

>   ???

src/pluggy/callers/cythonized.pyx:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
    ???
src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall
    ???
src/pluggy/callers/cythonized.pyx:14: in pluggy.callers.cythonized._c_multicall
    ???
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
    ???
src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

request = <SubRequest 'hello' for <Function test>>

    @pytest.fixture
    def hello(request):
>       raise ValueError("xyz")
E       ValueError: xyz

conftest.py:5: ValueError
=========================== short test summary info ============================
ERROR test_traceback_argsetup.py::test - ValueError: xyz
=============================== 1 error in 0.03s ===============================
______________________________________________________________________ test_yield_fixture_with_no_value ______________________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-ran/pytest-9/test_yield_fixture_with_no_value0')>

    def test_yield_fixture_with_no_value(testdir):
        testdir.makepyfile(
            """
            import pytest
            @pytest.fixture(name='custom')
            def empty_yield():
                if False:
                    yield
    
            def test_fixt(custom):
                pass
            """
        )
        expected = "E               ValueError: custom did not yield a value"
        result = testdir.runpytest()
        result.assert_outcomes(error=1)
>       result.stdout.fnmatch_lines([expected])
E       Failed: nomatch: 'E               ValueError: custom did not yield a value'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.8.3, pytest-5.4.1.dev459+g10bfcf590.d20200606, py-1.8.1, pluggy-1.0.0.dev17+g05b57e7'
E           and: 'rootdir: /tmp/pytest-of-ran/pytest-9/test_yield_fixture_with_no_value0'
E           and: 'collected 1 item'
E           and: ''
E           and: 'test_yield_fixture_with_no_value.py E                                    [100%]'
E           and: ''
E           and: '==================================== ERRORS ===================================='
E           and: '_________________________ ERROR at setup of test_fixt __________________________'
E           and: ''
E           and: '>   ???'
E           and: ''
E           and: 'src/pluggy/callers/cythonized.pyx:14: '
E           and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ '
E           and: 'src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall'
E           and: '    ???'
E           and: 'src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall'
E           and: '    ???'
E           and: 'src/pluggy/callers/cythonized.pyx:14: in pluggy.callers.cythonized._c_multicall'
E           and: '    ???'
E           and: 'src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall'
E           and: '    ???'
E           and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ '
E           and: ''
E           and: '>   ???'
E           and: 'E   ValueError: custom did not yield a value'
E           and: ''
E           and: 'src/pluggy/callers/cythonized.pyx:43: ValueError'
E           and: '=========================== short test summary info ============================'
E           and: 'ERROR test_yield_fixture_with_no_value.py::test_fixt - ValueError: custom did...'
E           and: '=============================== 1 error in 0.02s ==============================='
E       remains unmatched: 'E               ValueError: custom did not yield a value'

/home/ran/src/pytest/testing/python/fixtures.py:4346: Failed
---------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-5.4.1.dev459+g10bfcf590.d20200606, py-1.8.1, pluggy-1.0.0.dev17+g05b57e7
rootdir: /tmp/pytest-of-ran/pytest-9/test_yield_fixture_with_no_value0
collected 1 item

test_yield_fixture_with_no_value.py E                                    [100%]

==================================== ERRORS ====================================
_________________________ ERROR at setup of test_fixt __________________________

>   ???

src/pluggy/callers/cythonized.pyx:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
    ???
src/pluggy/callers/cythonized.pyx:43: in pluggy.callers.cythonized._c_multicall
    ???
src/pluggy/callers/cythonized.pyx:14: in pluggy.callers.cythonized._c_multicall
    ???
src/pluggy/callers/cythonized.pyx:64: in pluggy.callers.cythonized._c_multicall
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   ValueError: custom did not yield a value

src/pluggy/callers/cythonized.pyx:43: ValueError
=========================== short test summary info ============================
ERROR test_yield_fixture_with_no_value.py::test_fixt - ValueError: custom did...
=============================== 1 error in 0.02s ===============================

I haven't looked at them, might be just something on pytest's side.


And here is a small pytest benchmark that I've used recently, which is not egrigously micro-benchmarky:

import pytest
@pytest.mark.parametrize("x", range(5000))
def test_foo(x): pass

it fires 100,408 hooks (I think). I run it under cProfile (best of 5, output is trimmed at 1s cumtime):

Results

Before:

         10927116 function calls (10371875 primitive calls) in 9.935 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    365/1    0.007    0.000    9.947    9.947 {built-in method builtins.exec}
        1    0.000    0.000    9.947    9.947 __main__.py:1(<module>)
        1    0.000    0.000    9.841    9.841 __init__.py:146(console_main)
        1    0.000    0.000    9.841    9.841 __init__.py:101(main)
100221/56    0.068    0.000    9.795    0.175 manager.py:90(_hookexec)
100221/56    0.152    0.000    9.795    0.175 manager.py:84(<lambda>)
100221/56    0.826    0.000    9.795    0.175 callers.py:157(_multicall)
 100058/2    0.372    0.000    9.793    4.897 hooks.py:272(__call__)
        1    0.000    0.000    9.553    9.553 main.py:249(pytest_cmdline_main)
        1    0.000    0.000    9.553    9.553 main.py:188(wrap_session)
        1    0.000    0.000    9.366    9.366 main.py:253(_main)
        1    0.023    0.023    8.563    8.563 main.py:270(pytest_runtestloop)
     5000    0.040    0.000    7.552    0.002 runner.py:97(pytest_runtest_protocol)
     5000    0.039    0.000    6.668    0.001 runner.py:104(runtestprotocol)
    15000    0.090    0.000    6.610    0.000 runner.py:203(call_and_report)
    15002    0.110    0.000    4.980    0.000 runner.py:275(from_call)
    15000    0.057    0.000    4.606    0.000 runner.py:224(call_runtest_hook)
    15000    0.036    0.000    4.221    0.000 runner.py:239(<lambda>)
140057/120053 0.090   0.000    2.581    0.000 {built-in method builtins.next}
95050/95048  0.064    0.000    2.294    0.000 {method 'send' of 'generator' objects}
    30000    0.077    0.000    1.269    0.000 capture.py:707(item_capture)
    20003    0.029    0.000    1.122    0.000 contextlib.py:117(__exit__)

After:

         10287357 function calls (9752107 primitive calls) in 9.441 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    366/1    0.006    0.000    9.453    9.453 {built-in method builtins.exec}
        1    0.000    0.000    9.453    9.453 __main__.py:1(<module>)
        1    0.000    0.000    9.345    9.345 __init__.py:146(console_main)
        1    0.000    0.000    9.345    9.345 __init__.py:101(main)
100221/56    0.065    0.000    9.300    0.166 manager.py:90(_hookexec)
100221/56    0.079    0.000    9.300    0.166 manager.py:84(<lambda>)
100221/56    0.643    0.000    9.299    0.166 {built-in method pluggy.callers.cythonized._c_multicall}
 100058/2    0.310    0.000    9.297    4.649 hooks.py:268(__call__)
        1    0.000    0.000    9.056    9.056 main.py:249(pytest_cmdline_main)
        1    0.000    0.000    9.056    9.056 main.py:188(wrap_session)
        1    0.000    0.000    8.868    8.868 main.py:253(_main)
        1    0.023    0.023    8.073    8.073 main.py:270(pytest_runtestloop)
     5000    0.040    0.000    7.108    0.001 runner.py:97(pytest_runtest_protocol)
     5000    0.040    0.000    6.250    0.001 runner.py:104(runtestprotocol)
    15000    0.091    0.000    6.190    0.000 runner.py:203(call_and_report)
    15002    0.109    0.000    4.815    0.000 runner.py:275(from_call)
    15000    0.057    0.000    4.446    0.000 runner.py:224(call_runtest_hook)
    15000    0.035    0.000    4.056    0.000 runner.py:239(<lambda>)
    45007    0.033    0.000    1.901    0.000 {built-in method builtins.next}
    30000    0.076    0.000    1.268    0.000 capture.py:707(item_capture)
    20003    0.035    0.000    1.133    0.000 contextlib.py:117(__exit__)

So it's a 5% improvement. Note that cProfile slows down the total execution time but relatively it should be about right.

IMO, 5% is probably not enough to be worth it, given that Cython, C extensions, wheels etc. are a huge headache all around. But if the Cython implementation can be sped up further I'll be happy to check again.

@goodboy
Copy link
Contributor Author

goodboy commented Jun 6, 2020

So it's a 5% improvement.

Yeah nothing to write home about.

IMO, 5% is probably not enough to be worth it, given that Cython, C extensions, wheels etc. are a huge headache all around.

Agreed, at least not as the default packaging. I think that if we want to include it as an optional dependency once we can get a better speedup that's something to consider.

@RonnyPfannschmidt
Copy link
Member

What's the raw execution time on cryptography for example, or a test suite with 40k tests

@goodboy
Copy link
Contributor Author

goodboy commented Jun 6, 2020

Check this out.

@oremanj just pointed to mypyc in the trio gitter which might be worth investigating as well for other modules in pytest and pluggy.

I know @bluetech is gonna luv the type hints :)

@RonnyPfannschmidt
Copy link
Member

@goodboy mypyc cant do pluggy ^^ - i tried it before ^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants