BENCH: add ufunc argument parsing benchmarks.

[mhvk]

As requested in #11351 (comment)

[charris]

LGTM, but I'm not clear on what "# On purpose pick no-ops" means in all the cases, so could you make the comment more informative?

[mhvk]

OK, I wrote a more general comment for the whole class on the choice of arguments.

[eric-wieser]

I'd maybe be tempted to write something weirder like

class CallArgParsing(Benchmark):
    param_names = ['args']
    params = [
        ((x, y), {}),
        ((x, y, out), {}),
        ((x, y, (out,)), {}),
    ]
    def time_call(self, arg_pair):
        np.add(*arg_pair[0], **arg_pair[1])

This has the advantage of not incurring a variable number of self.attribute lookups in each test. It also means the tests will appear on the same graph, since they go through one function.

You'd need a separate benchmark group for reduce, but that's probably fine

[mhvk]

That sounds like a nice idea. (I wish asv worked better for me, so I could check the output etc....).

[pv]

How does it not work?

[mhvk]

@pv - checking on my office machine, it actually does work. The difference is that there I have python2.7 development stuff installed, while on my laptop, where I tried earlier, I have not. I'll look into where it decides to try python2.7 anyway.

[mhvk]

OK, well, that was simple, asv.conf.json has "pythons": ["2.7"],; see
https://github.com/numpy/numpy/blob/master/benchmarks/asv.conf.json#L38

A separate issue, though: #11458

[pv]

The default configuration in the file asv.conf.json is set to use python2.7.
Override with asv run --python=3.6.

[mhvk]

Indeed, works for asv, but not for --bench_compare - looking at runtests.py, I see that for the regular --bench command, python=same is passed on, but this is not the case for --bench_compare - would it make sense to change that default? If so, I'll make a quick PR.

[pv]

--python=same is short for --environment=existing:same which is different. I.e. it needs explicit version number.

[mhvk]

@eric-wieser - I updated the benchmarks per your suggestion; certainly the output of just running --bench or --bench_compare looks quite nice this way.

p.s. It also looks like there is a substantial regression in speed compared to 1.14.4 - much more than what my PRs are improving. Sigh.

[tylerjereddy]

I checked out the PR branch and ran the benchmarks locally with asv run -e -b "ArgParsing*" >& log.txt and the output looks sensible (using conda env & Python 2.7 + 3.6):

· Creating environments                                                            
· Discovering benchmarks                                                           
·· Uninstalling from conda-py2.7-six                                               
·· Building 5b672cbe for conda-py2.7-six                                           
·· Installing into conda-py2.7-six                                                 
· Running 4 total benchmarks (1 commits * 2 environments * 2 benchmarks)           
[  0.00%] · For numpy commit hash 5b672cbe:                                        
[  0.00%] ·· Building for conda-py2.7-six                                          
[  0.00%] ·· Benchmarking conda-py2.7-six                                          
[ 25.00%] ··· Running bench_ufunc.ArgParsing.time_add_arg_parsing             ok  
[ 25.00%] ····                                                                     
               ============================================================================ =============
                                                arg_kwarg                           
               ---------------------------------------------------------------------------- -------------
                                       ((array(1.), array(2.)), {})                            974±3ns   
                                 ((array(1.), array(2.), array(3.)), {})                     1.00±0.01μs 
                               ((array(1.), array(2.)), {'out': array(3.)})                  1.08±0.01μs 
                             ((array(1.), array(2.)), {'out': (array(3.),)})                   952±10ns  
                ((array(1.), array(2.)), {'subok': True, 'where': True, 'out': array(3.)})   1.00±0.01μs 
                                ((array(1.), array(2.)), {'subok': True})                    1.12±0.02μs 
                         ((array(1.), array(2.)), {'subok': True, 'where': True})            1.09±0.01μs 
                   ((array(1.), array(2.), array(3.)), {'subok': True, 'where': True})       1.04±0.01μs 
               ============================================================================ =============
                                                                                   
[ 50.00%] ··· Running ...ArgParsingReduce.time_add_reduce_arg_parsing         ok  
[ 50.00%] ····                                                                     
               ==================================================================== =============
                                            arg_kwarg                               
               -------------------------------------------------------------------- -------------
                                     ((array([0., 1.]),), {})                        1.61±0.02μs 
                                    ((array([0., 1.]), 0), {})                       1.69±0.01μs 
                                ((array([0., 1.]),), {'axis': 0})                    1.98±0.02μs 
                                 ((array([0., 1.]), 0, None), {})                    1.90±0.03μs 
                         ((array([0., 1.]),), {'dtype': None, 'axis': 0})            1.88±0.01μs 
                           ((array([0., 1.]), 0, None, array(0.)), {})               1.62±0.02μs 
                ((array([0., 1.]),), {'dtype': None, 'out': array(0.), 'axis': 0})   1.81±0.01μs 
                             ((array([0., 1.]),), {'out': array(0.)})                1.88±0.02μs 
               ==================================================================== =============
                                                                                   
[ 50.00%] ·· Building for conda-py3.6-six                                          
[ 50.00%] ·· Benchmarking conda-py3.6-six                                          
[ 75.00%] ··· Running bench_ufunc.ArgParsing.time_add_arg_parsing             ok  
[ 75.00%] ····                                                                     
               ============================================================================ =============
                                                arg_kwarg                           
               ---------------------------------------------------------------------------- -------------
                                       ((array(1.), array(2.)), {})                          1.11±0.03μs 
                                 ((array(1.), array(2.), array(3.)), {})                     1.23±0.02μs 
                               ((array(1.), array(2.)), {'out': array(3.)})                  1.24±0.03μs 
                             ((array(1.), array(2.)), {'out': (array(3.),)})                 1.05±0.01μs 
                ((array(1.), array(2.)), {'subok': True, 'where': True, 'out': array(3.)})   1.26±0.01μs 
                                ((array(1.), array(2.)), {'subok': True})                    1.37±0.01μs 
                         ((array(1.), array(2.)), {'subok': True, 'where': True})            1.32±0.02μs 
                   ((array(1.), array(2.), array(3.)), {'subok': True, 'where': True})       1.44±0.01μs 
               ============================================================================ =============
                                                                                   
[100.00%] ··· Running ...ArgParsingReduce.time_add_reduce_arg_parsing         ok  
[100.00%] ····                                                                     
               ==================================================================== =============
                                            arg_kwarg                               
               -------------------------------------------------------------------- -------------
                                     ((array([0., 1.]),), {})                        1.85±0.03μs 
                                    ((array([0., 1.]), 0), {})                       2.00±0.01μs 
                                ((array([0., 1.]),), {'axis': 0})                    2.26±0.01μs 
                                 ((array([0., 1.]), 0, None), {})                    1.98±0.02μs 
                         ((array([0., 1.]),), {'dtype': None, 'axis': 0})            2.32±0.06μs 
                           ((array([0., 1.]), 0, None, array(0.)), {})               2.06±0.02μs 
                ((array([0., 1.]),), {'dtype': None, 'out': array(0.), 'axis': 0})   2.50±0.02μs 
                             ((array([0., 1.]),), {'out': array(0.)})                2.27±0.01μs 
               ==================================================================== =============

[charris]

Thanks Marten.

[eric-wieser]

@pv: Will this actually be able to compare benchmarks? How is one benchmark matched against another? Tuples of ndarrays are unlikely to compare equal...

[pv]

Each parameter set is treated as a separate benchmark, if that is what you are asking.