Can't run optimize benchmarks

[andyfaff]

@mdhaber trying to run the global benchmarks to test solutions for #11438. However, I can't actually start the benchmarks off:

$ export SCIPY_XSLOW=1
$ export SCIPY_GLOBAL_BENCH=AMGM
$ python runtests.py --bench optimize.BenchGlobal

gives

   STDOUT -------->
   
   STDERR -------->
   Traceback (most recent call last):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1315, in <module>
       main()
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1308, in main
       commands[mode](args)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1004, in main_discover
       list_benchmarks(benchmark_dir, fp)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 989, in list_benchmarks
       for benchmark in disc_benchmarks(root):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 887, in disc_benchmarks
       for module in disc_modules(root_name, ignore_import_errors=ignore_import_errors):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 869, in disc_modules
       for item in disc_modules(name, ignore_import_errors=ignore_import_errors):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 857, in disc_modules
       module = import_module(module_name)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/importlib/__init__.py", line 127, in import_module
       return _bootstrap._gcd_import(name[level:], package, level)
     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
     File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
     File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
     File "<frozen importlib._bootstrap_external>", line 728, in exec_module
     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
     File "/Users/anz/Documents/Andy/programming/scipy/benchmarks/benchmarks/optimize_linprog.py", line 24, in <module>
       slow = int(os.environ.get('SCIPY_XSLOW', 0))
   NameError: name 'os' is not defined

I tried fixing that one by importing os outside the try:except block then came to:

STDOUT -------->
   
   STDERR -------->
   Traceback (most recent call last):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1315, in <module>
       main()
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1308, in main
       commands[mode](args)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1004, in main_discover
       list_benchmarks(benchmark_dir, fp)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 989, in list_benchmarks
       for benchmark in disc_benchmarks(root):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 887, in disc_benchmarks
       for module in disc_modules(root_name, ignore_import_errors=ignore_import_errors):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 869, in disc_modules
       for item in disc_modules(name, ignore_import_errors=ignore_import_errors):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 857, in disc_modules
       module = import_module(module_name)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/importlib/__init__.py", line 127, in import_module
       return _bootstrap._gcd_import(name[level:], package, level)
     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
     File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
     File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
     File "<frozen importlib._bootstrap_external>", line 728, in exec_module
     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
     File "/Users/anz/Documents/Andy/programming/scipy/benchmarks/benchmarks/optimize_linprog.py", line 33, in <module>
       rr_methods = [_remove_redundancy, _remove_redundancy_dense,
   NameError: name '_remove_redundancy' is not defined

At this point I stopped.

[andyfaff]

I commented out the try:except block at https://github.com/scipy/scipy/blob/master/benchmarks/benchmarks/optimize_linprog.py#L7 to see why it was required:

   STDOUT -------->
   
   STDERR -------->
   Traceback (most recent call last):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1315, in <module>
       main()
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1308, in main
       commands[mode](args)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 1004, in main_discover
       list_benchmarks(benchmark_dir, fp)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 989, in list_benchmarks
       for benchmark in disc_benchmarks(root):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 887, in disc_benchmarks
       for module in disc_modules(root_name, ignore_import_errors=ignore_import_errors):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 869, in disc_modules
       for item in disc_modules(name, ignore_import_errors=ignore_import_errors):
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/asv/benchmark.py", line 857, in disc_modules
       module = import_module(module_name)
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/importlib/__init__.py", line 127, in import_module
       return _bootstrap._gcd_import(name[level:], package, level)
     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
     File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
     File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
     File "<frozen importlib._bootstrap_external>", line 728, in exec_module
     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
     File "/Users/anz/Documents/Andy/programming/scipy/benchmarks/benchmarks/optimize_linprog.py", line 12, in <module>
       from scipy.optimize.tests.test_linprog import lpgen_2d, magic_square
     File "/Users/anz/Documents/Andy/programming/scipy/build/testenv/lib/python3.7/site-packages/scipy/optimize/tests/test_linprog.py", line 10, in <module>
       from pytest import raises as assert_raises
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/pytest.py", line 6, in <module>
       from _pytest.assertion import register_assert_rewrite
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/_pytest/assertion/__init__.py", line 6, in <module>
       from _pytest.assertion import rewrite
     File "/Users/anz/miniconda3/envs/dev3/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 16, in <module>
       import atomicwrites
   ModuleNotFoundError: No module named 'atomicwrites'

Whilst I had pytest, it seems that its atomicwrites dependency wasn't installed. Once I'd reinstalled pytest all was well. However, having the try/except block masks the real reason why the benchmark suite wouldn't run. Could we remove the try/except block and have a hard fail instead?

[mdhaber]

Yes. I can't figure out why the try except is there. I'm guessing when I first started this I started with another file as a template and never looked back : / sorry for the trouble.

[andyfaff]

I'll make a PR

[peterbell10]

I guess this should be closed since gh-11443 is merged.

@mdhaber the benchmark you copied from was probably catching import errors so the benchmarks could run on older SciPy versions. I had to do the same thing in the FFT benchmarks so they work both with and without scipy.fft

scipy/benchmarks/benchmarks/fft_basic.py

Lines 13 to 18 in 832bdf3

	try:
	import scipy.fft as scipy_fft
	has_scipy_fft = True
	except ImportError:
	scipy_fft = {}
	has_scipy_fft = False

[tylerjereddy]

the benchmark you copied from was probably catching import errors so the benchmarks could run on older SciPy versions.

Yes, I think the try/except should actually be restored in this case, but only for scipy-specific features (not the os & numpy imports that were also guarded) that may not exist farther back in history. It does make debugging harder when running benchmarks in a targeted fashion, but the tradeoff for improved bench history is one the project has been making for a while I think.

[andyfaff]

Having the os + numpy imports inside the try/except block doesn't make a lot of sense anyway. os will be there, and it's a pretty good bet that numpy will be installed.

The solution isn't to reintroduce the try/except block in its original form. This is because if any of the imports fails then we'll get the NameError: name '_remove_redundancy' is not defined error. When an import can fail it's not good if there's then variables declared outside of functions:

rr_methods = [_remove_redundancy, _remove_redundancy_dense,
              _remove_redundancy_sparse]

which rely on one or more of the imports that didn't get picked up.

[andyfaff]

If running the benchmarks means pytest is going to be imported, then perhaps we need to install it for running the benchmarks (https://github.com/scipy/scipy/blob/master/benchmarks/asv.conf.json#L35)

[mdhaber]

I've run into these troubles running the benchmarks, too. I think I got around it without figuring out what was wrong.

The arguments in favor of try around the imports make sense. But they can't all be grouped in the same try block.
And I'm +1 for adding pytest in asv.conf.json if it's going to be imported.

[tylerjereddy]

Right, I'm saying it was premature to completely remove the block instead of adjust it.

[tylerjereddy]

You can see some reasonably granular try/except blocks in benchmarks here: https://github.com/scipy/scipy/blob/master/benchmarks/benchmarks/spatial.py

[mdhaber]

OK. What about something like this to avoid all the explicit try except blocks?

from importlib import import_module

def try_import(from_, name_=None):
    try:
        module = import_module(from_)
        if name_ is not None:
            return getattr(module, name_)
        return module
    except ImportError as e:
        print("Warning: " + str(e))

# import numpy
numpy = try_import("numpy")
print(numpy)

# import numpy as np
np = try_import("numpy")
print(np)

# from scipy.linalg import linprog
linprog = try_import("scipy.optimize", "linprog")
print(linprog)

# import not_found
not_found = try_import("not_found")

Produces:

<module 'numpy' from 'C:\\Users\\matth\\Miniconda3\\lib\\site-packages\\numpy\\__init__.py'>
<module 'numpy' from 'C:\\Users\\matth\\Miniconda3\\lib\\site-packages\\numpy\\__init__.py'>
<function linprog at 0x000002214DFA7AE8>
Warning: No module named 'not_found'