Pure python package?

[rth]

If this was a pure python package it would have been significantly easier to install and use.

It seems that no one would be willing to spend time on building wheels for this package. Which means that to currently install this package on windows one needs MSVC compiler which is problematic.

The only code currently using cython is sklearn_extra/utils/_cyfht.pyx, but there is also of course the question of future contributions.

The workarounds could be,

• use numba as a dependency (there is a numba version for _cyfht.pyx). That would simplify maintenance a bit. Maybe @NicolasHug you would have an opinion after your experience with using numba for GBDT. Though it would make integrating methods back into scikit-learn require more work.
• make the cython compilation (and modules that depend on it) optional at build time, so that this package can still be installed without a compilers.

[NicolasHug]

I don't recommend numba for scikit-learn for now because:

1. error messages aren't helpful
2. compilation time (last time I checked caching wasn't possible, but apparently it's OK now, need to check again)
3. it's fairly unstable compared to cython. Bugs can be extremely hard to catch, and it takes time to realize that the bug comes from numba, and not from your code. I literally wasted days with pygbm on this.
4. API is not entirely stable yet (e.g. deprecation of reflected lists for typed lists)
5. A few features are missing, like being able to parallelize methods. It doesn't look like much, but it makes the code look like spaghetti after a while.

So, if the goal is to eventually have these packages in scikit-learn, I wouldn't recommend it for scikit-learn-contrib either.

OTOH, if we're not planning on integrating them in scikit-learn, that could be a neat idea. Numba is still a very cool tool and the more people use it, the better it's gonna get. That would also be the occasion for scikit-learn core-devs to have some more hands-on experience with numba.

---

third option would be to use conda-forge? I guess windows users mostly use conda anyway.

[rth]

Thanks for the feedback!

if the goal is to eventually have these packages in scikit-learn, I wouldn't recommend it for scikit-learn-contrib either.

I'm not sure that all of the estimators in this repo would be eventually included in scikit-learn.

Then we need some other way to make this package easily usable and installable. Trying to avoid cython code (for the cost of some performance) could be another way.

[rth]

third option would be to use conda-forge? I guess windows users mostly use conda anyway.

Yeah, that could be a possibility, but that would require to make some releases, which initially we didn't want to do.

We could make release and label them as alpha or beta, to avoid backward compatibility constraints though.

[amueller]

I would also rather go in the direction of doing some form of binary distribution.

[rth]

I would also rather go in the direction of doing some form of binary distribution.

That also works. Closing. Thanks for the feedback.