Add permutation based feature importance?

[amueller]

I think adding permutation based feature importances would be cool:
https://link.springer.com/article/10.1186%2F1471-2105-8-25

There is a python package that does it for our random forests with bagging:
https://github.com/parrt/random-forest-importances

But I'd rather like to see a generic permutation based importance score with cross-validation or hold-out.
I think this would be great analysis tool.

One easy way to implement it would be to provide a function for plotting etc. But Ideally we'd be able to use it in feature selection, I think, so we'd need to create a meta-estimator PermutationImportanceCV that only provides feature_importances_, I guess, so it can be wrapped with SelectFromModel?

[amueller]

Also see this blog post:
http://parrt.cs.usfca.edu/doc/rf-importance/index.html

I'm not suggesting we change our RF feature importances, but having a more expensive but possibly higher quality alternative would be great.

[jnothman]

something in this direction is implemented in eli5 too

[glemaitre]

I'm not suggesting we change our RF feature importances, but having a more expensive but possibly higher quality alternative would be great.

It was something we thought about with @ogrisel.

Moreover, the implementation describes in the blog post is model agnostic if I remember right.

[amueller]

Their implementation is not model agnostic, but the general algorithm is. And I'd be happy to upstream from eli5 given that this seems a relatively natural and accepted method.

[jnothman]

IIRC elli5 discovered their implementation involved making decisions, i.e. it might not be completely straightforward to implement in a universal way​

[kv-cache]

I am waiting for this method to be included. We need reliable results more than anything even if it's not fast. We prefer not be wrong anytime. I implemented it myself for my model.(for a competition)

[jnothman]

use eli5 for now

[datajanko]

I think PermutationImportanceCV is a good approach. I also see the advantage of a uniform interface to all functions. The question is, wouldn't a keyword argument for the random forest classes, to use the a different feature importance algorithm? We also provide multiple impurity measures. So why not maintain multiple importance measures?

[datajanko]

the rfpimp package allows to treat multiple features as one. I think it would make sense to include this as well (maybe later). What are your thoughts on that? I think, it would require changes in SelectFromModel. But this also would raise immediately the question, if one could be more flexible when splitting features in trees. E.g. defining some kind of feature group one could enforce that e.g. latitude and longitude appearing always together in splits.

[datajanko]

So instead of permuting, I tried an additional approach presented in the articles: dropping columns.

To show some code: The FeatureImportanceCV could roughly look like this:

class FeatureImportanceCV(BaseEstimator, MetaEstimatorMixin):

def __init__(self, estimator, cv=KFold(3)):
    self.estimator = estimator
    self.cv = cv
    
def fit(self, X, y):
    n_splits = self.cv.get_n_splits()
    fold_importances = np.zeros((n_splits, X.shape[1]))
    for i, (train, test) in enumerate(self.cv.split(y)):
        baseline_fold_i = self.estimator.fit(X[train], y[train]).score(X[test], y[test])
        except_feature_fold_i = [(self.estimator.fit(np.delete(X[train], j, axis=1),y[train])
                                                     .score(np.delete(X[test], j, axis=1), y[test])) 
                               for j in np.arange(X.shape[1])]
        fold_importances[i, :] = baseline_fold_i - except_feature_fold_i
    self.feature_importances_ = np.mean(fold_importances, axis=0)

Would this be the way to go? Of course the list comprehension should be replaces by something less dense. Permuting can be integrated here easily as well (except for maybe introducing more .copy()calls).

What should the class be capable of? Am I doing something completely wrong, just by looking at it.

I was able to run this on my machine with:

X, y = make_classification(n_samples=10, n_features=5, n_clusters_per_class=1, n_informative=3)
ficv = FeatureImportanceCV(LogisticRegression())
ficv.fit(X, y)
ficv.feature_importances_
array([0.38888889, 0.        , 0.19444444, 0.        , 0.11111111])

Additionally, the call to SelectFromModel works as well:

sfm = SelectFromModel(ficv).fit(X, y)
sfm.transform(X)

chooses a subset of the data

array([[-0.58617384, -1.30229985],
       [-0.52552405, -1.07458527],
       [ 0.94055998, -0.91529262],
       [-1.04839412, -0.2912939 ],
       [ 0.22273055, -1.36454126],
       [ 1.32253391, -1.68018815],
       [-0.13443558, -0.66608385],
       [ 0.29920044,  0.44704055],
       [ 1.94086301, -3.04870667],
       [ 0.4591779 ,  0.22705557]])

[parrt]

I do a drop column thing too in rfpimp. It gives slightly different results but takes longer due to all the retraining.

[datajanko]

compared to what? I think doing the random permutations and re_fitting should have the same complexity. You should only see the performance gains when you are computing the feature importance out-of-bag. But this is what you meant, right?

[parrt]

Well, isn't it the case that training a model is vastly more expensive than using the model? It is always that way in my experience, except for kNN. If you have 50 features you have to retrain the model 50 times for drop col importance, which could take overnight to train even once. Permuting and running predictions should be many times faster.

[datajanko]

Of course it is, and I'd really like to see the feature directly in sklearn for random forests. However, this feature importance approach might be useful also for different classifiers. Using for example dask (and dask-ml) one might be able to scale this quite easily.