Missing features removal with SimpleImputer

[vitorsrg]

Code sample

In the sample code below, a column is removed from the dataset during the pipeline

>>> from sklearn.impute import SimpleImputer
>>> import numpy as np
>>> imp = SimpleImputer()
>>> imp.fit([[0, np.nan], [1, np.nan]])
>>> imp.transform([[0, np.nan], [1, 1]])
array([[0.],
       [1.]])

Problem description

Currently sklearn.impute.SimpleImputer silently removes features that are np.nan on every training sample.

This may cause further issues on pipelines because the dataset's shape has changed, e.g.

dataset[:, columns_to_impute_with_median] = imp.fit_transform(dataset[:, columns_to_impute_with_median])

Possible solutions

For the problematic features, either keep their values if valid or impute the fill_value during transform. I suggest adding a new parameter to trigger this behaviour with a warning highlighting the referred features.

As I'm willing to implement this feature, I look forward advices.

[rth]

Thanks for the report!

Currently sklearn.impute.SimpleImputer silently removes features that are np.nan on every training sample.

That's a fairly surprising (and I think undocumented) behavior. Though I imagine keeping columns with all 0s (or other fill_value) is not very helpful either, for most use cases (putting aside API consistency).

It's also seems dependent on the strategy,

from sklearn.impute import SimpleImputer

>>> X = pd.DataFrame({'a': [1, np.nan], 'b': [np.nan, np.nan]})
>>> SimpleImputer(strategy='median').fit_transform(X)
array([[1.],
       [1.]])
>>> SimpleImputer(strategy='constant').fit_transform(X)
array([[1., 0.],
       [0., 0.]])

TBH, I have mixed feeling between documenting it as is, fixing as a bug, or adding an optional parameter to switch between these two behaviors as you suggest. Maybe other reviewers have stronger options about it.

[jnothman]

I think if we solved the feature names issue this would be a much smaller problem...

[glemaitre]

This behaviour ring the bell. We should check the behaviour across the different imputers.

[vitorsrg]

The expected behaviour could be to keep the NaN, so we can use another SimpleImputer(strategy='constant') after trying to impute with the mean:

>>> from sklearn.impute import SimpleImputer
>>> import numpy as np

>>> train = [[0, np.nan], [1, np.nan]]
>>> test = [[0, np.nan], [1, 1]]

>>> imp1 = SimpleImputer(strategy='mean')
>>> imp2 = SimpleImputer(strategy='constant')

>>> imp1.fit(train)
>>> imp2.fit(train)

>>> imp1.transform(test)
array([[0. np.nan], 
       [1. 1.]])
>>> imp2.transform(imp1.transform(test))
array([[0., 0.],
       [1., 1.]])

[glemaitre]

Regarding the over imputer, I meant the KNNImputer and IterativeImputer. I recall a discussion about this in the IterativeImputer. If we are going to change the behaviour it would need to be consistent. I did not have time to look at it yet.

[micahjsmith]

chiming in to agree that my expected behavior in the original post is that the column with all np.nan should be passed through. Never would I expect the column to be silently dropped. Thanks to all for looking into this.

[jnothman]

Maybe you're right that we should abide by the expectation that imputation should do nothing to the order or set of features. I'm okay to change the default behaviour over time. I had advocated for the dropping feature approach because a zero-variance training feature is unhelpful in terms of machine learning... but this behaviour is unhelpful in terms of software development.

[jnothman]

I'd happily review a pull request (given time) making the behaviour optional and perhaps changing the default

[glemaitre]

FYI, the behaviour is consistent across imputers:

from sklearn.impute import SimpleImputer
from sklearn.impute import KNNImputer
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
import numpy as np

for Imputer in [SimpleImputer, KNNImputer, IterativeImputer]:
    imp = Imputer()
    imp.fit([[0, np.nan], [1, np.nan]])
    X_trans = imp.transform([[0, np.nan], [1, 1]])
    print(imp.__class__.__name__)
    print(X_trans)

SimpleImputer
[[0.]
 [1.]]
KNNImputer
[[0.]
 [1.]]
IterativeImputer
[[0.]
 [1.]]

If we are going to change the behaviour, it needs to be shared between all imputers.