sample_weights array can't be used with GridSearchCV

[shenberg]

The internal cross-validation isn't aware of sample weights, so and exception is thrown if a sample_weights sequence is passed to the grid search, because fit_grid_point does not split the weights into training and test sets.

[ndawe]

I have added support for sample_weight in GridSearchCV in #1574. Hopefully I can get that merged soon.

[shenberg]

Thanks, that was extremely fast turn-around.

As a meta-question, is sampling weighted? e.g. Is it better for stratified folds to try maintain the ratio of total weight per class instead of the number of samples?

[amueller]

sample_weight support was merged long ago

[jnothman]

@amueller, sample_weight is still not supported as an argument to fit in *SearchCV

[amueller]

There is fit_params

[stephen-hoover]

@amueller , I'm running into a similar issue. It's possible to set a sample_weights array as an instance attribute via the GridSearchCV.fit_params dictionary, and then everything works fine. But this fails in nested cross-validation with model_selection.cross_val_predict, because the GridSearchCV.fit (and RandomizedSearchCV.fit) method doesn't accept fit parameters.

Is there a reason why the BaseSearchCV subclasses can't accept fit parameters? Having to set data-dependent fit parameters as instance attributes appears to contradict http://scikit-learn.org/stable/developers/contributing.html#fitting .

I would be willing to make that change (adding fit parameters) if you'd accept that PR.

[jnothman]

Yes, it's broken. One issue is that we need to be clear whether sample_weight is being passed only to scoring, only to fit, or both. Feel free to propose and champion a solution.

[ManasHardas]

fit_params is deprecated since 0.19 and will be removed in 0.21
How else to make "sample_weights" a part of cross-validation?

[amueller]

@ManasHardas pass them to fit.

[farfan92]

@amueller Trying your solution when using RandomizedSearchCV, with a RandomForestClassifier.

Attempting to pass: fit_params ={'sample_weight':s_weights}
to the .fit method of RandomizedSearchCV results in

TypeError: fit() got an unexpected keyword argument 'fit_params'

But will still manage to run with the deprecation warning when passed to the constructor instead.

[jnothman]

just pass keyword arguments to fit these days

[farfan92]

I see, should have more closely looked at doc

[rishabhgit]

Hi @amueller , @jnothman ,

Has this issue been fixed? I'm trying to run Randomized Search CV with sample_weights both as a scoring param and fit param. Here is a code snippet:

scorer = make_scorer(r2_score,sample_weight=weights)
rs = RandomizedSearchCV(estimator=rf, param_distributions=param_grid, n_iter=3, 
                           scoring = scorer,
                           n_jobs=-1, cv=3)
    
rs = rs.fit(X_train, y_train) 

But, RandomizedSearchCV does not split the sample weights column when splitting the data into train and test sets. Here's some output about the array shapes and the error:

X_train shape  (349978, 367)
y_train shape  (349978,)
Sample weight shape  (349978,)

ValueError: Found input variables with inconsistent numbers of samples: [116660, 116660, 349978]

[jnothman]

Has this issue been fixed?

No we don't currently support weighted scoring in cross validation. Sorry. :'( Soon?

[doriang102]

Is this still unresolved? GridSearchCV and RandomSearchCV do not throw an error when sample_weight is passed to the fit method, yet the cross validation seems to ignore them entirely.

[jnothman]

Yes, they cannot at present be passed to scoring (without making them part of X), unfortunately.

[doriang102]

Thanks. Is there any current workaround other than oversampling the class in a way way proportional to the weights?

[jnothman]

Well if you make sample_weight the first column in X all you need is to make scorers that extract that (lambda est, X, y: orig_scorer(est, X[:, 1:], y, sample_weight=X[:, 0])) and use a Pipeline with a FunctionTransformer(lambda X: X[:, 1:]) at the start. But it's certainly a hack. (And you should probably use something other than lambdas.

[david-r-wasserman]

Now that this issue has been fixed, the new feature should be documented at https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html and https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html.

Also, this only works for sample_weights, right? If the estimator's fit() function has other params that are sample-specific, those still won't work, or will they? This should be made clear in the documentation.

[lpossberg]

Has this issue been fixed?

No we don't currently support weighted scoring in cross validation. Sorry. :'( Soon?

Hi @jnothman ,
is there any progress concerning the weighted scoring in cross validation?

[Demetrio92]

I have a work-around for evaluating with weights. It's somewhat inefficient, but works.

from sklearn.metrics import accuracy_score
from sklearn.utils import compute_sample_weight
from sklearn.metrics import make_scorer


def weighted_accuracy_eval(y_pred, y_true, **kwargs):
    balanced_class_weights_eval = compute_sample_weight(
        class_weight='balanced',
        y=y_true
    )
    out = accuracy_score(y_pred=y_pred, y_true=y_true, sample_weight=balanced_class_weights_eval, **kwargs)
    return out


weighted_accuracy_eval_skl = make_scorer(weighted_accuracy_eval)

gridsearch = GridSearchCV(
    estimator=model,
    scoring=weighted_accuracy_eval,
    param_grid=paramGrid,
)

cv_result = gridsearch.fit(
    X_train,
    y_train,
    **fit_params
)

---

@lpossberg @jnothman