[MRG] ENH Show split/candidate ids in verbose output of SearchCV #16935

cmarmo · 2020-04-15T22:58:56Z

Reference Issues/PRs

Continue and close #8281

What does this implement/fix? Explain your changes.

Address @TomDLT comments.

Any other comments?

This PR is in the 0.23 milestone, I don't know if this is still needed, I just like the idea that perhaps he would have been happy to see this PR merged.

…rbose > 9

cmarmo · 2020-04-18T22:32:31Z

The example in #8281:

from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV

verbose=2
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svr = svm.SVC()
clf = GridSearchCV(svr, parameters, n_jobs=8, verbose=verbose)
clf.fit(iris.data, iris.target)

gives:
verbose = 1

Fitting 5 folds for each of 4 candidates, totalling 20 fits

verbose = 2

Fitting 5 folds for each of 4 candidates, totalling 20 fits
[CV] START C=1, kernel='rbf'....................................................
[CV] END ..................................C=1, kernel='rbf'; total time=   0.0s
[CV] START C=1, kernel='linear'.................................................
[CV] START C=1, kernel='linear'.................................................
[CV] END ...............................C=1, kernel='linear'; total time=   0.0s
[CV] END ...............................C=1, kernel='linear'; total time=   0.0s
[CV] START C=1, kernel='linear'.................................................
[CV] START C=1, kernel='rbf'....................................................
[CV] END ..................................C=1, kernel='rbf'; total time=   0.0s
[CV] END ...............................C=1, kernel='linear'; total time=   0.0s
[CV] START C=1, kernel='rbf'....................................................
[CV] END ..................................C=1, kernel='rbf'; total time=   0.0s
[CV] START C=10, kernel='linear'................................................
[CV] START C=10, kernel='linear'................................................
[CV] END ..............................C=10, kernel='linear'; total time=   0.0s
[CV] START C=10, kernel='linear'................................................
[CV] END ..............................C=10, kernel='linear'; total time=   0.0s
[CV] START C=10, kernel='linear'................................................
[CV] START C=10, kernel='linear'................................................
[CV] END ..............................C=10, kernel='linear'; total time=   0.0s
[CV] START C=10, kernel='rbf'...................................................
[CV] END ..............................C=10, kernel='linear'; total time=   0.0s
[CV] START C=10, kernel='rbf'...................................................
[CV] END .................................C=10, kernel='rbf'; total time=   0.0s
[CV] START C=10, kernel='rbf'...................................................
[CV] END ..............................C=10, kernel='linear'; total time=   0.0s
[CV] START C=10, kernel='rbf'...................................................
[CV] END .................................C=10, kernel='rbf'; total time=   0.0s
[CV] END .................................C=10, kernel='rbf'; total time=   0.0s
[CV] START C=10, kernel='rbf'...................................................
[CV] END .................................C=10, kernel='rbf'; total time=   0.0s
[CV] END .................................C=10, kernel='rbf'; total time=   0.0s
[CV] START C=1, kernel='linear'.................................................
[CV] END ...............................C=1, kernel='linear'; total time=   0.0s
[CV] START C=1, kernel='rbf'....................................................
[CV] END ..................................C=1, kernel='rbf'; total time=   0.0s
[CV] START C=1, kernel='linear'.................................................
[CV] END ...............................C=1, kernel='linear'; total time=   0.0s
[CV] START C=1, kernel='rbf'....................................................
[CV] END ..................................C=1, kernel='rbf'; total time=   0.0s

verbose = 3

Fitting 5 folds for each of 4 candidates, totalling 20 fits
[CV 4/5] START C=1, kernel='linear'.............................................
[CV 4/5] END .......C=1, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 4/5] START C=1, kernel='rbf'................................................
[CV 4/5] END ..........C=1, kernel='rbf'; score: (test=0.933) total time=   0.0s
[CV 5/5] START C=1, kernel='rbf'................................................
[CV 5/5] END ..........C=1, kernel='rbf'; score: (test=1.000) total time=   0.0s
[CV 1/5] START C=10, kernel='linear'............................................
[CV 1/5] END ......C=10, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 2/5] START C=10, kernel='linear'............................................
[CV 2/5] END ......C=10, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 3/5] START C=10, kernel='linear'............................................
[CV 3/5] END ......C=10, kernel='linear'; score: (test=0.900) total time=   0.0s
[CV 4/5] START C=10, kernel='linear'............................................
[CV 4/5] END ......C=10, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 5/5] START C=10, kernel='linear'............................................
[CV 5/5] END ......C=10, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 1/5] START C=10, kernel='rbf'...............................................
[CV 1/5] END .........C=10, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 2/5] START C=10, kernel='rbf'...............................................
[CV 2/5] END .........C=10, kernel='rbf'; score: (test=1.000) total time=   0.0s
[CV 3/5] START C=10, kernel='rbf'...............................................
[CV 3/5] END .........C=10, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 4/5] START C=10, kernel='rbf'...............................................
[CV 4/5] END .........C=10, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 5/5] START C=10, kernel='rbf'...............................................
[CV 5/5] END .........C=10, kernel='rbf'; score: (test=1.000) total time=   0.0s
[CV 1/5] START C=1, kernel='linear'.............................................
[CV 1/5] END .......C=1, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 2/5] START C=1, kernel='linear'.............................................
[CV 2/5] END .......C=1, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 3/5] START C=1, kernel='linear'.............................................
[CV 3/5] END .......C=1, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 2/5] START C=1, kernel='rbf'................................................
[CV 2/5] END ..........C=1, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 1/5] START C=1, kernel='rbf'................................................
[CV 1/5] END ..........C=1, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 3/5] START C=1, kernel='rbf'................................................
[CV 3/5] END ..........C=1, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 5/5] START C=1, kernel='linear'.............................................
[CV 5/5] END .......C=1, kernel='linear'; score: (test=1.000) total time=   0.0s

verbose = 10

Fitting 5 folds for each of 4 candidates, totalling 20 fits
[CV 2/5; 1/4] START C=1, kernel='linear'........................................
[CV 2/5; 1/4] END ..C=1, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 4/5; 2/4] START C=1, kernel='rbf'...........................................
[CV 4/5; 2/4] END .....C=1, kernel='rbf'; score: (test=0.933) total time=   0.0s
[CV 5/5; 2/4] START C=1, kernel='rbf'...........................................
[CV 5/5; 2/4] END .....C=1, kernel='rbf'; score: (test=1.000) total time=   0.0s
[CV 1/5; 3/4] START C=10, kernel='linear'.......................................
[CV 1/5; 3/4] END .C=10, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 2/5; 3/4] START C=10, kernel='linear'.......................................
[CV 2/5; 3/4] END .C=10, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 3/5; 3/4] START C=10, kernel='linear'.......................................
[CV 3/5; 3/4] END .C=10, kernel='linear'; score: (test=0.900) total time=   0.0s
[CV 4/5; 3/4] START C=10, kernel='linear'.......................................
[CV 4/5; 3/4] END .C=10, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 5/5; 3/4] START C=10, kernel='linear'.......................................
[CV 5/5; 3/4] END .C=10, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 1/5; 4/4] START C=10, kernel='rbf'..........................................
[CV 1/5; 4/4] END ....C=10, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 2/5; 4/4] START C=10, kernel='rbf'..........................................
[CV 2/5; 4/4] END ....C=10, kernel='rbf'; score: (test=1.000) total time=   0.0s
[CV 3/5; 4/4] START C=10, kernel='rbf'..........................................
[CV 3/5; 4/4] END ....C=10, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 4/5; 4/4] START C=10, kernel='rbf'..........................................
[CV 4/5; 4/4] END ....C=10, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 5/5; 4/4] START C=10, kernel='rbf'..........................................
[CV 5/5; 4/4] END ....C=10, kernel='rbf'; score: (test=1.000) total time=   0.0s
[CV 1/5; 1/4] START C=1, kernel='linear'........................................
[CV 1/5; 1/4] END ..C=1, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 1/5; 2/4] START C=1, kernel='rbf'...........................................
[CV 1/5; 2/4] END .....C=1, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 4/5; 1/4] START C=1, kernel='linear'........................................
[CV 4/5; 1/4] END ..C=1, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 5/5; 1/4] START C=1, kernel='linear'........................................
[CV 5/5; 1/4] END ..C=1, kernel='linear'; score: (test=1.000) total time=   0.0s
[CV 3/5; 1/4] START C=1, kernel='linear'........................................
[CV 3/5; 1/4] END ..C=1, kernel='linear'; score: (test=0.967) total time=   0.0s
[CV 2/5; 2/4] START C=1, kernel='rbf'...........................................
[CV 2/5; 2/4] END .....C=1, kernel='rbf'; score: (test=0.967) total time=   0.0s
[CV 3/5; 2/4] START C=1, kernel='rbf'...........................................
[CV 3/5; 2/4] END .....C=1, kernel='rbf'; score: (test=0.967) total time=   0.0s

cmarmo · 2020-04-19T08:49:03Z

I have removed the old verbosity test because the verbose behavior has changed. Let me know if this is not correct.

NicolasHug

thanks @cmarmo , made a quick review but looks good

(IDK if you don't know or just forgot but you can use syntax highlighting in github using

```py 
code here
```

)

sklearn/model_selection/tests/test_validation.py

sklearn/model_selection/_search.py

sklearn/model_selection/_validation.py

cmarmo · 2020-04-19T13:12:29Z

(IDK if you don't know or just forgot but you can use syntax highlighting in github using

Thanks! :)

jnothman · 2020-05-20T13:21:34Z

Personally I find showing both START and END (and params and score!) for each fit very verbose already. So I'm not sure that verbose is a very helpful control for users.

jnothman

but in terms of adding the feature, this is fine.

Please add an entry to the change log at doc/whats_new/v0.24.rst. Like the other entries there, please reference this pull request with :pr: and credit yourself (and other contributors if applicable) with :user:

cmarmo · 2020-05-22T10:58:04Z

Personally I find showing both START and END (and params and score!) for each fit very verbose already. So I'm not sure that verbose is a very helpful control for users.

I have removed the START thing from all levels except 10. So verbose 1 and 10 are as above.
verbose = 2 gives

Fitting 5 folds for each of 4 candidates, totalling 20 fits
[CV] END .................................C=1, kernel=linear; total time=   0.0s
[CV] END ....................................C=1, kernel=rbf; total time=   0.0s
[CV] END ....................................C=1, kernel=rbf; total time=   0.0s
[CV] END ................................C=10, kernel=linear; total time=   0.0s
[CV] END ................................C=10, kernel=linear; total time=   0.0s
[CV] END ................................C=10, kernel=linear; total time=   0.0s
[CV] END ................................C=10, kernel=linear; total time=   0.0s
[CV] END ................................C=10, kernel=linear; total time=   0.0s
[CV] END ...................................C=10, kernel=rbf; total time=   0.0s
[CV] END ...................................C=10, kernel=rbf; total time=   0.0s
[CV] END ...................................C=10, kernel=rbf; total time=   0.0s
[CV] END ...................................C=10, kernel=rbf; total time=   0.0s
[CV] END ...................................C=10, kernel=rbf; total time=   0.0s
[CV] END ....................................C=1, kernel=rbf; total time=   0.0s
[CV] END ....................................C=1, kernel=rbf; total time=   0.0s
[CV] END .................................C=1, kernel=linear; total time=   0.0s
[CV] END .................................C=1, kernel=linear; total time=   0.0s
[CV] END .................................C=1, kernel=linear; total time=   0.0s
[CV] END .................................C=1, kernel=linear; total time=   0.0s
[CV] END ....................................C=1, kernel=rbf; total time=   0.0s

verbose = 3 gives

Fitting 5 folds for each of 4 candidates, totalling 20 fits
[CV 1/5] END ............C=1, kernel=rbf; score: (test=0.967) total time=   0.0s
[CV 4/5] END ............C=1, kernel=rbf; score: (test=0.933) total time=   0.0s
[CV 5/5] END ............C=1, kernel=rbf; score: (test=1.000) total time=   0.0s
[CV 1/5] END ........C=10, kernel=linear; score: (test=1.000) total time=   0.0s
[CV 2/5] END ........C=10, kernel=linear; score: (test=1.000) total time=   0.0s
[CV 3/5] END ........C=10, kernel=linear; score: (test=0.900) total time=   0.0s
[CV 4/5] END ........C=10, kernel=linear; score: (test=0.967) total time=   0.0s
[CV 5/5] END ........C=10, kernel=linear; score: (test=1.000) total time=   0.0s
[CV 1/5] END ...........C=10, kernel=rbf; score: (test=0.967) total time=   0.0s
[CV 2/5] END ...........C=10, kernel=rbf; score: (test=1.000) total time=   0.0s
[CV 3/5] END ...........C=10, kernel=rbf; score: (test=0.967) total time=   0.0s
[CV 4/5] END ...........C=10, kernel=rbf; score: (test=0.967) total time=   0.0s
[CV 5/5] END ...........C=10, kernel=rbf; score: (test=1.000) total time=   0.0s
[CV 2/5] END .........C=1, kernel=linear; score: (test=1.000) total time=   0.0s
[CV 1/5] END .........C=1, kernel=linear; score: (test=0.967) total time=   0.0s
[CV 3/5] END .........C=1, kernel=linear; score: (test=0.967) total time=   0.0s
[CV 4/5] END .........C=1, kernel=linear; score: (test=0.967) total time=   0.0s
[CV 2/5] END ............C=1, kernel=rbf; score: (test=0.967) total time=   0.0s
[CV 5/5] END .........C=1, kernel=linear; score: (test=1.000) total time=   0.0s
[CV 3/5] END ............C=1, kernel=rbf; score: (test=0.967) total time=   0.0s

Let me know if this is satisfying or I should revert. Thanks!

jnothman · 2020-05-24T08:59:20Z

I prefer no START. Undecided on candidate number. Might benefit from notes in the user guide to help users choose.

cmarmo · 2020-06-04T13:51:43Z

The check failure does not depend on my changes.

cmarmo · 2020-06-11T16:04:19Z

@NicolasHug maybe you are available to finalize this one? Thanks!

ogrisel

LGTM as well. I think all comments by other reviewers have been addressed. Merging.

Thank you very much @cmarmo!

jnothman · 2020-06-21T00:15:20Z

Finally! Thanks @cmarmo :)

…kit-learn#16935)

raghavrv and others added 15 commits February 14, 2017 18:58

ENH show split/candidate ids in verbose output of SearchCV

162f3d9

FIX stupidity

bd6a775

REV use no message for params=None; Explicitly test None

53aeb49

ENH Right align score/time; Don't print 'params'; Print cand # for ve…

a0dc7b0

…rbose > 9

TST Test the verbose output of _fit_and_score extensively

5cbf466

FIX Don't pass verbose to Parallel (joblib)

7cb1602

FIX don't pass verbose to Parallel in SearchCV too

90d4758

ENH Simplify; current_fit_msg --> progress_msg; Use START/END

e0703de

ENH Use %r for param_values; Sort param by keys

f50af3e

TST/ENH Test against explicit values; Test for string param

4b2bbfe

MERGE master

2abb2ce

Merge with upstream.

f3e3b74

Fix errors.

181e6e9

Update verbose display. Tests still failing.

da96cf3

Address comments. Test still failing.

4b53993

github-actions bot added the module:model_selection label Apr 15, 2020

cmarmo added 3 commits April 16, 2020 01:14

Fix linting errors.

38cada1

Fix tests.

f3a1e79

Fix linting errors.

9a9662d

cmarmo changed the title ~~[WIP] ENH Show split/candidate ids in verbose output of SearchCV~~ [MRG] ENH Show split/candidate ids in verbose output of SearchCV Apr 18, 2020

Remove old verbosity test.

6898aa7

cmarmo changed the title ~~[MRG] ENH Show split/candidate ids in verbose output of SearchCV~~ [WIP] ENH Show split/candidate ids in verbose output of SearchCV Apr 19, 2020

NicolasHug reviewed Apr 19, 2020

View reviewed changes

cmarmo added 4 commits April 19, 2020 18:05

Address some of the comments. Tests are still WIP.

87a3dc8

Fix linting and formatting errors.

eccfa85

Fix linting errors again...

5577829

Fix linting errors again...

8671d57

cmarmo added 2 commits April 20, 2020 11:23

Homogenize candidate nomenclature.

aadf94f

Sync with upstream.

86685d5

cmarmo added the Waiting for Reviewer label May 3, 2020

jnothman approved these changes May 20, 2020

View reviewed changes

cmarmo removed the Waiting for Reviewer label May 22, 2020

cmarmo added 2 commits May 22, 2020 11:16

Merge branch 'master' into gridsearchcv_verbose_enh

7883c0f

Remove start on low verbose. Add what's new entry.

6d875b6

cmarmo added 5 commits May 25, 2020 00:11

Merge branch 'master' into gridsearchcv_verbose_enh

d755523

Add description of versbose in the user guide.

cd4830e

Fix trailing whitespace.

5ec9bd9

Resolve conflicts.

4b16aed

Resolve conflicts.

a1d070b

cmarmo added the Waiting for Reviewer label Jun 4, 2020

Force build.

2e56789

jnothman approved these changes Jun 5, 2020

View reviewed changes

Sync with upstream.

6f0a820

ogrisel approved these changes Jun 19, 2020

View reviewed changes

ogrisel merged commit fc0d2c3 into scikit-learn:master Jun 19, 2020

dsandeep0138 pushed a commit to dsandeep0138/scikit-learn that referenced this pull request Jun 21, 2020

[MRG] ENH Show split/candidate ids in verbose output of SearchCV (sci…

f22cf48

…kit-learn#16935)

cmarmo deleted the gridsearchcv_verbose_enh branch June 21, 2020 11:45

rubywerman pushed a commit to MLH-Fellowship/scikit-learn that referenced this pull request Jun 24, 2020

[MRG] ENH Show split/candidate ids in verbose output of SearchCV (sci…

6eae6d0

…kit-learn#16935)

viclafargue pushed a commit to viclafargue/scikit-learn that referenced this pull request Jun 26, 2020

[MRG] ENH Show split/candidate ids in verbose output of SearchCV (sci…

024128b

…kit-learn#16935)

jayzed82 pushed a commit to jayzed82/scikit-learn that referenced this pull request Oct 22, 2020

[MRG] ENH Show split/candidate ids in verbose output of SearchCV (sci…

b8794a0

…kit-learn#16935)

jpauwels mentioned this pull request Mar 25, 2022

Fix verbosity regression in cross-validation grid search #22946

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[MRG] ENH Show split/candidate ids in verbose output of SearchCV #16935

[MRG] ENH Show split/candidate ids in verbose output of SearchCV #16935

Uh oh!

cmarmo commented Apr 15, 2020 •

edited

Loading

Uh oh!

cmarmo commented Apr 18, 2020 •

edited

Loading

Uh oh!

cmarmo commented Apr 19, 2020

Uh oh!

NicolasHug left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmarmo commented Apr 19, 2020

Uh oh!

jnothman commented May 20, 2020

Uh oh!

jnothman left a comment

Uh oh!

cmarmo commented May 22, 2020

Uh oh!

jnothman commented May 24, 2020 via email

Uh oh!

cmarmo commented Jun 4, 2020

Uh oh!

cmarmo commented Jun 11, 2020 •

edited

Loading

Uh oh!

ogrisel left a comment

Uh oh!

jnothman commented Jun 21, 2020 via email

Uh oh!

Uh oh!

Uh oh!

[MRG] ENH Show split/candidate ids in verbose output of SearchCV #16935

[MRG] ENH Show split/candidate ids in verbose output of SearchCV #16935

Uh oh!

Conversation

cmarmo commented Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

cmarmo commented Apr 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmarmo commented Apr 19, 2020

Uh oh!

NicolasHug left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmarmo commented Apr 19, 2020

Uh oh!

jnothman commented May 20, 2020

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

cmarmo commented May 22, 2020

Uh oh!

jnothman commented May 24, 2020 via email

Uh oh!

cmarmo commented Jun 4, 2020

Uh oh!

cmarmo commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

jnothman commented Jun 21, 2020 via email

Uh oh!

Uh oh!

cmarmo commented Apr 15, 2020 •

edited

Loading

cmarmo commented Apr 18, 2020 •

edited

Loading

cmarmo commented Jun 11, 2020 •

edited

Loading