Fix flake8 in examples

[rth]

Now that black fixed PEP8 for most of the code base in #18948 it would be nice to stop running flak8 on the diff, and instead make,

flake8 .

work.

This requires fixing a few flake8 issues in examples/ and doc/. That would correspond to just 50 LoC diff and I think it would be worth it.

Full list of errors,

./doc/conf.py:30:1: E402 module level import not at top of file
./doc/conf.py:31:1: E402 module level import not at top of file
./doc/conf.py:92:1: E402 module level import not at top of file
./doc/conftest.py:31:5: F841 local variable 'data_home' is assigned to but never used
./examples/applications/plot_face_recognition.py:152:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/applications/plot_out_of_core_classification.py:56:1: E303 too many blank lines (3)
./examples/applications/svm_gui.py:336:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/applications/wikipedia_principal_eigenvector.py:219:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/calibration/plot_calibration_curve.py:83:5: F841 local variable 'fig' is assigned to but never used
./examples/calibration/plot_calibration_curve.py:129:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/cluster/plot_adjusted_for_chance_measures.py:33:1: E302 expected 2 blank lines, found 1
./examples/cluster/plot_adjusted_for_chance_measures.py:61:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/cluster/plot_agglomerative_clustering_metrics.py:54:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/cluster/plot_digits_linkage.py:38:1: E302 expected 2 blank lines, found 1
./examples/cluster/plot_digits_linkage.py:44:35: E128 continuation line under-indented for visual indent
./examples/cluster/plot_digits_linkage.py:45:35: E128 continuation line under-indented for visual indent
./examples/cluster/plot_digits_linkage.py:46:35: E124 closing bracket does not match visual indentation
./examples/cluster/plot_digits_linkage.py:55:1: E265 block comment should start with '# '
./examples/cluster/plot_digits_linkage.py:74:1: E265 block comment should start with '# '
./examples/cluster/plot_digits_linkage.py:76:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/cluster/plot_kmeans_stability_low_dim_dense.py:72:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/datasets/plot_digits_last_image.py:29:1: E265 block comment should start with '# '
./examples/datasets/plot_digits_last_image.py:32:1: E265 block comment should start with '# '
./examples/decomposition/plot_faces_decomposition.py:66:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/decomposition/plot_ica_blind_source_separation.py:62:34: W291 trailing whitespace
./examples/decomposition/plot_ica_vs_pca.py:83:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/decomposition/plot_image_denoising.py:123:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/decomposition/plot_pca_3d.py:40:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/ensemble/plot_adaboost_regression.py:36:27: E128 continuation line under-indented for visual indent
./examples/ensemble/plot_gradient_boosting_regularization.py:44:89: E501 line too long (99 > 88 characters)
./examples/ensemble/plot_gradient_boosting_regularization.py:73:13: E128 continuation line under-indented for visual indent
./examples/feature_selection/plot_rfe_digits.py:11:89: E501 line too long (94 > 88 characters)
./examples/gaussian_process/plot_gpc_isoprobability.py:37:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/gaussian_process/plot_gpr_noisy_targets.py:43:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/linear_model/plot_robust_fit.py:69:89: E501 line too long (101 > 88 characters)
./examples/manifold/plot_swissroll.py:21:1: E265 block comment should start with '# '
./examples/manifold/plot_swissroll.py:32:1: E265 block comment should start with '# '
./examples/mixture/plot_concentration_prior.py:88:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/mixture/plot_gmm_covariances.py:69:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/model_selection/grid_search_text_feature_extraction.py:74:1: E265 block comment should start with '# '
./examples/model_selection/plot_precision_recall.py:176:1: F811 redefinition of unused 'precision_recall_curve' from line 136
./examples/model_selection/plot_precision_recall.py:190:5: E128 continuation line under-indented for visual indent
./examples/model_selection/plot_underfitting_overfitting.py:36:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/neighbors/plot_kde_1d.py:106:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/neighbors/plot_species_kde.py:14:89: E501 line too long (89 > 88 characters)
./examples/preprocessing/plot_all_scaling.py:109:1: E302 expected 2 blank lines, found 1
./examples/svm/plot_custom_kernel.py:19:23: E114 indentation is not a multiple of 4 (comment)
./examples/svm/plot_custom_kernel.py:19:23: E116 unexpected indentation (comment)
./examples/svm/plot_rbf_parameters.py:108:1: E305 expected 2 blank lines after class or function definition, found 1
./examples/compose/plot_column_transformer_mixed_types.py:183:7: F541 f-string is missing placeholders
./sklearn/preprocessing/_polynomial.py:281:21: F402 import 'comb' from line 11 shadowed by loop variable

[NicolasHug]

it would be nice to stop running flak8 on the diff

I'm all for making flake8 . pass and use it as a CI check, but I think we should still support and recommend git diff upstream/main -u -- "*.py" | flake8 --diff for PR contributions, as it's likely to be much faster to run than flake8 .

[rth]

That's what pre-commit is for, it only runs checks on modified files and should be fairly fast. Also is cross plaform, unlike pipes in bash.

The issue is that flake8 passes on the diff does not strictly imply that user is not introducing new flake8 issues #11336 (comment)

[rth]

Also on a desktop with SSD,

$ time flake8 . > /dev/null

real    0m4,584s
user    1m8,725s
sys     0m0,769s

so it's not that long. But I agree that running flake8 only on modified files is a preferred workflow anyway.

[NicolasHug]

I have an SSD too (and a beefy laptop) but on mine it takes about 1min 😬:

(sklearn-dev) ➜  scikit-learn git:(main) ✗ time flake8 . > /dev/null

flake8 . > /dev/null  52.44s user 0.62s system 99% cpu 53.218 total

That's what pre-commit is for

Sounds good, as long as we don't tell users to use flake8 . on all PRs

[rth]

And actually running flake8 on the diff is slower for me,

time git diff upstream/main -u -- "*.py" | flake8 --diff

real    0m6,447s
user    0m6,422s
sys     0m0,050s

likely because of git operations.

[NicolasHug]

lol I got it reversed. Checking out #20298:

(sklearn-dev) ➜  scikit-learn git:(flake8_fix) ✗ time git diff origin/main -u -- "*.py" | flake8 --diff

git diff origin/main -u -- "*.py"  0.01s user 0.02s system 85% cpu 0.041 total
flake8 --diff  1.35s user 0.07s system 99% cpu 1.432 total

[rth]

So I just learned that flake8 is using multiprocessing by default, so it likely also depends on the number of available CPU.

[NicolasHug]

Just curious, what version do you have?

(sklearn-dev) ➜  scikit-learn git:(flake8_fix) ✗ flake8 --version
3.9.2 (mccabe: 0.6.1, pycodestyle: 2.7.0, pyflakes: 2.3.1) CPython 3.9.1 on Darwin

I have 16 CPUs and only one of them is being used by flake8, barely at 80%

[rth]

Exactly the same versions, on Linux. It looks like it's disabled on MacOS (PyCQA/flake8#1337 (comment)) and possibly on Windows PyCQA/flake8#1140. We should tell them to use Joblib if it's installed :)

[ogrisel]

Reopening this as #20298 was merged making flake8 . work on the main branch but the contributors documentation was not updated yet.

[ogrisel]

For reference, running flake8 . on the top level folder on my macbook air m1 (with ssd) takes 29s. It seems to be using a single core.

So asking people to run it on the whole code base is problematic in my opinion...