forked from scikit-learn/scikit-learn.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathensemble.html
1390 lines (1336 loc) · 119 KB
/
ensemble.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>1.11. Ensemble methods — scikit-learn 0.20.dev0 documentation</title>
<!-- htmltitle is before nature.css - we use this hack to load bootstrap first -->
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="stylesheet" href="../_static/css/bootstrap.min.css" media="screen" />
<link rel="stylesheet" href="../_static/css/bootstrap-responsive.css"/>
<link rel="stylesheet" href="../_static/nature.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '0.20.dev0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/js/copybutton.js"></script>
<script type="text/javascript" src="../_static/js/extra.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_SVG"></script>
<link rel="shortcut icon" href="../_static/favicon.ico"/>
<link rel="author" title="About these documents" href="../about.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="1.12. Multiclass and multilabel algorithms" href="multiclass.html" />
<link rel="prev" title="1.10. Decision Trees" href="tree.html" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<script src="../_static/js/bootstrap.min.js" type="text/javascript"></script>
<script>
VERSION_SUBDIR = (function(groups) {
return groups ? groups[1] : null;
})(location.href.match(/^https?:\/\/scikit-learn.org\/([^\/]+)/));
</script>
<link rel="canonical" href="http://scikit-learn.org/stable/modules/ensemble.html" />
<script type="text/javascript">
$("div.buttonNext, div.buttonPrevious").hover(
function () {
$(this).css('background-color', '#FF9C34');
},
function () {
$(this).css('background-color', '#A7D6E2');
}
);
function showMenu() {
var topNav = document.getElementById("scikit-navbar");
if (topNav.className === "navbar") {
topNav.className += " responsive";
} else {
topNav.className = "navbar";
}
};
</script>
</head>
<body>
<div class="header-wrapper">
<div class="header">
<p class="logo"><a href="../index.html">
<img src="../_static/scikit-learn-logo-small.png" alt="Logo"/>
</a>
</p><div class="navbar" id="scikit-navbar">
<ul>
<li><a href="../index.html">Home</a></li>
<li><a href="../install.html">Installation</a></li>
<li class="btn-li"><div class="btn-group">
<a href="../documentation.html">Documentation</a>
<a class="btn dropdown-toggle" data-toggle="dropdown">
<span class="caret"></span>
</a>
<ul class="dropdown-menu">
<li class="link-title">Scikit-learn <script>document.write(DOCUMENTATION_OPTIONS.VERSION + (VERSION_SUBDIR ? " (" + VERSION_SUBDIR + ")" : ""));</script></li>
<li><a href="../tutorial/index.html">Tutorials</a></li>
<li><a href="../user_guide.html">User guide</a></li>
<li><a href="classes.html">API</a></li>
<li><a href="../glossary.html">Glossary</a></li>
<li><a href="../faq.html">FAQ</a></li>
<li><a href="../developers/contributing.html">Contributing</a></li>
<li class="divider"></li>
<script>if (VERSION_SUBDIR != "stable") document.write('<li><a href="http://scikit-learn.org/stable/documentation.html">Stable version</a></li>')</script>
<script>if (VERSION_SUBDIR != "dev") document.write('<li><a href="http://scikit-learn.org/dev/documentation.html">Development version</a></li>')</script>
<li><a href="http://scikit-learn.org/dev/versions.html">All available versions</a></li>
<li><a href="../_downloads/scikit-learn-docs.pdf">PDF documentation</a></li>
</ul>
</div>
</li>
<li><a href="../auto_examples/index.html">Examples</a></li>
</ul>
<a href="javascript:void(0);" onclick="showMenu()">
<div class="nav-icon">
<div class="hamburger-line"></div>
<div class="hamburger-line"></div>
<div class="hamburger-line"></div>
</div>
</a>
<div class="search_form">
<div class="gcse-search" id="cse" style="width: 100%;"></div>
</div>
</div> <!-- end navbar --></div>
</div>
<!-- GitHub "fork me" ribbon -->
<a href="https://github.com/scikit-learn/scikit-learn">
<img class="fork-me"
style="position: absolute; top: 0; right: 0; border: 0;"
src="../_static/img/forkme.png"
alt="Fork me on GitHub" />
</a>
<div class="content-wrapper">
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<div class="rel">
<div class="rellink">
<a href="tree.html"
accesskey="P">Previous
<br/>
<span class="smallrellink">
1.10. Decision Trees
</span>
<span class="hiddenrellink">
1.10. Decision Trees
</span>
</a>
</div>
<div class="spacer">
</div>
<div class="rellink">
<a href="multiclass.html"
accesskey="N">Next
<br/>
<span class="smallrellink">
1.12. Multicl...
</span>
<span class="hiddenrellink">
1.12. Multiclass and multilabel algorithms
</span>
</a>
</div>
<!-- Ad a link to the 'up' page -->
<div class="spacer">
</div>
<div class="rellink">
<a href="../supervised_learning.html">
Up
<br/>
<span class="smallrellink">
1. Supervised...
</span>
<span class="hiddenrellink">
1. Supervised learning
</span>
</a>
</div>
</div>
<p class="doc-version"><b>scikit-learn v0.20.dev0</b><br/>
<a href="http://scikit-learn.org/stable/support.html#documentation-resources">Other versions</a></p>
<p class="citing">Please <b><a href="../about.html#citing-scikit-learn" style="font-size: 110%;">cite us </a></b>if you use the software.</p>
<ul>
<li><a class="reference internal" href="#">1.11. Ensemble methods</a><ul>
<li><a class="reference internal" href="#bagging-meta-estimator">1.11.1. Bagging meta-estimator</a></li>
<li><a class="reference internal" href="#forests-of-randomized-trees">1.11.2. Forests of randomized trees</a><ul>
<li><a class="reference internal" href="#random-forests">1.11.2.1. Random Forests</a></li>
<li><a class="reference internal" href="#extremely-randomized-trees">1.11.2.2. Extremely Randomized Trees</a></li>
<li><a class="reference internal" href="#parameters">1.11.2.3. Parameters</a></li>
<li><a class="reference internal" href="#parallelization">1.11.2.4. Parallelization</a></li>
<li><a class="reference internal" href="#feature-importance-evaluation">1.11.2.5. Feature importance evaluation</a></li>
<li><a class="reference internal" href="#totally-random-trees-embedding">1.11.2.6. Totally Random Trees Embedding</a></li>
</ul>
</li>
<li><a class="reference internal" href="#adaboost">1.11.3. AdaBoost</a><ul>
<li><a class="reference internal" href="#usage">1.11.3.1. Usage</a></li>
</ul>
</li>
<li><a class="reference internal" href="#gradient-tree-boosting">1.11.4. Gradient Tree Boosting</a><ul>
<li><a class="reference internal" href="#classification">1.11.4.1. Classification</a></li>
<li><a class="reference internal" href="#regression">1.11.4.2. Regression</a></li>
<li><a class="reference internal" href="#fitting-additional-weak-learners">1.11.4.3. Fitting additional weak-learners</a></li>
<li><a class="reference internal" href="#controlling-the-tree-size">1.11.4.4. Controlling the tree size</a></li>
<li><a class="reference internal" href="#mathematical-formulation">1.11.4.5. Mathematical formulation</a><ul>
<li><a class="reference internal" href="#loss-functions">1.11.4.5.1. Loss Functions</a></li>
</ul>
</li>
<li><a class="reference internal" href="#regularization">1.11.4.6. Regularization</a><ul>
<li><a class="reference internal" href="#shrinkage">1.11.4.6.1. Shrinkage</a></li>
<li><a class="reference internal" href="#subsampling">1.11.4.6.2. Subsampling</a></li>
</ul>
</li>
<li><a class="reference internal" href="#interpretation">1.11.4.7. Interpretation</a><ul>
<li><a class="reference internal" href="#feature-importance">1.11.4.7.1. Feature importance</a></li>
<li><a class="reference internal" href="#partial-dependence">1.11.4.7.2. Partial dependence</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#voting-classifier">1.11.5. Voting Classifier</a><ul>
<li><a class="reference internal" href="#majority-class-labels-majority-hard-voting">1.11.5.1. Majority Class Labels (Majority/Hard Voting)</a><ul>
<li><a class="reference internal" href="#id25">1.11.5.1.1. Usage</a></li>
</ul>
</li>
<li><a class="reference internal" href="#weighted-average-probabilities-soft-voting">1.11.5.2. Weighted Average Probabilities (Soft Voting)</a></li>
<li><a class="reference internal" href="#using-the-votingclassifier-with-gridsearch">1.11.5.3. Using the <cite>VotingClassifier</cite> with <cite>GridSearch</cite></a><ul>
<li><a class="reference internal" href="#id26">1.11.5.3.1. Usage</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
</div>
<input type="checkbox" id="nav-trigger" class="nav-trigger" checked />
<label for="nav-trigger"></label>
<div class="content">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="ensemble-methods">
<span id="ensemble"></span><h1>1.11. Ensemble methods<a class="headerlink" href="#ensemble-methods" title="Permalink to this headline">¶</a></h1>
<p>The goal of <strong>ensemble methods</strong> is to combine the predictions of several
base estimators built with a given learning algorithm in order to improve
generalizability / robustness over a single estimator.</p>
<p>Two families of ensemble methods are usually distinguished:</p>
<ul>
<li><p class="first">In <strong>averaging methods</strong>, the driving principle is to build several
estimators independently and then to average their predictions. On average,
the combined estimator is usually better than any of the single base
estimator because its variance is reduced.</p>
<p><strong>Examples:</strong> <a class="reference internal" href="#bagging"><span class="std std-ref">Bagging methods</span></a>, <a class="reference internal" href="#forest"><span class="std std-ref">Forests of randomized trees</span></a>, …</p>
</li>
<li><p class="first">By contrast, in <strong>boosting methods</strong>, base estimators are built sequentially
and one tries to reduce the bias of the combined estimator. The motivation is
to combine several weak models to produce a powerful ensemble.</p>
<p><strong>Examples:</strong> <a class="reference internal" href="#adaboost"><span class="std std-ref">AdaBoost</span></a>, <a class="reference internal" href="#gradient-boosting"><span class="std std-ref">Gradient Tree Boosting</span></a>, …</p>
</li>
</ul>
<div class="section" id="bagging-meta-estimator">
<span id="bagging"></span><h2>1.11.1. Bagging meta-estimator<a class="headerlink" href="#bagging-meta-estimator" title="Permalink to this headline">¶</a></h2>
<p>In ensemble algorithms, bagging methods form a class of algorithms which build
several instances of a black-box estimator on random subsets of the original
training set and then aggregate their individual predictions to form a final
prediction. These methods are used as a way to reduce the variance of a base
estimator (e.g., a decision tree), by introducing randomization into its
construction procedure and then making an ensemble out of it. In many cases,
bagging methods constitute a very simple way to improve with respect to a
single model, without making it necessary to adapt the underlying base
algorithm. As they provide a way to reduce overfitting, bagging methods work
best with strong and complex models (e.g., fully developed decision trees), in
contrast with boosting methods which usually work best with weak models (e.g.,
shallow decision trees).</p>
<p>Bagging methods come in many flavours but mostly differ from each other by the
way they draw random subsets of the training set:</p>
<blockquote>
<div><ul class="simple">
<li>When random subsets of the dataset are drawn as random subsets of the
samples, then this algorithm is known as Pasting <a class="reference internal" href="#b1999" id="id1">[B1999]</a>.</li>
<li>When samples are drawn with replacement, then the method is known as
Bagging <a class="reference internal" href="#b1996" id="id2">[B1996]</a>.</li>
<li>When random subsets of the dataset are drawn as random subsets of
the features, then the method is known as Random Subspaces <a class="reference internal" href="#h1998" id="id3">[H1998]</a>.</li>
<li>Finally, when base estimators are built on subsets of both samples and
features, then the method is known as Random Patches <a class="reference internal" href="#lg2012" id="id4">[LG2012]</a>.</li>
</ul>
</div></blockquote>
<p>In scikit-learn, bagging methods are offered as a unified
<a class="reference internal" href="generated/sklearn.ensemble.BaggingClassifier.html#sklearn.ensemble.BaggingClassifier" title="sklearn.ensemble.BaggingClassifier"><code class="xref py py-class docutils literal"><span class="pre">BaggingClassifier</span></code></a> meta-estimator (resp. <a class="reference internal" href="generated/sklearn.ensemble.BaggingRegressor.html#sklearn.ensemble.BaggingRegressor" title="sklearn.ensemble.BaggingRegressor"><code class="xref py py-class docutils literal"><span class="pre">BaggingRegressor</span></code></a>),
taking as input a user-specified base estimator along with parameters
specifying the strategy to draw random subsets. In particular, <code class="docutils literal"><span class="pre">max_samples</span></code>
and <code class="docutils literal"><span class="pre">max_features</span></code> control the size of the subsets (in terms of samples and
features), while <code class="docutils literal"><span class="pre">bootstrap</span></code> and <code class="docutils literal"><span class="pre">bootstrap_features</span></code> control whether
samples and features are drawn with or without replacement. When using a subset
of the available samples the generalization accuracy can be estimated with the
out-of-bag samples by setting <code class="docutils literal"><span class="pre">oob_score=True</span></code>. As an example, the
snippet below illustrates how to instantiate a bagging ensemble of
<code class="xref py py-class docutils literal"><span class="pre">KNeighborsClassifier</span></code> base estimators, each built on random subsets of
50% of the samples and 50% of the features.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="k">import</span> <span class="n">BaggingClassifier</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.neighbors</span> <span class="k">import</span> <span class="n">KNeighborsClassifier</span>
<span class="gp">>>> </span><span class="n">bagging</span> <span class="o">=</span> <span class="n">BaggingClassifier</span><span class="p">(</span><span class="n">KNeighborsClassifier</span><span class="p">(),</span>
<span class="gp">... </span> <span class="n">max_samples</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">max_features</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
</pre></div>
</div>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/ensemble/plot_bias_variance.html#sphx-glr-auto-examples-ensemble-plot-bias-variance-py"><span class="std std-ref">Single estimator versus bagging: bias-variance decomposition</span></a></li>
</ul>
</div>
<div class="topic">
<p class="topic-title first">References</p>
<table class="docutils citation" frame="void" id="b1999" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[B1999]</a></td><td>L. Breiman, “Pasting small votes for classification in large
databases and on-line”, Machine Learning, 36(1), 85-103, 1999.</td></tr>
</tbody>
</table>
<table class="docutils citation" frame="void" id="b1996" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[B1996]</a></td><td>L. Breiman, “Bagging predictors”, Machine Learning, 24(2),
123-140, 1996.</td></tr>
</tbody>
</table>
<table class="docutils citation" frame="void" id="h1998" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[H1998]</a></td><td>T. Ho, “The random subspace method for constructing decision
forests”, Pattern Analysis and Machine Intelligence, 20(8), 832-844,
1998.</td></tr>
</tbody>
</table>
<table class="docutils citation" frame="void" id="lg2012" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[LG2012]</a></td><td>G. Louppe and P. Geurts, “Ensembles on Random Patches”,
Machine Learning and Knowledge Discovery in Databases, 346-361, 2012.</td></tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="forests-of-randomized-trees">
<span id="forest"></span><h2>1.11.2. Forests of randomized trees<a class="headerlink" href="#forests-of-randomized-trees" title="Permalink to this headline">¶</a></h2>
<p>The <a class="reference internal" href="classes.html#module-sklearn.ensemble" title="sklearn.ensemble"><code class="xref py py-mod docutils literal"><span class="pre">sklearn.ensemble</span></code></a> module includes two averaging algorithms based
on randomized <a class="reference internal" href="tree.html#tree"><span class="std std-ref">decision trees</span></a>: the RandomForest algorithm
and the Extra-Trees method. Both algorithms are perturb-and-combine
techniques <a class="reference internal" href="#b1998" id="id5">[B1998]</a> specifically designed for trees. This means a diverse
set of classifiers is created by introducing randomness in the classifier
construction. The prediction of the ensemble is given as the averaged
prediction of the individual classifiers.</p>
<p>As other classifiers, forest classifiers have to be fitted with two
arrays: a sparse or dense array X of size <code class="docutils literal"><span class="pre">[n_samples,</span> <span class="pre">n_features]</span></code> holding the
training samples, and an array Y of size <code class="docutils literal"><span class="pre">[n_samples]</span></code> holding the
target values (class labels) for the training samples:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="k">import</span> <span class="n">RandomForestClassifier</span>
<span class="gp">>>> </span><span class="n">X</span> <span class="o">=</span> <span class="p">[[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]]</span>
<span class="gp">>>> </span><span class="n">Y</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">)</span>
</pre></div>
</div>
<p>Like <a class="reference internal" href="tree.html#tree"><span class="std std-ref">decision trees</span></a>, forests of trees also extend
to <a class="reference internal" href="tree.html#tree-multioutput"><span class="std std-ref">multi-output problems</span></a> (if Y is an array of size
<code class="docutils literal"><span class="pre">[n_samples,</span> <span class="pre">n_outputs]</span></code>).</p>
<div class="section" id="random-forests">
<h3>1.11.2.1. Random Forests<a class="headerlink" href="#random-forests" title="Permalink to this headline">¶</a></h3>
<p>In random forests (see <a class="reference internal" href="generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier" title="sklearn.ensemble.RandomForestClassifier"><code class="xref py py-class docutils literal"><span class="pre">RandomForestClassifier</span></code></a> and
<a class="reference internal" href="generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor" title="sklearn.ensemble.RandomForestRegressor"><code class="xref py py-class docutils literal"><span class="pre">RandomForestRegressor</span></code></a> classes), each tree in the ensemble is
built from a sample drawn with replacement (i.e., a bootstrap sample)
from the training set. In addition, when splitting a node during the
construction of the tree, the split that is chosen is no longer the
best split among all features. Instead, the split that is picked is the
best split among a random subset of the features. As a result of this
randomness, the bias of the forest usually slightly increases (with
respect to the bias of a single non-random tree) but, due to averaging,
its variance also decreases, usually more than compensating for the
increase in bias, hence yielding an overall better model.</p>
<p>In contrast to the original publication <a class="reference internal" href="#b2001" id="id6">[B2001]</a>, the scikit-learn
implementation combines classifiers by averaging their probabilistic
prediction, instead of letting each classifier vote for a single class.</p>
</div>
<div class="section" id="extremely-randomized-trees">
<h3>1.11.2.2. Extremely Randomized Trees<a class="headerlink" href="#extremely-randomized-trees" title="Permalink to this headline">¶</a></h3>
<p>In extremely randomized trees (see <a class="reference internal" href="generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier" title="sklearn.ensemble.ExtraTreesClassifier"><code class="xref py py-class docutils literal"><span class="pre">ExtraTreesClassifier</span></code></a>
and <a class="reference internal" href="generated/sklearn.ensemble.ExtraTreesRegressor.html#sklearn.ensemble.ExtraTreesRegressor" title="sklearn.ensemble.ExtraTreesRegressor"><code class="xref py py-class docutils literal"><span class="pre">ExtraTreesRegressor</span></code></a> classes), randomness goes one step
further in the way splits are computed. As in random forests, a random
subset of candidate features is used, but instead of looking for the
most discriminative thresholds, thresholds are drawn at random for each
candidate feature and the best of these randomly-generated thresholds is
picked as the splitting rule. This usually allows to reduce the variance
of the model a bit more, at the expense of a slightly greater increase
in bias:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="k">import</span> <span class="n">cross_val_score</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="k">import</span> <span class="n">make_blobs</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="k">import</span> <span class="n">RandomForestClassifier</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="k">import</span> <span class="n">ExtraTreesClassifier</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.tree</span> <span class="k">import</span> <span class="n">DecisionTreeClassifier</span>
<span class="gp">>>> </span><span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">make_blobs</span><span class="p">(</span><span class="n">n_samples</span><span class="o">=</span><span class="mi">10000</span><span class="p">,</span> <span class="n">n_features</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">centers</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">max_depth</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">min_samples_split</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span> <span class="o">=</span> <span class="n">cross_val_score</span><span class="p">(</span><span class="n">clf</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="go">0.97...</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">max_depth</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">min_samples_split</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span> <span class="o">=</span> <span class="n">cross_val_score</span><span class="p">(</span><span class="n">clf</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="go">0.999...</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">ExtraTreesClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">max_depth</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">min_samples_split</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span> <span class="o">=</span> <span class="n">cross_val_score</span><span class="p">(</span><span class="n">clf</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span> <span class="o">></span> <span class="mf">0.999</span>
<span class="go">True</span>
</pre></div>
</div>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/ensemble/plot_forest_iris.html"><img alt="../_images/sphx_glr_plot_forest_iris_0011.png" src="../_images/sphx_glr_plot_forest_iris_0011.png" style="width: 480.0px; height: 360.0px;" /></a>
</div>
</div>
<div class="section" id="parameters">
<h3>1.11.2.3. Parameters<a class="headerlink" href="#parameters" title="Permalink to this headline">¶</a></h3>
<p>The main parameters to adjust when using these methods is <code class="docutils literal"><span class="pre">n_estimators</span></code>
and <code class="docutils literal"><span class="pre">max_features</span></code>. The former is the number of trees in the forest. The
larger the better, but also the longer it will take to compute. In
addition, note that results will stop getting significantly better
beyond a critical number of trees. The latter is the size of the random
subsets of features to consider when splitting a node. The lower the
greater the reduction of variance, but also the greater the increase in
bias. Empirical good default values are <code class="docutils literal"><span class="pre">max_features=n_features</span></code>
for regression problems, and <code class="docutils literal"><span class="pre">max_features=sqrt(n_features)</span></code> for
classification tasks (where <code class="docutils literal"><span class="pre">n_features</span></code> is the number of features
in the data). Good results are often achieved when setting <code class="docutils literal"><span class="pre">max_depth=None</span></code>
in combination with <code class="docutils literal"><span class="pre">min_samples_split=2</span></code> (i.e., when fully developing the
trees). Bear in mind though that these values are usually not optimal, and
might result in models that consume a lot of RAM. The best parameter values
should always be cross-validated. In addition, note that in random forests,
bootstrap samples are used by default (<code class="docutils literal"><span class="pre">bootstrap=True</span></code>)
while the default strategy for extra-trees is to use the whole dataset
(<code class="docutils literal"><span class="pre">bootstrap=False</span></code>).
When using bootstrap sampling the generalization accuracy can be estimated
on the left out or out-of-bag samples. This can be enabled by
setting <code class="docutils literal"><span class="pre">oob_score=True</span></code>.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">The size of the model with the default parameters is <span class="math">\(O( M * N * log (N) )\)</span>,
where <span class="math">\(M\)</span> is the number of trees and <span class="math">\(N\)</span> is the number of samples.
In order to reduce the size of the model, you can change these parameters:
<code class="docutils literal"><span class="pre">min_samples_split</span></code>, <code class="docutils literal"><span class="pre">min_samples_leaf</span></code>, <code class="docutils literal"><span class="pre">max_leaf_nodes</span></code> and <code class="docutils literal"><span class="pre">max_depth</span></code>.</p>
</div>
</div>
<div class="section" id="parallelization">
<h3>1.11.2.4. Parallelization<a class="headerlink" href="#parallelization" title="Permalink to this headline">¶</a></h3>
<p>Finally, this module also features the parallel construction of the trees
and the parallel computation of the predictions through the <code class="docutils literal"><span class="pre">n_jobs</span></code>
parameter. If <code class="docutils literal"><span class="pre">n_jobs=k</span></code> then computations are partitioned into
<code class="docutils literal"><span class="pre">k</span></code> jobs, and run on <code class="docutils literal"><span class="pre">k</span></code> cores of the machine. If <code class="docutils literal"><span class="pre">n_jobs=-1</span></code>
then all cores available on the machine are used. Note that because of
inter-process communication overhead, the speedup might not be linear
(i.e., using <code class="docutils literal"><span class="pre">k</span></code> jobs will unfortunately not be <code class="docutils literal"><span class="pre">k</span></code> times as
fast). Significant speedup can still be achieved though when building
a large number of trees, or when building a single tree requires a fair
amount of time (e.g., on large datasets).</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/ensemble/plot_forest_iris.html#sphx-glr-auto-examples-ensemble-plot-forest-iris-py"><span class="std std-ref">Plot the decision surfaces of ensembles of trees on the iris dataset</span></a></li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_forest_importances_faces.html#sphx-glr-auto-examples-ensemble-plot-forest-importances-faces-py"><span class="std std-ref">Pixel importances with a parallel forest of trees</span></a></li>
<li><a class="reference internal" href="../auto_examples/plot_multioutput_face_completion.html#sphx-glr-auto-examples-plot-multioutput-face-completion-py"><span class="std std-ref">Face completion with a multi-output estimators</span></a></li>
</ul>
</div>
<div class="topic">
<p class="topic-title first">References</p>
<table class="docutils citation" frame="void" id="b2001" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id6">[B2001]</a></td><td><ol class="first last upperalpha simple" start="12">
<li>Breiman, “Random Forests”, Machine Learning, 45(1), 5-32, 2001.</li>
</ol>
</td></tr>
</tbody>
</table>
<table class="docutils citation" frame="void" id="b1998" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[B1998]</a></td><td><ol class="first last upperalpha simple" start="12">
<li>Breiman, “Arcing Classifiers”, Annals of Statistics 1998.</li>
</ol>
</td></tr>
</tbody>
</table>
<ul class="simple">
<li>P. Geurts, D. Ernst., and L. Wehenkel, “Extremely randomized
trees”, Machine Learning, 63(1), 3-42, 2006.</li>
</ul>
</div>
</div>
<div class="section" id="feature-importance-evaluation">
<span id="random-forest-feature-importance"></span><h3>1.11.2.5. Feature importance evaluation<a class="headerlink" href="#feature-importance-evaluation" title="Permalink to this headline">¶</a></h3>
<p>The relative rank (i.e. depth) of a feature used as a decision node in a
tree can be used to assess the relative importance of that feature with
respect to the predictability of the target variable. Features used at
the top of the tree contribute to the final prediction decision of a
larger fraction of the input samples. The <strong>expected fraction of the
samples</strong> they contribute to can thus be used as an estimate of the
<strong>relative importance of the features</strong>.</p>
<p>By <strong>averaging</strong> those expected activity rates over several randomized
trees one can <strong>reduce the variance</strong> of such an estimate and use it
for feature selection.</p>
<p>The following example shows a color-coded representation of the relative
importances of each individual pixel for a face recognition task using
a <a class="reference internal" href="generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier" title="sklearn.ensemble.ExtraTreesClassifier"><code class="xref py py-class docutils literal"><span class="pre">ExtraTreesClassifier</span></code></a> model.</p>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/ensemble/plot_forest_importances_faces.html"><img alt="../_images/sphx_glr_plot_forest_importances_faces_0011.png" src="../_images/sphx_glr_plot_forest_importances_faces_0011.png" style="width: 360.0px; height: 360.0px;" /></a>
</div>
<p>In practice those estimates are stored as an attribute named
<code class="docutils literal"><span class="pre">feature_importances_</span></code> on the fitted model. This is an array with shape
<code class="docutils literal"><span class="pre">(n_features,)</span></code> whose values are positive and sum to 1.0. The higher
the value, the more important is the contribution of the matching feature
to the prediction function.</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/ensemble/plot_forest_importances_faces.html#sphx-glr-auto-examples-ensemble-plot-forest-importances-faces-py"><span class="std std-ref">Pixel importances with a parallel forest of trees</span></a></li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_forest_importances.html#sphx-glr-auto-examples-ensemble-plot-forest-importances-py"><span class="std std-ref">Feature importances with forests of trees</span></a></li>
</ul>
</div>
</div>
<div class="section" id="totally-random-trees-embedding">
<span id="random-trees-embedding"></span><h3>1.11.2.6. Totally Random Trees Embedding<a class="headerlink" href="#totally-random-trees-embedding" title="Permalink to this headline">¶</a></h3>
<p><a class="reference internal" href="generated/sklearn.ensemble.RandomTreesEmbedding.html#sklearn.ensemble.RandomTreesEmbedding" title="sklearn.ensemble.RandomTreesEmbedding"><code class="xref py py-class docutils literal"><span class="pre">RandomTreesEmbedding</span></code></a> implements an unsupervised transformation of the
data. Using a forest of completely random trees, <a class="reference internal" href="generated/sklearn.ensemble.RandomTreesEmbedding.html#sklearn.ensemble.RandomTreesEmbedding" title="sklearn.ensemble.RandomTreesEmbedding"><code class="xref py py-class docutils literal"><span class="pre">RandomTreesEmbedding</span></code></a>
encodes the data by the indices of the leaves a data point ends up in. This
index is then encoded in a one-of-K manner, leading to a high dimensional,
sparse binary coding.
This coding can be computed very efficiently and can then be used as a basis
for other learning tasks.
The size and sparsity of the code can be influenced by choosing the number of
trees and the maximum depth per tree. For each tree in the ensemble, the coding
contains one entry of one. The size of the coding is at most <code class="docutils literal"><span class="pre">n_estimators</span> <span class="pre">*</span> <span class="pre">2</span>
<span class="pre">**</span> <span class="pre">max_depth</span></code>, the maximum number of leaves in the forest.</p>
<p>As neighboring data points are more likely to lie within the same leaf of a tree,
the transformation performs an implicit, non-parametric density estimation.</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/ensemble/plot_random_forest_embedding.html#sphx-glr-auto-examples-ensemble-plot-random-forest-embedding-py"><span class="std std-ref">Hashing feature transformation using Totally Random Trees</span></a></li>
<li><a class="reference internal" href="../auto_examples/manifold/plot_lle_digits.html#sphx-glr-auto-examples-manifold-plot-lle-digits-py"><span class="std std-ref">Manifold learning on handwritten digits: Locally Linear Embedding, Isomap…</span></a> compares non-linear
dimensionality reduction techniques on handwritten digits.</li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_feature_transformation.html#sphx-glr-auto-examples-ensemble-plot-feature-transformation-py"><span class="std std-ref">Feature transformations with ensembles of trees</span></a> compares
supervised and unsupervised tree based feature transformations.</li>
</ul>
</div>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference internal" href="manifold.html#manifold"><span class="std std-ref">Manifold learning</span></a> techniques can also be useful to derive non-linear
representations of feature space, also these approaches focus also on
dimensionality reduction.</p>
</div>
</div>
</div>
<div class="section" id="adaboost">
<span id="id7"></span><h2>1.11.3. AdaBoost<a class="headerlink" href="#adaboost" title="Permalink to this headline">¶</a></h2>
<p>The module <a class="reference internal" href="classes.html#module-sklearn.ensemble" title="sklearn.ensemble"><code class="xref py py-mod docutils literal"><span class="pre">sklearn.ensemble</span></code></a> includes the popular boosting algorithm
AdaBoost, introduced in 1995 by Freund and Schapire <a class="reference internal" href="#fs1995" id="id8">[FS1995]</a>.</p>
<p>The core principle of AdaBoost is to fit a sequence of weak learners (i.e.,
models that are only slightly better than random guessing, such as small
decision trees) on repeatedly modified versions of the data. The predictions
from all of them are then combined through a weighted majority vote (or sum) to
produce the final prediction. The data modifications at each so-called boosting
iteration consist of applying weights <span class="math">\(w_1\)</span>, <span class="math">\(w_2\)</span>, …, <span class="math">\(w_N\)</span>
to each of the training samples. Initially, those weights are all set to
<span class="math">\(w_i = 1/N\)</span>, so that the first step simply trains a weak learner on the
original data. For each successive iteration, the sample weights are
individually modified and the learning algorithm is reapplied to the reweighted
data. At a given step, those training examples that were incorrectly predicted
by the boosted model induced at the previous step have their weights increased,
whereas the weights are decreased for those that were predicted correctly. As
iterations proceed, examples that are difficult to predict receive
ever-increasing influence. Each subsequent weak learner is thereby forced to
concentrate on the examples that are missed by the previous ones in the sequence
<a class="reference internal" href="#htf" id="id9">[HTF]</a>.</p>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/ensemble/plot_adaboost_hastie_10_2.html"><img alt="../_images/sphx_glr_plot_adaboost_hastie_10_2_0011.png" src="../_images/sphx_glr_plot_adaboost_hastie_10_2_0011.png" style="width: 480.0px; height: 360.0px;" /></a>
</div>
<p>AdaBoost can be used both for classification and regression problems:</p>
<blockquote>
<div><ul class="simple">
<li>For multi-class classification, <a class="reference internal" href="generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier" title="sklearn.ensemble.AdaBoostClassifier"><code class="xref py py-class docutils literal"><span class="pre">AdaBoostClassifier</span></code></a> implements
AdaBoost-SAMME and AdaBoost-SAMME.R <a class="reference internal" href="#zzrh2009" id="id10">[ZZRH2009]</a>.</li>
<li>For regression, <a class="reference internal" href="generated/sklearn.ensemble.AdaBoostRegressor.html#sklearn.ensemble.AdaBoostRegressor" title="sklearn.ensemble.AdaBoostRegressor"><code class="xref py py-class docutils literal"><span class="pre">AdaBoostRegressor</span></code></a> implements AdaBoost.R2 <a class="reference internal" href="#d1997" id="id11">[D1997]</a>.</li>
</ul>
</div></blockquote>
<div class="section" id="usage">
<h3>1.11.3.1. Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h3>
<p>The following example shows how to fit an AdaBoost classifier with 100 weak
learners:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="k">import</span> <span class="n">cross_val_score</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="k">import</span> <span class="n">load_iris</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="k">import</span> <span class="n">AdaBoostClassifier</span>
<span class="gp">>>> </span><span class="n">iris</span> <span class="o">=</span> <span class="n">load_iris</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">AdaBoostClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span> <span class="o">=</span> <span class="n">cross_val_score</span><span class="p">(</span><span class="n">clf</span><span class="p">,</span> <span class="n">iris</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">iris</span><span class="o">.</span><span class="n">target</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">scores</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="go">0.9...</span>
</pre></div>
</div>
<p>The number of weak learners is controlled by the parameter <code class="docutils literal"><span class="pre">n_estimators</span></code>. The
<code class="docutils literal"><span class="pre">learning_rate</span></code> parameter controls the contribution of the weak learners in
the final combination. By default, weak learners are decision stumps. Different
weak learners can be specified through the <code class="docutils literal"><span class="pre">base_estimator</span></code> parameter.
The main parameters to tune to obtain good results are <code class="docutils literal"><span class="pre">n_estimators</span></code> and
the complexity of the base estimators (e.g., its depth <code class="docutils literal"><span class="pre">max_depth</span></code> or
minimum required number of samples at a leaf <code class="docutils literal"><span class="pre">min_samples_leaf</span></code> in case of
decision trees).</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/ensemble/plot_adaboost_hastie_10_2.html#sphx-glr-auto-examples-ensemble-plot-adaboost-hastie-10-2-py"><span class="std std-ref">Discrete versus Real AdaBoost</span></a> compares the
classification error of a decision stump, decision tree, and a boosted
decision stump using AdaBoost-SAMME and AdaBoost-SAMME.R.</li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_adaboost_multiclass.html#sphx-glr-auto-examples-ensemble-plot-adaboost-multiclass-py"><span class="std std-ref">Multi-class AdaBoosted Decision Trees</span></a> shows the performance
of AdaBoost-SAMME and AdaBoost-SAMME.R on a multi-class problem.</li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_adaboost_twoclass.html#sphx-glr-auto-examples-ensemble-plot-adaboost-twoclass-py"><span class="std std-ref">Two-class AdaBoost</span></a> shows the decision boundary
and decision function values for a non-linearly separable two-class problem
using AdaBoost-SAMME.</li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_adaboost_regression.html#sphx-glr-auto-examples-ensemble-plot-adaboost-regression-py"><span class="std std-ref">Decision Tree Regression with AdaBoost</span></a> demonstrates regression
with the AdaBoost.R2 algorithm.</li>
</ul>
</div>
<div class="topic">
<p class="topic-title first">References</p>
<table class="docutils citation" frame="void" id="fs1995" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id8">[FS1995]</a></td><td>Y. Freund, and R. Schapire, “A Decision-Theoretic Generalization of
On-Line Learning and an Application to Boosting”, 1997.</td></tr>
</tbody>
</table>
<table class="docutils citation" frame="void" id="zzrh2009" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id10">[ZZRH2009]</a></td><td>J. Zhu, H. Zou, S. Rosset, T. Hastie. “Multi-class AdaBoost”,
2009.</td></tr>
</tbody>
</table>
<table class="docutils citation" frame="void" id="d1997" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id11">[D1997]</a></td><td><ol class="first last upperalpha simple" start="8">
<li>Drucker. “Improving Regressors using Boosting Techniques”, 1997.</li>
</ol>
</td></tr>
</tbody>
</table>
<table class="docutils citation" frame="void" id="htf" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id9">[HTF]</a></td><td>T. Hastie, R. Tibshirani and J. Friedman, “Elements of
Statistical Learning Ed. 2”, Springer, 2009.</td></tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="section" id="gradient-tree-boosting">
<span id="gradient-boosting"></span><h2>1.11.4. Gradient Tree Boosting<a class="headerlink" href="#gradient-tree-boosting" title="Permalink to this headline">¶</a></h2>
<p><a class="reference external" href="https://en.wikipedia.org/wiki/Gradient_boosting">Gradient Tree Boosting</a>
or Gradient Boosted Regression Trees (GBRT) is a generalization
of boosting to arbitrary
differentiable loss functions. GBRT is an accurate and effective
off-the-shelf procedure that can be used for both regression and
classification problems. Gradient Tree Boosting models are used in a
variety of areas including Web search ranking and ecology.</p>
<p>The advantages of GBRT are:</p>
<blockquote>
<div><ul class="simple">
<li>Natural handling of data of mixed type (= heterogeneous features)</li>
<li>Predictive power</li>
<li>Robustness to outliers in output space (via robust loss functions)</li>
</ul>
</div></blockquote>
<p>The disadvantages of GBRT are:</p>
<blockquote>
<div><ul class="simple">
<li>Scalability, due to the sequential nature of boosting it can
hardly be parallelized.</li>
</ul>
</div></blockquote>
<p>The module <a class="reference internal" href="classes.html#module-sklearn.ensemble" title="sklearn.ensemble"><code class="xref py py-mod docutils literal"><span class="pre">sklearn.ensemble</span></code></a> provides methods
for both classification and regression via gradient boosted regression
trees.</p>
<div class="section" id="classification">
<h3>1.11.4.1. Classification<a class="headerlink" href="#classification" title="Permalink to this headline">¶</a></h3>
<p><a class="reference internal" href="generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier" title="sklearn.ensemble.GradientBoostingClassifier"><code class="xref py py-class docutils literal"><span class="pre">GradientBoostingClassifier</span></code></a> supports both binary and multi-class
classification.
The following example shows how to fit a gradient boosting classifier
with 100 decision stumps as weak learners:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="k">import</span> <span class="n">make_hastie_10_2</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="k">import</span> <span class="n">GradientBoostingClassifier</span>
<span class="gp">>>> </span><span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">make_hastie_10_2</span><span class="p">(</span><span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span> <span class="o">=</span> <span class="n">X</span><span class="p">[:</span><span class="mi">2000</span><span class="p">],</span> <span class="n">X</span><span class="p">[</span><span class="mi">2000</span><span class="p">:]</span>
<span class="gp">>>> </span><span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">y</span><span class="p">[:</span><span class="mi">2000</span><span class="p">],</span> <span class="n">y</span><span class="p">[</span><span class="mi">2000</span><span class="p">:]</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">GradientBoostingClassifier</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">max_depth</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span> <span class="n">y_test</span><span class="p">)</span>
<span class="go">0.913...</span>
</pre></div>
</div>
<p>The number of weak learners (i.e. regression trees) is controlled by the parameter <code class="docutils literal"><span class="pre">n_estimators</span></code>; <a class="reference internal" href="#gradient-boosting-tree-size"><span class="std std-ref">The size of each tree</span></a> can be controlled either by setting the tree depth via <code class="docutils literal"><span class="pre">max_depth</span></code> or by setting the number of leaf nodes via <code class="docutils literal"><span class="pre">max_leaf_nodes</span></code>. The <code class="docutils literal"><span class="pre">learning_rate</span></code> is a hyper-parameter in the range (0.0, 1.0] that controls overfitting via <a class="reference internal" href="#gradient-boosting-shrinkage"><span class="std std-ref">shrinkage</span></a> .</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Classification with more than 2 classes requires the induction
of <code class="docutils literal"><span class="pre">n_classes</span></code> regression trees at each iteration,
thus, the total number of induced trees equals
<code class="docutils literal"><span class="pre">n_classes</span> <span class="pre">*</span> <span class="pre">n_estimators</span></code>. For datasets with a large number
of classes we strongly recommend to use
<a class="reference internal" href="generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier" title="sklearn.ensemble.RandomForestClassifier"><code class="xref py py-class docutils literal"><span class="pre">RandomForestClassifier</span></code></a> as an alternative to <a class="reference internal" href="generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier" title="sklearn.ensemble.GradientBoostingClassifier"><code class="xref py py-class docutils literal"><span class="pre">GradientBoostingClassifier</span></code></a> .</p>
</div>
</div>
<div class="section" id="regression">
<h3>1.11.4.2. Regression<a class="headerlink" href="#regression" title="Permalink to this headline">¶</a></h3>
<p><a class="reference internal" href="generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor" title="sklearn.ensemble.GradientBoostingRegressor"><code class="xref py py-class docutils literal"><span class="pre">GradientBoostingRegressor</span></code></a> supports a number of
<a class="reference internal" href="#gradient-boosting-loss"><span class="std std-ref">different loss functions</span></a>
for regression which can be specified via the argument
<code class="docutils literal"><span class="pre">loss</span></code>; the default loss function for regression is least squares (<code class="docutils literal"><span class="pre">'ls'</span></code>).</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="k">import</span> <span class="n">mean_squared_error</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="k">import</span> <span class="n">make_friedman1</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="k">import</span> <span class="n">GradientBoostingRegressor</span>
<span class="gp">>>> </span><span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">make_friedman1</span><span class="p">(</span><span class="n">n_samples</span><span class="o">=</span><span class="mi">1200</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">noise</span><span class="o">=</span><span class="mf">1.0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span> <span class="o">=</span> <span class="n">X</span><span class="p">[:</span><span class="mi">200</span><span class="p">],</span> <span class="n">X</span><span class="p">[</span><span class="mi">200</span><span class="p">:]</span>
<span class="gp">>>> </span><span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">y</span><span class="p">[:</span><span class="mi">200</span><span class="p">],</span> <span class="n">y</span><span class="p">[</span><span class="mi">200</span><span class="p">:]</span>
<span class="gp">>>> </span><span class="n">est</span> <span class="o">=</span> <span class="n">GradientBoostingRegressor</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">max_depth</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">loss</span><span class="o">=</span><span class="s1">'ls'</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">est</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">))</span>
<span class="go">5.00...</span>
</pre></div>
</div>
<p>The figure below shows the results of applying <a class="reference internal" href="generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor" title="sklearn.ensemble.GradientBoostingRegressor"><code class="xref py py-class docutils literal"><span class="pre">GradientBoostingRegressor</span></code></a>
with least squares loss and 500 base learners to the Boston house price dataset
(<a class="reference internal" href="generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston" title="sklearn.datasets.load_boston"><code class="xref py py-func docutils literal"><span class="pre">sklearn.datasets.load_boston</span></code></a>).
The plot on the left shows the train and test error at each iteration.
The train error at each iteration is stored in the
<code class="xref py py-attr docutils literal"><span class="pre">train_score_</span></code> attribute
of the gradient boosting model. The test error at each iterations can be obtained
via the <a class="reference internal" href="generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor.staged_predict" title="sklearn.ensemble.GradientBoostingRegressor.staged_predict"><code class="xref py py-meth docutils literal"><span class="pre">staged_predict</span></code></a> method which returns a
generator that yields the predictions at each stage. Plots like these can be used
to determine the optimal number of trees (i.e. <code class="docutils literal"><span class="pre">n_estimators</span></code>) by early stopping.
The plot on the right shows the feature importances which can be obtained via
the <code class="docutils literal"><span class="pre">feature_importances_</span></code> property.</p>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/ensemble/plot_gradient_boosting_regression.html"><img alt="../_images/sphx_glr_plot_gradient_boosting_regression_0011.png" src="../_images/sphx_glr_plot_gradient_boosting_regression_0011.png" style="width: 900.0px; height: 450.0px;" /></a>
</div>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/ensemble/plot_gradient_boosting_regression.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-regression-py"><span class="std std-ref">Gradient Boosting regression</span></a></li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_gradient_boosting_oob.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-oob-py"><span class="std std-ref">Gradient Boosting Out-of-Bag estimates</span></a></li>
</ul>
</div>
</div>
<div class="section" id="fitting-additional-weak-learners">
<span id="gradient-boosting-warm-start"></span><h3>1.11.4.3. Fitting additional weak-learners<a class="headerlink" href="#fitting-additional-weak-learners" title="Permalink to this headline">¶</a></h3>
<p>Both <a class="reference internal" href="generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor" title="sklearn.ensemble.GradientBoostingRegressor"><code class="xref py py-class docutils literal"><span class="pre">GradientBoostingRegressor</span></code></a> and <a class="reference internal" href="generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier" title="sklearn.ensemble.GradientBoostingClassifier"><code class="xref py py-class docutils literal"><span class="pre">GradientBoostingClassifier</span></code></a>
support <code class="docutils literal"><span class="pre">warm_start=True</span></code> which allows you to add more estimators to an already
fitted model.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">_</span> <span class="o">=</span> <span class="n">est</span><span class="o">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">n_estimators</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">warm_start</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="c1"># set warm_start and new nr of trees</span>
<span class="gp">>>> </span><span class="n">_</span> <span class="o">=</span> <span class="n">est</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span> <span class="c1"># fit additional 100 trees to est</span>
<span class="gp">>>> </span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">est</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">))</span>
<span class="go">3.84...</span>
</pre></div>
</div>
</div>
<div class="section" id="controlling-the-tree-size">
<span id="gradient-boosting-tree-size"></span><h3>1.11.4.4. Controlling the tree size<a class="headerlink" href="#controlling-the-tree-size" title="Permalink to this headline">¶</a></h3>
<p>The size of the regression tree base learners defines the level of variable
interactions that can be captured by the gradient boosting model. In general,
a tree of depth <code class="docutils literal"><span class="pre">h</span></code> can capture interactions of order <code class="docutils literal"><span class="pre">h</span></code> .
There are two ways in which the size of the individual regression trees can
be controlled.</p>
<p>If you specify <code class="docutils literal"><span class="pre">max_depth=h</span></code> then complete binary trees
of depth <code class="docutils literal"><span class="pre">h</span></code> will be grown. Such trees will have (at most) <code class="docutils literal"><span class="pre">2**h</span></code> leaf nodes
and <code class="docutils literal"><span class="pre">2**h</span> <span class="pre">-</span> <span class="pre">1</span></code> split nodes.</p>
<p>Alternatively, you can control the tree size by specifying the number of
leaf nodes via the parameter <code class="docutils literal"><span class="pre">max_leaf_nodes</span></code>. In this case,
trees will be grown using best-first search where nodes with the highest improvement
in impurity will be expanded first.
A tree with <code class="docutils literal"><span class="pre">max_leaf_nodes=k</span></code> has <code class="docutils literal"><span class="pre">k</span> <span class="pre">-</span> <span class="pre">1</span></code> split nodes and thus can
model interactions of up to order <code class="docutils literal"><span class="pre">max_leaf_nodes</span> <span class="pre">-</span> <span class="pre">1</span></code> .</p>
<p>We found that <code class="docutils literal"><span class="pre">max_leaf_nodes=k</span></code> gives comparable results to <code class="docutils literal"><span class="pre">max_depth=k-1</span></code>
but is significantly faster to train at the expense of a slightly higher
training error.
The parameter <code class="docutils literal"><span class="pre">max_leaf_nodes</span></code> corresponds to the variable <code class="docutils literal"><span class="pre">J</span></code> in the
chapter on gradient boosting in <a class="reference internal" href="#f2001" id="id13">[F2001]</a> and is related to the parameter
<code class="docutils literal"><span class="pre">interaction.depth</span></code> in R’s gbm package where <code class="docutils literal"><span class="pre">max_leaf_nodes</span> <span class="pre">==</span> <span class="pre">interaction.depth</span> <span class="pre">+</span> <span class="pre">1</span></code> .</p>
</div>
<div class="section" id="mathematical-formulation">
<h3>1.11.4.5. Mathematical formulation<a class="headerlink" href="#mathematical-formulation" title="Permalink to this headline">¶</a></h3>
<p>GBRT considers additive models of the following form:</p>
<blockquote>
<div><div class="math">
\[F(x) = \sum_{m=1}^{M} \gamma_m h_m(x)\]</div>
</div></blockquote>
<p>where <span class="math">\(h_m(x)\)</span> are the basis functions which are usually called
<em>weak learners</em> in the context of boosting. Gradient Tree Boosting
uses <a class="reference internal" href="tree.html#tree"><span class="std std-ref">decision trees</span></a> of fixed size as weak
learners. Decision trees have a number of abilities that make them
valuable for boosting, namely the ability to handle data of mixed type
and the ability to model complex functions.</p>
<p>Similar to other boosting algorithms GBRT builds the additive model in
a forward stagewise fashion:</p>
<blockquote>
<div><div class="math">
\[F_m(x) = F_{m-1}(x) + \gamma_m h_m(x)\]</div>
</div></blockquote>
<p>At each stage the decision tree <span class="math">\(h_m(x)\)</span> is chosen to
minimize the loss function <span class="math">\(L\)</span> given the current model
<span class="math">\(F_{m-1}\)</span> and its fit <span class="math">\(F_{m-1}(x_i)\)</span></p>
<blockquote>
<div><div class="math">
\[F_m(x) = F_{m-1}(x) + \arg\min_{h} \sum_{i=1}^{n} L(y_i,
F_{m-1}(x_i) + h(x))\]</div>
</div></blockquote>
<p>The initial model <span class="math">\(F_{0}\)</span> is problem specific, for least-squares
regression one usually chooses the mean of the target values.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">The initial model can also be specified via the <code class="docutils literal"><span class="pre">init</span></code>
argument. The passed object has to implement <code class="docutils literal"><span class="pre">fit</span></code> and <code class="docutils literal"><span class="pre">predict</span></code>.</p>
</div>
<p>Gradient Boosting attempts to solve this minimization problem
numerically via steepest descent: The steepest descent direction is
the negative gradient of the loss function evaluated at the current
model <span class="math">\(F_{m-1}\)</span> which can be calculated for any differentiable
loss function:</p>
<blockquote>
<div><div class="math">
\[F_m(x) = F_{m-1}(x) - \gamma_m \sum_{i=1}^{n} \nabla_F L(y_i,
F_{m-1}(x_i))\]</div>
</div></blockquote>
<p>Where the step length <span class="math">\(\gamma_m\)</span> is chosen using line search:</p>
<blockquote>
<div><div class="math">
\[\gamma_m = \arg\min_{\gamma} \sum_{i=1}^{n} L(y_i, F_{m-1}(x_i)
- \gamma \frac{\partial L(y_i, F_{m-1}(x_i))}{\partial F_{m-1}(x_i)})\]</div>
</div></blockquote>
<p>The algorithms for regression and classification
only differ in the concrete loss function used.</p>
<div class="section" id="loss-functions">
<span id="gradient-boosting-loss"></span><h4>1.11.4.5.1. Loss Functions<a class="headerlink" href="#loss-functions" title="Permalink to this headline">¶</a></h4>
<p>The following loss functions are supported and can be specified using
the parameter <code class="docutils literal"><span class="pre">loss</span></code>:</p>
<blockquote>
<div><ul class="simple">
<li>Regression<ul>
<li>Least squares (<code class="docutils literal"><span class="pre">'ls'</span></code>): The natural choice for regression due
to its superior computational properties. The initial model is
given by the mean of the target values.</li>
<li>Least absolute deviation (<code class="docutils literal"><span class="pre">'lad'</span></code>): A robust loss function for
regression. The initial model is given by the median of the
target values.</li>
<li>Huber (<code class="docutils literal"><span class="pre">'huber'</span></code>): Another robust loss function that combines
least squares and least absolute deviation; use <code class="docutils literal"><span class="pre">alpha</span></code> to
control the sensitivity with regards to outliers (see <a class="reference internal" href="#f2001" id="id14">[F2001]</a> for
more details).</li>
<li>Quantile (<code class="docutils literal"><span class="pre">'quantile'</span></code>): A loss function for quantile regression.
Use <code class="docutils literal"><span class="pre">0</span> <span class="pre"><</span> <span class="pre">alpha</span> <span class="pre"><</span> <span class="pre">1</span></code> to specify the quantile. This loss function
can be used to create prediction intervals
(see <a class="reference internal" href="../auto_examples/ensemble/plot_gradient_boosting_quantile.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-quantile-py"><span class="std std-ref">Prediction Intervals for Gradient Boosting Regression</span></a>).</li>
</ul>
</li>
<li>Classification<ul>
<li>Binomial deviance (<code class="docutils literal"><span class="pre">'deviance'</span></code>): The negative binomial
log-likelihood loss function for binary classification (provides
probability estimates). The initial model is given by the
log odds-ratio.</li>
<li>Multinomial deviance (<code class="docutils literal"><span class="pre">'deviance'</span></code>): The negative multinomial
log-likelihood loss function for multi-class classification with
<code class="docutils literal"><span class="pre">n_classes</span></code> mutually exclusive classes. It provides
probability estimates. The initial model is given by the
prior probability of each class. At each iteration <code class="docutils literal"><span class="pre">n_classes</span></code>
regression trees have to be constructed which makes GBRT rather
inefficient for data sets with a large number of classes.</li>
<li>Exponential loss (<code class="docutils literal"><span class="pre">'exponential'</span></code>): The same loss function
as <a class="reference internal" href="generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier" title="sklearn.ensemble.AdaBoostClassifier"><code class="xref py py-class docutils literal"><span class="pre">AdaBoostClassifier</span></code></a>. Less robust to mislabeled
examples than <code class="docutils literal"><span class="pre">'deviance'</span></code>; can only be used for binary
classification.</li>
</ul>
</li>
</ul>
</div></blockquote>
</div>
</div>
<div class="section" id="regularization">
<h3>1.11.4.6. Regularization<a class="headerlink" href="#regularization" title="Permalink to this headline">¶</a></h3>
<div class="section" id="shrinkage">
<span id="gradient-boosting-shrinkage"></span><h4>1.11.4.6.1. Shrinkage<a class="headerlink" href="#shrinkage" title="Permalink to this headline">¶</a></h4>
<p><a class="reference internal" href="#f2001" id="id15">[F2001]</a> proposed a simple regularization strategy that scales
the contribution of each weak learner by a factor <span class="math">\(\nu\)</span>:</p>
<div class="math">
\[F_m(x) = F_{m-1}(x) + \nu \gamma_m h_m(x)\]</div>
<p>The parameter <span class="math">\(\nu\)</span> is also called the <strong>learning rate</strong> because
it scales the step length the gradient descent procedure; it can
be set via the <code class="docutils literal"><span class="pre">learning_rate</span></code> parameter.</p>
<p>The parameter <code class="docutils literal"><span class="pre">learning_rate</span></code> strongly interacts with the parameter
<code class="docutils literal"><span class="pre">n_estimators</span></code>, the number of weak learners to fit. Smaller values
of <code class="docutils literal"><span class="pre">learning_rate</span></code> require larger numbers of weak learners to maintain
a constant training error. Empirical evidence suggests that small
values of <code class="docutils literal"><span class="pre">learning_rate</span></code> favor better test error. <a class="reference internal" href="#htf2009" id="id16">[HTF2009]</a>
recommend to set the learning rate to a small constant
(e.g. <code class="docutils literal"><span class="pre">learning_rate</span> <span class="pre"><=</span> <span class="pre">0.1</span></code>) and choose <code class="docutils literal"><span class="pre">n_estimators</span></code> by early
stopping. For a more detailed discussion of the interaction between
<code class="docutils literal"><span class="pre">learning_rate</span></code> and <code class="docutils literal"><span class="pre">n_estimators</span></code> see <a class="reference internal" href="#r2007" id="id17">[R2007]</a>.</p>
</div>
<div class="section" id="subsampling">
<h4>1.11.4.6.2. Subsampling<a class="headerlink" href="#subsampling" title="Permalink to this headline">¶</a></h4>
<p><a class="reference internal" href="#f1999" id="id18">[F1999]</a> proposed stochastic gradient boosting, which combines gradient
boosting with bootstrap averaging (bagging). At each iteration
the base classifier is trained on a fraction <code class="docutils literal"><span class="pre">subsample</span></code> of
the available training data. The subsample is drawn without replacement.
A typical value of <code class="docutils literal"><span class="pre">subsample</span></code> is 0.5.</p>
<p>The figure below illustrates the effect of shrinkage and subsampling
on the goodness-of-fit of the model. We can clearly see that shrinkage
outperforms no-shrinkage. Subsampling with shrinkage can further increase
the accuracy of the model. Subsampling without shrinkage, on the other hand,
does poorly.</p>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/ensemble/plot_gradient_boosting_regularization.html"><img alt="../_images/sphx_glr_plot_gradient_boosting_regularization_0011.png" src="../_images/sphx_glr_plot_gradient_boosting_regularization_0011.png" style="width: 480.0px; height: 360.0px;" /></a>
</div>
<p>Another strategy to reduce the variance is by subsampling the features
analogous to the random splits in <a class="reference internal" href="generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier" title="sklearn.ensemble.RandomForestClassifier"><code class="xref py py-class docutils literal"><span class="pre">RandomForestClassifier</span></code></a> .
The number of subsampled features can be controlled via the <code class="docutils literal"><span class="pre">max_features</span></code>
parameter.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Using a small <code class="docutils literal"><span class="pre">max_features</span></code> value can significantly decrease the runtime.</p>
</div>
<p>Stochastic gradient boosting allows to compute out-of-bag estimates of the
test deviance by computing the improvement in deviance on the examples that are
not included in the bootstrap sample (i.e. the out-of-bag examples).
The improvements are stored in the attribute
<code class="xref py py-attr docutils literal"><span class="pre">oob_improvement_</span></code>. <code class="docutils literal"><span class="pre">oob_improvement_[i]</span></code> holds
the improvement in terms of the loss on the OOB samples if you add the i-th stage
to the current predictions.
Out-of-bag estimates can be used for model selection, for example to determine
the optimal number of iterations. OOB estimates are usually very pessimistic thus
we recommend to use cross-validation instead and only use OOB if cross-validation
is too time consuming.</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/ensemble/plot_gradient_boosting_regularization.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-regularization-py"><span class="std std-ref">Gradient Boosting regularization</span></a></li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_gradient_boosting_oob.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-oob-py"><span class="std std-ref">Gradient Boosting Out-of-Bag estimates</span></a></li>
<li><a class="reference internal" href="../auto_examples/ensemble/plot_ensemble_oob.html#sphx-glr-auto-examples-ensemble-plot-ensemble-oob-py"><span class="std std-ref">OOB Errors for Random Forests</span></a></li>
</ul>
</div>
</div>
</div>
<div class="section" id="interpretation">
<h3>1.11.4.7. Interpretation<a class="headerlink" href="#interpretation" title="Permalink to this headline">¶</a></h3>
<p>Individual decision trees can be interpreted easily by simply
visualizing the tree structure. Gradient boosting models, however,
comprise hundreds of regression trees thus they cannot be easily
interpreted by visual inspection of the individual trees. Fortunately,
a number of techniques have been proposed to summarize and interpret
gradient boosting models.</p>
<div class="section" id="feature-importance">
<h4>1.11.4.7.1. Feature importance<a class="headerlink" href="#feature-importance" title="Permalink to this headline">¶</a></h4>
<p>Often features do not contribute equally to predict the target
response; in many situations the majority of the features are in fact
irrelevant.
When interpreting a model, the first question usually is: what are
those important features and how do they contributing in predicting
the target response?</p>
<p>Individual decision trees intrinsically perform feature selection by selecting
appropriate split points. This information can be used to measure the
importance of each feature; the basic idea is: the more often a
feature is used in the split points of a tree the more important that
feature is. This notion of importance can be extended to decision tree