Introduction Statistical Learning
Introduction Statistical Learning
Olivier Roustant
& Laurent Carraro for Part 2
Mines Saint-Étienne
2016/09
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 1 / 39
Part 1 : Famous traps !
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 2 / 39
Part 1 : Famous traps !
What do you think of the correlation of 0.99 between the two variables
illustrated below ?
!"# $
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 3 / 39
Part 1 : Famous traps !
What do you think of the correlation of 0.52 between two daily returns
of French stocks in 2 different sectors (food and construction) ?
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 4 / 39
Part 1 : Famous traps !
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 5 / 39
Part 1 : Famous traps !
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 6 / 39
Part 1 : Famous traps !
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 7 / 39
Part 1 : Famous traps !
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 8 / 39
Part 1 : Famous traps !
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 9 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 10 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 11 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 12 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 13 / 39
Part 2 : A guiding example
Bayes classifier
Here :
0.5fX |G=i (x)
P(G = i|X = x) =
0.5fX |G=1 (x) + 0.5fX |G=2 (x)
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 14 / 39
Part 2 : A guiding example
L(1, 1) = L(2, 2) = 0
L(1, 2) = L(2, 1) = α > 0
The Bayes classifier cancels L(i, Ĝ(x)) where P(G = i|X = x) is the
highest.
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 15 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 16 / 39
Part 2 : A guiding example
A classifier is :
if Ŷ (x) > 0.5, then decide that x is "blue"
if Ŷ (x) < 0.5, then decide that x is "red"
if Ŷ (x) = 0.5, then ?
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 17 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 18 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 19 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 20 / 39
Part 2 : A guiding example
1 X
Ŷ (x) = Yi
k
xi ∈Nk (x)
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 21 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 22 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 23 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 24 / 39
Part 2 : A guiding example
Temporary conclusions
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 25 / 39
Part 2 : A guiding example
Remarks
for kNN, the bias is ≈ 0
for the linear model, the bias is 0 if there is no model error (good
basis functions).
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 26 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 27 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 28 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 29 / 39
Part 2 : A guiding example
Validation
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 30 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 31 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 32 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 33 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 34 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 35 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 36 / 39
Part 2 : A guiding example
Conclusions
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 37 / 39
Part 2 : A guiding example
Cross validation
k-fold cross validation (CV) consists in choosing training & test sets
among the data, and rotating them.
CV errors are computed by averaging.
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 38 / 39
Part 2 : A guiding example
Olivier Roustant & Laurent Carraro (EMSE) Introduction to Statistical Learning 2016/09 39 / 39