Data Science notes

Giovanni Ficarra
October 6, 2020

Abstract
Some essential note about the course of Foundations of Data Science.
Most of the contents are from Doing Data Science, O’Reilly, 2014.
These notes are shared without any guarantee of complete correctness,
since I may have done typos or misunderstood something. Feel free to drop
an email at [email protected] to report errors.

1
1 Evaluation
The evaluation of binary classifiers compares two methods of assigning a binary
attribute (Wikipedia).
Let T P, T N, F P, F N be respectively the number of true positives, the number
of true negatives, the number of false positives and the number of false negatives,
Yi an observed value, Ŷi the prevision for that value, Ȳ the mean.

• Accuracy: How often the correct outcome is being predicted. How well
a binary classification test correctly identifies or excludes a condition.
TP + TN
ACC =
TP + TN + FP + FN
More: Wikipedia

• Precision (aka Positive Predictive Value): The fraction of relevant

instances among the retrieved instances.
TP
PPV =
TP + FP
More: Wikipedia
• Recall (aka Sensitivity or True Positive Rate): The fraction of rele-
vant instances that have been retrieved over the total amount of relevant
instances. The proportion of actual positives that are correctly identified
as such.
TP
TPR =
TP + FN
More: Wikipedia
• F-score: It combines precision and recall into a single score. It’s their
harmonic mean.
2T P
F1 =
2T P + T P + F N
More: Wikipedia

• Specificity (aka True Negative Rate): The proportion of actual neg-

atives that are correctly identified as such.
TN
TNR =
FP + TN
More: Wikipedia

2
• Negative Predictive Value: The proportions of negative results that
are true negative results.
TN
NPV =
TN + FN
More: Wikipedia

• Fall-out (False Positive Rate):

FPR = 1 − TNR

• Miss rate (False Negative Rate):

FNR = 1 − TPR

3
• False Discovery Rate:

F DR = 1 − P P V

• Mean squared error: The average squared distance between the pre-
dicted and actual values. It captures how much the predicted value varies
from the observed.

1X
n
M SE = (Yi − Ŷi )2 = SSE
n i=1
More: Wikipedia
• Root squared error: The square root of mean squared error.
v
u n
u1 X
RSE = t (Yi − Ŷi )2
n i=1

More: Wikipedia
• Mean absolute error: The average of the absolute value of the difference
between the predicted and actual values. It’s also the average horizontal
distance between each point and the identity line.

1X
n
M AE = Yi − Ŷi
n i=1
More: Wikipedia
• R-squared (aka coefficient of determination): The proportion of the
variance in the dependent variable that is predictable from the indepen-
dent variable(s). The proportion of variance explained by our model.
Pn
SSE 1
(Yi − Ŷi )2
R2 = 1 − =1− n
Pi=1
n
i=1 (Yi − Ȳi )
SST 1 2
n

It tells us the quality of our model comparing it with a naive one, that
ignores the Xi s and simply compute the average of the Yi s.

4
 The better the linear regression (on the right) fits the data in comparison to the
simple average (on the left graph), the closer the value of R2 is to 1.
The areas of the blue squares represent the squared residuals with respect to the
linear regression. The areas of the red squares represent the squared residuals
with respect to the average value.
More: Wikipedia
• p-values: Let’s make the null hypothesis that the coefficients of the equa-
tion of the line estimated through linear regression were 0; The p-value is
the probability of seeing the observed data under the null hypothesis.
It tells us how meaningful our model is, if it really represents what is hap-
pening behind the data or if it’s only casually similar to the data.
I.e., if the p-value relative to a certain coefficient is low, it is highly un-
likely to observe such a test-statistic under the null hypothesis, and the
coefficient we computed is highly likely to be nonzero and therefore sig-
nificant.
More: Wikipedia
• Receiver Operating Curve: It’s a probability curve, the plot of the
TPR against the FPR for a binary classification problem as you change a
threshold.
More: Wikipedia
• Area under the Receiver Operating Curve (AUC): It represents
degree or measure of separability, it tells us how much the model is capable
of distinguishing between classes (higher the AUC, better the model is at
predicting 0s as 0s and 1s as 1s).
More: TowardsDataScience

5
 • Area under the cumulative lift curve: captures how many times it
is better to use a model versus not using the model (i.e., just selecting at
random).
More: paper

2 Bias-variance decomposition of squared error

Ideally, one wants to choose a model that both accurately captures the regu-
larities in its training data, but also generalizes well to unseen data; but it is
typically impossible to do both simultaneously.

• The bias is an error from erroneous assumptions in the learning algorithm.

High bias can cause an algorithm to miss the relevant relations between
features and target outputs (underfitting).
• The variance is an error from sensitivity to small fluctuations in the
training set. High variance can cause an algorithm to model the random
noise in the training data, rather than the intended outputs (overfitting).

The more complex the model is, the more data points it will capture, and
the lower the bias will be. However, complexity will make the model ”move”
more to capture the data points, and hence its variance will be larger.

E[(Y − Ŷ )2 ] = (y − µŶ )2 + E[(µŶ − Ŷ )2 ] = bias2 + variance

More: Wikipedia