Non è possibile visualizzare l'immagine.

Lab 8 – Simple linear

regression
PROBABILITY AND STATISTICS - LABORATORY

BACHELOR’S DEGREE IN COMPUTER SCIENCE

A.Y. 2022/2023

DOTT. MATTEO CALGARO

[email protected]

Slides are rearranged from Prof. Silvia Storti’s lessons

Non è possibile visualizzare l'immagine.

2
Simple regression equation Observed value (y)

𝑌 = 𝛼 + 𝛽𝑥 + 𝑒
Intercept Slope

Describe a linear relationship between Y and x where:

u x is the independent variabile
u Y is the response/dependent variable
u e represents the random error. It is a random
variable having mean 0 Estimated regression line

NORMAL EQUATIONS
The Least Squares Estimators are used to estimate 𝛼 and 𝛽. It finds the line that minimizes the sum of the
∑! 𝑥! 𝑌! − 𝑛𝑥̅ 𝑌% squared differences between the observed
%
𝐴 = 𝑌 − 𝐵𝑥;̅ 𝐵 =
∑! 𝑥!" − 𝑛𝑥^2
̅ data points and the predicted values on the
line.

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

3
Simple regression equation – in R
set.seed(123) # Create some random data
n <- 10; x = 1:n; y = 1:n + runif(n, -2, 2)
df <- data.frame(x = x, y = y)
# Least Squares Estimates
B <- (sum(x * y) - n * mean(x) * mean(y)) /
(sum(x^2) - n * mean(x)^2)
A <- mean(y) - B * mean(x)
# Plot the regression line
library(ggplot2)
ggplot(df, aes(x = x, y = y)) +
geom_segment(aes(x = x, xend = x, y = A + B*x,
yend = y), color = "cyan4") +
geom_point(color = "brown3") +
geom_abline(aes(slope = B, intercept = A)) +
labs(title = "Estimated regression line",
subtitle = paste0("A = ", round(A, 2), ", B = ", round(B, 2)))

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

4
The lm() function
The lm() function uses the least squares regression method to fit a linear model:
u y is a vector of dependent variable values
u x is a vector of independent variable values
u data is the data.frame where x and y variables are contained
model <- lm(y ~ x, data = df)
model

# Call:
# lm(formula = y ~ x, data = df)
# Coefficients:
# (Intercept) x
# 0.2702 1.0078

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

5
lm()’s output
Using the names() on the linear model estimated by lm() we can see the many outputs
available:

names(model)

[1] "coefficients" "residuals" "effects" "rank"

[5] "fitted.values" "assign" "qr" "df.residual"
[9] "xlevels" "call" "terms" "model"

Where:
u coefficients is a named vector of coefficients
u the residuals are the response minus the fitted.values
u fitted.values are the fitted mean values

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

6
lm()’s output
To obtain the estimated coefficients we can use the following codes:
coefficients(model); coef(model)
model$coefficients
# (Intercept) x
# 0.2702156 1.0077772

To obtain the fitted values:

fitted.values(model); fitted(model)
model$fitted.values
# 1 2 3 4 5 6 …
# 1.277993 2.285770 3.293547 4.301324 5.309101 6.316879 …

Similarly, to obtain the residuals:

residuals(model)
model$residuals
# 1 2 3 4 5 6 …
# -1.12768 0.86745 -0.65763 1.23074 1.45276 -2.13465 …

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

Exercise 1 – Least squares estimates

The dataset below contain the weights and flipper lenghts of 10 penguins from the
palmerpenguins dataset:
weights <- c(4100, 3800, 4300, 4550, 3775, 2900, 4600, 5400, 4100, 4500)
flipper_lengths <- c(216, 191, 210, 205, 199, 187, 209, 228, 210, 211)

A. Build a simple regression line using the flipper length as the independent variable and the
weight as the response variable. Manually compute the least squares estimates for the
intercept and slope .
B. Use the lm() function and compare the coefficients to those obtained in A.
C. Use ggplot2() to plot the weights and flipper lengths as points. Then add the regression
line using the geom_abline() function.
D. Remove the regression line from the plot above and replace it using the geom_smooth()
function with method = "lm" and formula = "weights ~ flipper_length".

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
8
UTIO
SOL
Exercise 1 – Least squares estimates
A. Build a simple regression line using the flipper length as the independent variable and
the weight as the response variable. Manually compute the least squares estimates
for the intercept and slope.
weights <- c(4100, 3800, 4300, 4550, 3775, 2900, 4600, 5400, 4100, 4500)
flipper_lengths <- c(216, 191, 210, 205, 199, 187, 209, 228, 210, 211)

data <- data.frame(weights = weights, flipper_lengths = flipper_lengths)

n <- nrow(data)
# 47.88677
slope <- (sum(flipper_lengths * weights) –
n * mean(flipper_lengths) * mean(weights)) /
(sum(flipper_lengths^2) - n * mean(flipper_lengths)^2)
# -5690.908
intercept <- mean(weights) - slope * mean(flipper_lengths)

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
9
UTIO
SOL
Exercise 1 – Least squares estimates

B. Use the lm() function and compare the coefficients to those obtained in A.
C. Use ggplot2() to plot the weights and flipper lengths as points. Then add the
regression line using the geom_abline() function.

model <- lm(weights ~ flipper_lengths, data = data)

coef(model) # Intercept = -5690.90767, slope = 47.88677

library(ggplot2)
ggplot(data, aes(x = flipper_lengths,
y = weights)) +
geom_point() +
geom_abline(aes(intercept = intercept,
slope = slope))

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
10
UTIO
SOL
Exercise 1 – Least squares estimates

D. Remove the regression line from the plot above and replace it using the
geom_smooth() function with method = "lm" and formula = "y ~ x".

ggplot(data, aes(x = flipper_lengths, y = weights)) +

geom_point() +
geom_smooth(method = "lm", formula = "y ~ x")

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

11
Assessing the model: plot the regression line

Graphical evaluation
of a simple regression
line involves visually
inspecting the scatter
plot of the data Strong Weak No No linear
points and the fitted relationship relationship relationship relationship
line to assess the line's
appropriateness in
capturing the overall
trend and
relationship between
the variables.

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

12
Assessing the model: amount of variation
.𝟎 + 𝜷
-𝒊 = 𝜷
𝒚 . 𝟏 𝒙𝒊
The dependent variable 𝑦 has a certain
amount of variability. When we build a
regression model we can decompose the
𝒚𝒊
total variability into two components: a
portion explained by the independent
variable 𝑥 and the residual, which remains Unexplained sum of
unexplained. squares 𝑦$ − 𝑦2$ (
𝑆!! = 𝑆𝑆" + 𝑆𝑆#
' ' '
( ( = $(𝑌*$ − 𝑌)
$(𝑌$ −𝑌) ( ( + $(𝑌$ −𝑌*$ )( Total sum of
$%& $%& $%& squares 𝑦$ − 𝑦( (
Explained sum of
u 𝑆## = Total sum of squares squares 𝑦2$ − 𝑦( (
u 𝑆𝑆$ = Explained sum of squares (sum of
squares explained thanks to regression)
u 𝑆𝑆% = Residual sum of squares (unexplained +
𝒚
sum of squares)

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

13
Coefficient of determination 𝑅 !
► By decomposing the errors, we can gain insights into the variability explained by the
model and the remaining unexplained variation.
► The coefficient of determination 𝑹𝟐 measures the proportion of the variance in the
dependent variable that is explained by the independent variables in a regression
model.
► It ranges from 0 to 1, with 0 indicating no explanation and 1 indicating full explanation.

𝑆!! = 𝑆𝑆" + 𝑆𝑆#

' ' '
( ( = $(𝑌*$ − 𝑌)
$(𝑌$ −𝑌) ( ( + $(𝑌$ −𝑌*$ )( Higher 𝑅 ( values indicate a better
$%& $%& $%& fit of the model to the data.
𝑆𝑆# 𝑆𝑆"
𝑅( = 1 − =
𝑆!! 𝑆!!

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

14

Exercise 2 – Determination index

Consider a dataset that contains information about the number of hours studied
(independent variable) and the corresponding scores obtained on a test (dependent
variable) for a group of students.
hours <- c(0, 1, 1, 1, 2, 2, 3, 3, 4, 4, 4, 4, 5, 5, 5)
scores <- c(2, 2.5, 3.5, 4, 2, 5, 6, 5, 7.5, 5, 7, 8, 8, 9, 8.5)

A. Estimate a model without the intercept. Minimise the sum of squares for a model with the
sole slope and compute its value in the dataset. Then use the lm() function with a formula
of the form "y ~ -1 + x" to estimate the same model, named model1, and compare the
two values.
B. As above, estimate an intercept-only model, named model2. Then use the lm() function
with a formula of the form "y ~ 1" to estimate the same model and compare the two
values. What do you notice?
C. Estimate a third model, named model3, with intercept and slope. Plot the regression lines
for model1, model2, and model3 over the scatterplot of the data.
D. For model3, compute 𝑆## , 𝑆𝑆$ , and 𝑆𝑆% . Then compute the determination index.

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
15
UTIO
SOL
Exercise 2 – Determination index
A. Estimate a model without the intercept. Minimise the sum of squares for a model with the
sole slope and compute its value in the dataset. Then use the lm() function with a formula
of the form "y ~ -1 + x" to estimate the same model, named model1, and compare
the two values.
' ' ' '

$ 𝑦$ − 𝑏𝑥$ ( = $ 𝑦$( + 𝑏 ( $ 𝑥$( − 2𝑏 $ 𝑦$ 𝑥$ Function to minimise

$%& $%& $%& $%&
' ' ' ' '
𝜕 Setting the derivate
$ 𝑦$( + 𝑏 ( $ 𝑥$( − 2𝑏 $ 𝑦$ 𝑥$ = 2𝑏 $ 𝑥$( − 2 $ 𝑦$ 𝑥$ = 0
𝜕𝑏 equal to zero
$%& $%& $%& $%& $%&

hours <- c(0, 1, 1, 1, 2, 2, 3, 3, 4, 4, 4, 4, 5, 5, 5)

scores <- c(2, 2.5, 3.5, 4, 2, 5, 6, 5, 7.5, 5, 7, 8, 8, 9, 8.5)
data <- data.frame(hours = hours, scores = scores)
slope <- sum(scores * hours) / sum(hours^2) ∑'$%& 𝑦$ 𝑥$
model1 <- lm(scores ~ 0 + hours, data = data) 𝑏= '
∑$%& 𝑥$(
coef(model1)

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
16
UTIO
SOL
Exercise 2 – Determination index
B. As above, estimate an intercept-only model, named model2. Then use the lm()
function with a formula of the form "y ~ 1" to estimate the same model and
compare the two values. What do you notice?
' ' '

$ 𝑦$ − 𝑎 ( = $ 𝑦$( + 𝑛𝑎( − 2𝑎 $ 𝑦$ Function to minimise

$%& $%& $%&
' ' '
𝜕 Setting the derivate
$ 𝑦$( + 𝑛𝑎( − 2𝑎 $ 𝑦$ = 2𝑛𝑎 − 2 $ 𝑦$ = 0
𝜕𝑎 equal to zero
$%& $%& $%&

mean(scores)
∑'$%& 𝑦$
model2 <- lm(scores ~ 1, data = data) 𝑎= = 𝑦(
model2 𝑛

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
17
UTIO
SOL
Exercise 2 – Determination index

C. Estimate a third model, named model3, with intercept and slope. Plot the regression
lines for model1, model2, and model3 over the scatterplot of the data.
model3 <- lm(scores ~ hours, data = data)
ggplot(data, aes(x = hours, y = scores)) +
geom_point() +
geom_abline(aes(intercept = 0,
slope = coef(model1), color = "Slope")) +
geom_abline(aes(intercept = coef(model2),
slope = 0, color = "Intercept")) +
geom_abline(aes(intercept = coef(model3)[1],
slope = coef(model3)[2],
color = "Intercept and Slope")) +
guides(color = guide_legend("Model"))

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
18
UTIO
SOL
Exercise 2 – Determination index

D. For model3, compute 𝑆!! , 𝑆𝑆" , and 𝑆𝑆# . Then compute the determination index.

Syy <- sum((scores - mean(scores))^2)

SSe <- sum((fitted.values(model3) - mean(scores))^2)
SSr <- sum((scores - fitted.values(model3))^2)
# The same result as above
SSr <- sum(residuals(model3)^2)

R2 <- 1 - SSr/Syy
# The 82.85% of the scores variability is explained by the different
# amount of study hours!

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

19
Assessing the model: residuals
satisfactory funnel
Residuals are the differences between the
observed values 𝑦 of the dependent variable and
the values predicted by the regression model 𝑦.
2

If the regression model is correct, they should be

normally distributed and characterised by:
► Zero Mean: by construction, the sum of
residuals is zero, indicating that, on average,
the model is unbiased. double bow non linear
► Independence: Residuals should be
independent of each other, meaning the error
in one observation does not affect the error in
another.
► Constant Variance: Residuals should have a
constant variance, implying that the spread of
residuals should not change across different
values of the independent variable.
Dott. Matteo Calgaro
Non è possibile visualizzare l'immagine.

20
Assessing the model: residuals
Residual analysis helps assess how well the regression model captures the patterns and
trends in the data. They are usually standardised by dividing them by the unbiased
estimate of the standard deviation.
𝑦$ − 𝑦2$ Residuals
𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑𝑖𝑠𝑒𝑑 𝑟𝑒𝑠𝑖𝑑𝑢𝑎𝑙$ =
(
∑'$%& 𝑦$ − 𝑦2$ Unbiased
𝑛−2 SD estimate
They are useful for:
► Checking normality: Standardized residuals should be distributed as a standardized
normal random variable.
► Detecting Heteroscedasticity: Plotting standardized residuals against the predicted
values or the independent variable can reveal patterns, such as a funnel shape or cone
shape, which may suggest heteroscedasticity.
► Identifying Outliers: points with standardized residuals that fall outside a certain range
(e.g., ±2 or ±3) may indicate observations that have a significant impact on the
regression model. These outliers might be influential in influencing the estimated
coefficients or affecting the overall fit of the model.
Dott. Matteo Calgaro
Non è possibile visualizzare l'immagine.

21
Assessing the model: residuals’ plots
# Define a model
m1 <- lm(y ~ x, data = data)
# Extract the residuals
res <- residuals(m1)
# Extract the fitted values
fitted <- fitted(m1)
# Compute the standardized residuals
stdres <- residuals(m1) /
sqrt(sum(residuals(m1)^2)/m1$df.residual)
# Plot the standardized residuals and the
# fitted values
ggplot(df, aes(x = fitted, y = stdres)) +
geom_point(color = "brown3") +
geom_hline(aes(yintercept = 0),
linetype = "dashed") +
labs(x = "Fitted values",
y = "Standardized Residuals")
Dott. Matteo Calgaro
Non è possibile visualizzare l'immagine.

22
Assessing the model: residuals’ normality
Residuals' normality in simple linear regression can be
assessed using several methods:
► Normality test: Shapiro-Wilk or Anderson-Darling tests. P-
values > 0.05.
► Skewness and Kurtosis: values close to zero.
► Visual inspection: residuals vs fitted values, residuals
histogram and density plot, qq-plots.

A Q-Q plot compares the quantiles of the observed

residuals to the quantiles of a theoretical normal
distribution. If the points on the plot closely follow a straight
line, it suggests that the residuals are normally distributed.

probs <- seq(0, 1, by = 1/n)

theoretical_q <- qnorm(probs)
observed_q <- quantile(stdres, probs)
plot(theoretical_q, observed_q); abline(0,1)

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

23
Linear model summary()
In R, the command summary() used for a linear model object (e.g., produced by lm()
function) returns all the summary information about the regression model.

set.seed(123); n <- 10; x = 1:n function call

y = 1:n + runif(n, -2, 2)
df <- data.frame(x = x, y = y) residuals summary
model <- lm(y ~ x, data = df)
summary(model)

coefficient estimates

The lower the better

The higher the better

𝑺𝑺𝒓𝒆𝒔𝒊𝒅𝒖𝒂𝒍
𝑹𝟐𝒂𝒅𝒋
Is an intercept-only model better 𝑹𝟐
than this one? No! (p-value < 0.05)
𝑺𝑺𝒓𝒆𝒈𝒓𝒆𝒔𝒔𝒊𝒐𝒏
Dott. Matteo Calgaro 𝑭=
𝑺𝑺𝒓𝒆𝒔𝒊𝒅𝒖𝒂𝒍
Non è possibile visualizzare l'immagine.

24

Exercise 3 – Regression on penguins

Consider the penguins dataset from the palmerpenguins library. Use the
flipper_length_mm as the independent variable and study its relationship with the
body_mass_g variable (dependent variable). Estimate a simple regression model.

library(palmerpenguins)
df <- na.omit(penguins[, c("body_mass_g", "flipper_length_mm")])
A. Plot the body weights, the flipper lengths and draw the regression line in red.
B. How much variability of the body weights is explained by the model? (hint: compute the determination
index)
C. Compute the standardised residuals and plot them together with the fitted values. Are they
satisfactory? (hint: do you see any pattern?)
D. Compute the theoretical and observed quantiles for the standardised residuals. Draw the Q-Q plot.
Then use the function plot() on the regression model output. What are you obtaining?
E. Do A., B., C., and D., but instead of using the flipper length as independent variable, see if the
bill_length_mm variable could be a good regressor for body weight (hint: how is the determination
index? How are the standardised residuals distributed?)

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
25
UTIO
SOL
Exercise 3 – Regression on penguins

A. Plot the body weights, the flipper lengths and draw the regression line in red.

m1 <- lm(body_mass_g ~ flipper_length_mm,

data = df)
ggplot(data = df, aes(x = flipper_length_mm,
y = body_mass_g)) +
geom_point() +
geom_abline(aes(slope = coef(m1)[2],
intercept = coef(m1)[1]), color = "red") +
labs(x = "Flipper length (mm)",
y = "Body mass (g)",
title = "Palmer penguins",
subtitle = "Flipper length and body mass")

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
26
UTIO
SOL
Exercise 3 – Regression on penguins

B. How much variability of the body weights is explained by the model?

# Automatically
summary(m1)

# Or manually
SSy <- sum((df$body_mass_g –
mean(df$body_mass_g))^2)
SSexp <- sum((fitted(m1) –
mean(df$body_mass_g))^2)
SSunexp <- sum(residuals(m1)^2)
R2 <- SSexp/SSy

# ~ 76% of the total variance

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
27
UTIO
SOL
Exercise 3 – Regression on penguins
C. Compute the standardised residuals and plot them together with the fitted values.
Are they satisfactory?

df$stdres <- residuals(m1) /

sqrt(sum(residuals(m1)^2)/(nrow(df) - 2))
df$fitted <- fitted.values(m1)

ggplot(df, aes(x = fitted,

y = stdres)) +
geom_point() +
labs(x = "Fitted values",
y = "Standardized Residuals")

# No patterns, they are satisfactory

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
28
UTIO
SOL
Exercise 3 – Regression on penguins
D. Compute the theoretical and observed quantiles for the
standardised residuals. Draw the Q-Q plot. Then use the function
plot() on the regression model output. What are you obtaining?
Assess normality
probs <- seq(0, 1, length.out = nrow(df))
df$observed_q <- quantile(df$stdres, probs = probs)
df$theoretical_q <- qnorm(probs)
ggplot(df, aes(x = theoretical_q, y = observed_q)) +
geom_point() +
geom_abline(aes(intercept = 0, slope = 1)) +
labs(x = "Theoretical quantiles", "Observed quantiles")

par(mfrow = c(2,2)) std. res. ≅ obs. quantiles

plot(m1) Used to detect heteroschedasticity,
patterns, and outliers
Used to detect influential observations that
have a significant impact on the regression
Dott. Matteo Calgaro
Non è possibile visualizzare l'immagine.

N
29
UTIO
SOL
Exercise 3 – Regression on penguins
E. Do A., B., C., and D. but instead of using the flipper length as
independent variable, see if the bill_length_mm variable could
be a good regressor for body weight.
df <- na.omit(penguins[, c("body_mass_g", "bill_length_mm")])
m2 <- lm(body_mass_g ~ bill_length_mm, data = df)
ggplot(data = df, aes(x = bill_length_mm, y = body_mass_g)) +
geom_point() + geom_abline(aes(slope = coef(m2)[2],
intercept = coef(m2)[1]), color = "red") +
labs(x = "Bill length (mm)", y = "Body mass (g)",
title = "Palmer penguins",
subtitle = "Bill length and body mass")
summary(m2)

# F-statistic’s p-value is very significant: the model is so

# much better than an intercept-only model. However the
# amount of variability of body weight explained by it,
# R2 = 0.3542, is very low.

Dott. Matteo Calgaro

Non è possibile visualizzare l'immagine.

N
30
UTIO
SOL
Exercise 3 – Regression on penguins
par(mfrow = c(2,2)
plot(m2)

u In "Residuals vs Fitted" plot we can clearly see a funnel

meaning heteroskedasticity of the residuals (here the
residuals are not standardised).
u Heteroskedasticity is also confirmed in "Scale-Location"
plot (the square root of the standardized residuals
absolute values) where we can clearly see a positive
trend (the variance increases with the fitted values).
u In "Normal Q-Q" plot we can’t see deviations from
normality.
u In "Residuals vs Leverage" we can’t see any influential
observation.
In summary, residuals don’t have a constant variance and
the model has a low goodness of fit. More complex models
and/or data transformation are needed to improve it.

Dott. Matteo Calgaro