2.3.

The total sum of squares explained sum of squares and residual

sum of squares

In the previous subchapter, we minimised the sum of squared residuals:

n n

min ∑ e2i = min ∑(𝑌𝑖 − 𝑌̂𝑖 )2 .

i=1 i=1
The sum of squared residuals is denoted by SSE (sum of squares due to error). The variation of
the dependent variable around the sample mean can be divided into parts. Let us start with the
formula of the residual:
ei = 𝑌𝑖 − 𝑌̂𝑖 ,
Yi = 𝑌̂𝑖 + ei .
Subtracting the sample mean from both sides of the equation, we obtain:
𝑌𝑖 − 𝑌̅ = 𝑌̂𝑖 − 𝑌̅ + ei ,
Substituting 𝑌𝑖 − 𝑌̂𝑖 instead of ei , we get:
𝑌𝑖 − 𝑌̅ = 𝑌̂𝑖 − 𝑌̅ + 𝑌𝑖 − 𝑌̂𝑖 .
The difference between the actual value of Y and the mean value of Y has two parts:
- (𝑌̂𝑖 − 𝑌̅) the difference between the estimated value of Y and the mean value of Y, and
- (𝑌𝑖 − 𝑌̂𝑖 ) the difference between the actual value of Y and the estimated value of Y (Fig.
2.7).

Figure 2.7 The explained and unexplained parts of Y

Yi − Y ei = Yi − Yi

𝑌̂𝑖 = β̂0 + β̂1 ∙ Xi

Y
Yi − Y
β̂0

We take the squared sum of both sides of the equation, and we obtain:
2 2
∑𝑛𝑖=1(𝑌𝑖 − 𝑌̅ )2 = ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅) + ∑𝑛𝑖=1(𝑌𝑖 − 𝑌̂𝑖 ) + 2 ∙ ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅ ) ∙ (𝑌𝑖 − 𝑌̂𝑖 ),

2 ∙ ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅) ∙ (𝑌𝑖 − 𝑌̂𝑖 ) = 0.

From the first normal equation (2.4a), we get:
∑𝑛𝑖=1 𝑒𝑖 = ∑𝑛𝑖=1(𝑌𝑖 − 𝑌̂𝑖 ) = ∑𝑛𝑖=1 Yi − n ∙ β̂0 − β̂1 ∙ ∑𝑛𝑖=1 Xi =0,

2 ∙ ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅) ∙ (𝑌𝑖 − 𝑌̂𝑖 ) = 2 ∙ ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅ ) ∙ 𝑒𝑖 = 2 ∙ ∑𝑛𝑖=1(β̂0 + β̂1 ∙ Xi − 𝑌̅ ) ∙ 𝑒𝑖 ,

 2 ∙ ∑𝑛𝑖=1(β̂0 + β̂1 ∙ X i − 𝑌̅ ) ∙ 𝑒𝑖 = 2 ∙ (β̂0 ∙ ∑𝑛𝑖=1 𝑒𝑖 + β̂1 ∙ ∑ni=1 Xi ∙ 𝑒𝑖 − 𝑌̅ ∙ ∑ni=1 𝑒𝑖 ),

∑ 𝑒𝑖 = 0
𝑖=1

From the first normal equation (2.5a), we get:

n n

β̂1 ∙ ∑ Xi ∙ 𝑒𝑖 = β̂1 ∙ ∑ Xi ∙ (Yi − n ∙ β̂0 − β̂1 ∙ Xi ) = 0.

i=1 i=1
Finally we get:

2 2
∑𝑛𝑖=1(𝑌𝑖 − 𝑌̅ )2 = ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅) + ∑𝑛𝑖=1(𝑌𝑖 − 𝑌̂𝑖 ) , (2.10)

If we would like to analyse the total sample variability, we can examine the variation of the
dependent variable about the sample mean 𝑌̅ :
n

SST = ∑(𝑌𝑖 − 𝑌̅ )2
i=1
Variance Analysis
The total sum of squares shows the total variations of Y around the sample mean. The total sum
of squares consists of two parts: the explained sum of squares (SSR) and the residual sum of
squares (SSE):
𝑛
2
𝑆𝑆𝑅 = ∑(𝑌̂𝑖 − 𝑌̅ ) ,
𝑖=1
𝑛
2
𝑆𝑆𝐸 = ∑(𝑌𝑖 − 𝑌̂𝑖 ) .
𝑖=1

SST = 𝑆𝑆𝑅 + 𝑆𝑆𝐸.

The decomposition of the variance is:

𝑛 𝑛 𝑛
2 2
∑(𝑌𝑖 − 𝑌 ̅ )2 = ∑(𝑌̂𝑖 − 𝑌̅ ) + ∑(𝑌𝑖 − 𝑌̂𝑖 ) .
𝑖=1 𝑖=1 𝑖=1
The variation of the estimated values of Y around the sample mean is the explained sum of
squares. It is the sum of squares due to the regression. SSR is the component of total variation
in Y around its mean that is explained by the regression. SSE is the component of total variation
in Y around its mean that is not explained by the regression.

“Looking at the overall fit of an estimated model is useful not only for evaluating the quality of the
regression, but also for comparing models that have different data sets or combinations of independent
variables. We can never be sure that one estimated model represents the truth any more than another,
but evaluating the quality of the fit of the equation is one ingredient in a choice between different
formulations of a regression model. Be careful, however! The quality of the fit is a minor ingredient in
this choice, and many beginning researchers allow themselves to be overly influenced by it.”
Studenmund (2014)
Example 2.5
Let us give the decomposition of the variance for exercise 2.4.

Price Floor area

Million Square
Number HUF metres (𝒀𝒊 − 𝒀) 𝒀𝒊 (𝒀𝒊 − 𝒀) (𝒀𝒊 − 𝒀𝒊 )
1 16.20 48.00 -6.90 16.75 -6.351 -0.55
2 19.40 55.00 -3.70 19.17 -3.928 0.23
3 24.50 71.00 1.40 24.71 1.609 -0.21
4 28.20 82.00 5.10 28.51 5.416 -0.31
5 35.20 100.00 12.10 34.74 11.646 0.46
6 29.10 85.00 6.00 29.55 6.455 -0.45
7 24.60 70.00 1.50 24.36 1.263 0.24
8 25.80 73.00 2.70 25.40 2.302 0.40
9 26.10 74.00 3.00 25.75 2.648 0.35
10 22.20 66.00 -0.90 22.98 -0.121 -0.78
11 14.50 35.00 -8.60 12.25 -10.850 2.25
12 17.60 53.00 -5.50 18.48 -4.620 -0.88
13 24.80 73.00 1.70 25.40 2.302 -0.60
14 14.10 39.00 -9.00 13.63 -9.466 0.47
15 23.75 67.00 0.65 23.32 0.225 0.43
16 16.80 49.00 -6.30 17.09 -6.005 -0.29
17 18.40 51.00 -4.70 17.78 -5.313 0.62
18 21.50 61.00 -1.60 21.25 -1.852 0.25
19 16.20 53.00 -6.90 18.48 -4.620 -2.28
20 43.00 122.00 19.90 42.36 19.260 0.64
Sum 461.95 1327.00
Mean 23.0975 66.35
Sum of
squares 987.512375 972.9288 14.5836

SST = ∑(𝑌𝑖 − 𝑌̅)2 = 987.5124

𝑖=1

𝑛
2
𝑆𝑆𝑅 = ∑(𝑌̂𝑖 − 𝑌̅ ) = 972.9288,
𝑖=1
𝑛
2
𝑆𝑆𝐸 = ∑(𝑌𝑖 − 𝑌̂𝑖 ) = 14.5836.
𝑖=1
SST = SSR + SSE,
𝑛

𝑆𝑆𝑇 = ∑(𝑌𝑖 − 𝑌̅ )2 = 987.5124 = 972.9288 + 14.5836.

𝑖=1
2.4. Goodness of fit

The main purpose of the regression analysis to provide a good estimation and explain the
variation of the dependent variable Y. Using the R-squared (coefficient of determination)
indicator we can measure what proportion of the total sum squares is explained by the
regression. R-squared is the coefficient of determination providing information about how well
the independent variable explains the dependent variable. R-squared is equal to the ratio of the
explained sum of squares to the total sum of squares:
𝑆𝑆𝑅 ∑𝑛 (𝑌̂ −𝑌̅)2
R2 = SST = ∑𝑖=1 𝑖
n (𝑌 −𝑌
̅ )2
.
i=1 𝑖

The indicator shows the proportion of the sample variation in Y that is explained by the
independent variable (X):

𝑆𝑆𝑅 𝑆𝑆𝑇−𝑆𝑆𝐸 𝑆𝑆𝐸

R2 = SST = = 1 − SST.
SST

The value of the coefficient of determination can be between zero and 1:

0 ≤ R2 ≤ 1.
The coefficient is equal to one, if the data points fit on the regression line.

Figure 2.8 The perfect fit

𝑌̂𝑖 = β̂0 + β̂1 ∙ Xi

β̂0 R2 = 1

In this case the fit is perfect, and the residuals are zero (Fig. 2.8). Indeed, the minimum criterion
of the residual sum of squares is equivalent to the maximum criterion of the coefficient of
determination:
𝑆𝑆𝑅 𝑆𝑆𝑇 − 𝑆𝑆𝐸 𝑆𝑆𝐸
R2 = = = 1− .
SST SST SST

R2 is equal to zero, if the dependent and independent variable are not correlated:
𝑆𝑆𝑅
R2 = = 0,
SST
𝑛
2
𝑆𝑆𝑅 = ∑(𝑌̂𝑖 − 𝑌̅) = 0.
𝑖=1
The regression line is a horizontal (regression) line, if the value of x changes, the value of Y
remains the same; i.e. it does not change (Fig. 2.9):
𝑌̂𝑖 = 𝑌̅ .

Figure 2.9 Zero coefficient of determination

𝑌̂𝑖 = β̂0 + β̂1 ∙ Xi

β̂0

R2 = 0

If the value of the coefficient (R2 ) is greater, and the regression line fits the sample data better.

What is the acceptable value of (R2 )? There is no general rule to evaluate R2 ; however, a
coefficient above 0.5 may represent a good fit. It is very important to discover the variables that
significantly influence the dependent variable. Our estimation can result in a high determination
of the coefficient (R2 ); however the independent variable chosen and the dependent variable
can, in fact, be influenced by another variable.

Example 2.6
Try to answer the following question:
How well does the estimated regression function fit the data?
Let us compute the coefficient of determination for the example 2.4.

𝑆𝑆𝑅 972.9288
R2 = = = 0.985232.
SST 987.5124

On the basis of the result of R2 , we can state that 98.5232% of the total sum of squares is
explained by the regression equation.
We started the second chapter by introducing the linear correlation coefficient. Finally, at the
end of the chapter, we return to a discussion of the correlation coefficient. The main reason for
this is that there is a relationship between the coefficient of determination (R2 ) and the linear
correlation coefficient. In the last chapter, we discuss the association between the two
indicators.
2.5. The relationship between 𝐑𝟐 and the linear correlation
coefficient

In the previous chapter, we have found that the coefficient of determination is as follows:
𝑆𝑆𝑅 ∑𝑛 (𝑌̂𝑖 −𝑌̅)2
R2 = = ∑𝑖=1
n ̅ 2
.
SST i=1(𝑌𝑖 −𝑌)

Let us give the correlation coefficient:

∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅) ∙ (𝑌𝑖 − 𝑌̅ )

𝑟𝑥𝑦 = ,
√∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅ )2 ∙ ∑𝑛𝑖=1(𝑌𝑖 − 𝑌̅)2

The square of the linear correlation coefficient is equal to the coefficient of determination (R2 ):
2
(∑𝑛𝑖=1(𝑋𝑖 − X) ∙ (𝑌𝑖 − 𝑌̅))2
R = 𝑛 = 𝑟𝑥𝑦 2 .
∑𝑖=1(Xi − 𝑋̅)2 ∙ ∑ni=1(𝑌𝑖 − 𝑌̅)2
Let us prove the statement:
2 2
∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅ ) = ∑𝑛𝑖=1(β̂0 + β̂1 ∙ Xi − 𝑌̅ ) . (2.11)

Substituting the regression parameters into the equation (2.11):

β̂0 = 𝑌̅ − β̂1 ∙ 𝑋̅,

∑𝑛𝑖=1(𝑋𝑖 − X) ∙ (𝑌𝑖 − 𝑌̅ )
β̂1 = ,
∑𝑛𝑖=1(Xi − 𝑋̅)2

2 2 2
∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅ ) = ∑𝑛𝑖=1(β̂0 + β̂1 ∙ Xi − 𝑌̅ ) = ∑𝑛𝑖=1(𝑌̅ − β̂1 ∙ 𝑋̅ + β̂1 ∙ Xi − 𝑌̅ ) = ∙,

𝑛
2
= ∑(−β̂1 ∙ 𝑋̅ + β̂1 ∙ Xi ) =
𝑖=1
𝑛 𝑛
2 (∑𝑛𝑖=1(𝑋𝑖 − X) ∙ (𝑌𝑖 − 𝑌̅ ))2
= β̂1 ̅ 2
∙ ∑(Xi − 𝑋) = ∑(Xi − 𝑋̅)2 =
(∑𝑛𝑖=1(Xi − 𝑋̅)2 )2
𝑖=1 𝑖=1

(∑𝑛𝑖=1(𝑋𝑖 − X) ∙ (𝑌𝑖 − 𝑌̅ ))2

=
∑𝑛𝑖=1(Xi − 𝑋̅)2
2
Insert the given formula instead of ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅ ) into the equation for R2 :
2
𝑆𝑆𝑅 ∑𝑛𝑖=1(𝑌̂𝑖 − 𝑌̅ )
2
R = = ,
SST ∑ni=1(𝑌𝑖 − 𝑌̅)2
 2
(∑𝑛 ̅
𝑖=1(𝑋𝑖 −X)∙(𝑌𝑖 −𝑌 ))
R2 = ∑𝑛 ( ̅ ) 2 ∙∑n (𝑌 −𝑌̅ )2
= 𝑟𝑥𝑦 2 . (2.12)
𝑖=1 iX −𝑋 i=1 𝑖

At this moment, we know that the association between the two indicators is valid if there is only
one independent variable and one dependent variable. Later (in Chapter 5), we will examine
the validity of (2.12) in multiple regression models. Do not forget that the Pearson correlation
coefficient measures the strength and direction of the linear association between two variables.

Example 2.7
Let us compute the linear correlation coefficient for exercise 2.4.

∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅) ∙ (𝑌𝑖 − 𝑌̅) 2811.1675

𝑟𝑥𝑦 = = = 0.9925885,
√8122.55 ∙ 987.512375
√∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅)2 ∙ ∑𝑛𝑖=1(𝑌𝑖 − 𝑌̅)2

𝑟𝑥𝑦 2 = 0.985232 = 𝑅2 .
There is a strong linear association between the dependent and independent variable.