STAT 3008 Applied Regression Analysis

Tutorial 1: Estimates

September 25, 2023

1 Abbreviation
For simple linear regression, yi = 0 + 1 xi + "i , i = 1, 2, ..., n.
n n
1X 1X
ȳ = yi , x̄ = xi ,
n n
i=1 i=1
n
X Xn
SY Y = (yi ȳ)2 = yi2 nȳ 2 ,
i=1 i=1
n
X Xn
SXX = (xi x̄)2 = x2i nx̄2 ,
i=1 i=1
n
X n
X
SXY = (xi x̄) (yi ȳ) = x i yi nx̄ȳ,
i=1 i=1
n
X n
X
ˆ0 ˆ1 xi )2 = SY Y SXY 2 ˆ2 SXX.
RSS = "ˆ2i = (yi = SY Y 1
SXX
i=1 i=1

2 Least Squares Estimates (LSE)

“Least square” is a criterion, an idea to find the “best fit”. It doesn’t matter what the
distribution of yi is and what the specific model is. Minimize RSS (Residual Sum of Squares)
to get estimates of parameters.
Write down RSS, take partial derivatives to each parameter i, let them equal to 0 and then
get estimates.
For simple linear regression,
ˆ1 = SXY ,
SXX
ˆ0 = ȳ ˆ
⇣1 x̄, ⌘
SXY 2
ˆ 2 = n 1 2 SY Y SXX , by the property of RSS/ 2 ⇠ 2
n 2.

1
 STAT 3008 Applied Regression Analysis
Tutorial 1: Estimates

September 25, 2023

1 Abbreviation
For simple linear regression, yi = 0 + 1 xi + "i , i = 1, 2, ..., n.
n n
1X 1X
ȳ = yi , x̄ = xi ,
n n
i=1 i=1
n
X Xn
SY Y = (yi ȳ)2 = yi2 nȳ 2 ,
i=1 i=1
n
X Xn
SXX = (xi x̄)2 = x2i nx̄2 ,
i=1 i=1
n
X n
X
SXY = (xi x̄) (yi ȳ) = x i yi nx̄ȳ,
i=1 i=1
n
X n
X
ˆ0 ˆ1 xi )2 = SY Y SXY 2 ˆ2 SXX.
RSS = "ˆ2i = (yi = SY Y 1
SXX
i=1 i=1

2 Least Squares Estimates (LSE)

“Least square” is a criterion, an idea to find the “best fit”. It doesn’t matter what the
distribution of yi is and what the specific model is. Minimize RSS (Residual Sum of Squares)
to get estimates of parameters.
Write down RSS, take partial derivatives to each parameter i, let them equal to 0 and then
get estimates.
For simple linear regression,
ˆ1 = SXY ,
SXX
ˆ0 = ȳ ˆ
⇣1 x̄, ⌘
SXY 2
ˆ 2 = n 1 2 SY Y SXX , by the property of RSS/ 2 ⇠ 2
n 2.

1
 STAT 3008 Applied Regression Analysis
Tutorial 1: Estimates

September 25, 2023

1 Abbreviation
For simple linear regression, yi = 0 + 1 xi + "i , i = 1, 2, ..., n.
n n
1X 1X
ȳ = yi , x̄ = xi ,
n n
i=1 i=1
n
X Xn
SY Y = (yi ȳ)2 = yi2 nȳ 2 ,
i=1 i=1
n
X Xn
SXX = (xi x̄)2 = x2i nx̄2 ,
i=1 i=1
n
X n
X
SXY = (xi x̄) (yi ȳ) = x i yi nx̄ȳ,
i=1 i=1
n
X n
X
ˆ0 ˆ1 xi )2 = SY Y SXY 2 ˆ2 SXX.
RSS = "ˆ2i = (yi = SY Y 1
SXX
i=1 i=1

2 Least Squares Estimates (LSE)

“Least square” is a criterion, an idea to find the “best fit”. It doesn’t matter what the
distribution of yi is and what the specific model is. Minimize RSS (Residual Sum of Squares)
to get estimates of parameters.
Write down RSS, take partial derivatives to each parameter i, let them equal to 0 and then
get estimates.
For simple linear regression,
ˆ1 = SXY ,
SXX
ˆ0 = ȳ ˆ
⇣1 x̄, ⌘
SXY 2
ˆ 2 = n 1 2 SY Y SXX , by the property of RSS/ 2 ⇠ 2
n 2.

1
3 Maximum Likelihood Estimates (MLE)
Maximize the (log-)likelihood function to get estimates of parameters. The log-likelihood
function is given as
l(✓; x, y) = log(L(✓; x, y)) = log(f (x, y; ✓)).
Q
n
If i.i.d. sample, then f (x, y; ✓) = f (xi , yi ; ✓).
i=1
For simple linear regression, we have
n
2 2 n n 2 1 X 2
l 0, 1, = log L 0, 1, = log(2⇡) log 2
(yi 0 1 xi ) .
2 2 2
i=1

@l @l @l
Set @ 0 ˜ , ˜ ,˜ 2 = @ 1 ˜ , ˜ ,˜ 2 = @ 2 ˜0 , ˜1 ,˜ 2
= 0, then we can get the estimates as
0 1 0 1
˜1 = SXY = ˆ1 ,
SXX
˜0 = ȳ ˜1 x̄ = ˆ0 ,
⇣ ⌘
SXY 2
˜ 2 = n1 SY Y SXX = n 2 2
n ˆ .

4 Coefficient Table
iid 2
For the simple linear regression, yi = 0 + 1 xi + "i , "i ⇠ N 0, .
The hypothesis testing is conducted on hypotheses including H0 : = ⇤ and H1 : 6= ⇤
1 1 1 1.
ˆ1 ⇤
Under H0 , t statistics is given as t0 = 1
⇠ tn 2.
se( ˆ1 )
It is a two-sided test, then p-value = 2 Pr(tn 2 > |t0 |).

Figure 1: Illustration for two-sided tests.

If we set ⇤ = ⇤ = 0 and then derive the t statistics t0 for the following coefficient table.
0 1

Variable Coefficient Std. Error t-statistic p-value

ˆ0
Constant ˆ0 se( ˆ0 ) t0 = 2 Pr (tn 2 > |t0 |)
se( ˆ0 )
ˆ1
X ˆ1 se( ˆ1 ) t0 = 2 Pr (tn 2 > |t0 |)
se( ˆ1 )

2
5 Properties of the LSE
For the simple linear regression:

iid 2
yi = 0 + 1 xi + "i , "i ⇠ N 0, .

Key properties of LSE:

• Property 1: Both ˆ0 and ˆ1 are linear combinations of yi .

• Property 2: The fitted line ŷ = ˆ0 + ˆ1 x always passed through the average point (x̄, ȳ).

• Property 3: ˆ0 , ˆ1 and ˆ 2 are unbiased estimates.

P
n
• Property 4: Sum of residuals is equal to 0. That is, "ˆi = 0.
i=1

Extension: Suppose the following regression model is fitted to a data set with observations
{(xi , yi ), i = 1, 2, ..., n}:
iid 2
yi = xi + "i , "i ⇠ N 0, . (5.1)

Compare it with the simple linear regression.

(i) If we assume that 0 = 0, the simple linear regression model s reduced to (5.1) and some
of properties would fail.
P P⇣ ⌘ P⇣ ⌘
Hints: In (5.1), no equation to ensure (yi ŷi ) = yi ˆxi = 0. When yi ˆxi =6
P P
0, yi 6= ˆ xi , i.e., ȳ 6= ˆx̄, the fitted line does NOT pass through (x̄, ȳ).
(ii) Here RSS / 2 ⇠ 2 with df = n 1, since df = n from the data and df = 1 from
n 1
estimating . For the simple linear regression, we have the conclusion that RSS / 2 ⇠ 2
n 2
with df = n 2 since df = 2 from estimating 0 and 1.

6 Exercise
Problem 1. Suppose the following regression model is fitted to a data set with observations
{(xi , yi ), i = 1, 2, ..., n}:
iid 2
yi = x i + "i , "i ⇠ N 0, .

(a) Least squares estimates (LSE) for and 2.

(b) Prove that ˆ is unbiasd, i.e., E( ˆ) = .

(c) Show that the fitted regression line (i) passes through the point
!
⇣ ⌘ n
1X 2 1X
n
x2 , xy = xi , x i yi ,
n n
i=1 i=1

but (ii) does not pass through the average point (x̄, ȳ).
(d) Maximum likelihood estimates (MLE) for and 2.

3
 Problem 2. Simple linear regression model with x1 = a and x2 = ... = xn = a + nb.
(a) Show that SXX = n(n 1)b2 .
(b) OLS estimate for 1. ⇣ ⌘
ȳ y1
(c) Consider if the regression line passes through the point (x, y) = a + nb, ȳ + n 1 .

Figure 2: Interpretation for Problem 2.

Problem 3. Show that {ˆ

"i , i = 1, 2, ..., n} are uncorrelated with the explanatory variables
1 P
n
{xi , i = 1, 2, ..., n}. It is equivalent to prove that Ĉov(x, "ˆ) = n 1 (xi x̄) "ˆi "ˆ = 0,
i=1
⇢ˆ(x, "ˆ) = 0.
Hints: First try to prove that "ˆ = 0, and then simplify the estimated covariance by replacing
the residuals with the expression "ˆi = yi ˆ0 ˆ1 xi . Next, replace the OLS estimates ˆ1 and
ˆ0 with the exact solutions in terms of data {(xi , yi ), i = 1, 2, ..., n}.
Below, we show geometrically how explanatory variables and residuals are orthogonal.

Figure 3: Interpretation for Problem 3.

4
a) RSS = (y= Bx )
-
=

RSS2(i) lyil
Ci (y -)=
= 0

=
-Xin -1)

E(() = n -
1

2= El

inyz- )
--
E(5k) :

i
 STAT 3008 Applied Regression Analysis
Tutorial 1 Solution

Problem 1.
(a)
n
X
g( ) = (yi xi ) 2
i=1
n
X
dg
= 2 xi (yi xi )
d
i=1
Pn
dg ˆ x i yi
= 0 ) = Pi=1 n 2
d i=1 xi
RSS
Since 2
⇠ x2n 1 ,
✓ ◆ ✓ ◆
RSS 2 RSS
E 2
=n 1) =E
n 1
n
" Pn #2
RSS 1 X j=1 x j y j
ˆ2 = = yi x i P n 2
n 1 n 1 j=1 xj
i=1

(b) Pn Pn
x i E (y i ) x ( x)
E( ˆ | x) = i=1
Pn 2
Pn i 2 i =
= i=1
i=1 xi i=1 xi
(c)
Pn X n ⇣ ⌘
ˆx¯2 = Pi=1 xi yi x̄2 = 1 x y
i i ) pass through x 2 , xy
n 2
i=1 xi n
i=1
Pn
ˆx̄ = Pi=1 xi yi (x̄) 6= ȳ. ) do not pass through (x̄, ȳ)
n 2
i=1 xi
.
(d)
@l , 2 @l , 2
2
l , ) = 2
=0
@ @
( ˜,˜ 2 ) ( ˜,˜ 2 )
Pn n Pn !2
˜ x i yi RSS 1X j=1 xj yj
) = Pi=1
n 2 , ˜2 = = yi xi Pn 2
i=1 xi n n j=1 xj
i=1

Problem 2. Pn
(a) x̄ = a + (n 1)b ) SXX = i=1 (xi x̄)2 = n(n 1)b2

1
(b)
n
X
SXY = (xi x̄) (yi ȳ)
i=1
n
X
= (a x̄) (y1 ȳ) + (a + nb x̄) (yi ȳ)
i=2
n
X
= (n 1)b (y1 ȳ) + b (yi ȳ)
i=2
n
X
= nb (ȳ y1 ) (* yi = nȳ y1 )
i=2

) ˆ1 = SXX
SXY
= (nȳ y1)b
1

(c) ˆ0 + ˆ1 (a + nb) = ȳ ˆ1 x̄ + ˆ1 (a + nb) = ȳ ˆ1 (x̄ a nb) = ȳ + ȳ y1

n 1
) pass through

Problem 3.

1 X⇣ ⌘ ⇣ ⌘
n
ê¯ = yi ˆ0 ˆ1 xi = ȳ ȳ ˆ1 x̄ ˆ1 x̄ = 0
n
i=1
n
X n
X
(xi x̄) êi ê¯ = (xi x̄) êi
i=1 i=1
Xn ⇣ ⌘
= (xi x̄) yi ˆ0 ˆ1 xi
i=1
n
X ⇣ ⇣ ⌘ ⌘
= (xi x̄) yi ȳ ˆ1 x̄ ˆ1 xi
i=1
n
X n
X
= (xi x̄) (yi ȳ) ˆ1 (xi x̄)2
i=1 i=1
SXY
= SXY SXX = 0
SXX