MAF3821 Mathematical Methods

MAF3821 Mathematical Methods

Lecture notes

Dr. Rodrigue G Batogna, Mathematics.

Dept. of Computing Mathematical & Statistical Sciences
F.A.E.N.S
University of Namibia

1/35
MAF3821 Mathematical Methods

Overview

1 LINEAR REGRESSION
General Definitions
Simple Linear Regression
Standard model
Assessing the fit
Correlation and Regression

2 Matrix Approach to Linear Least Square

The General Multiple Regression Model
Existence Condition Solution of Normal Equations

2/35
MAF3821 Mathematical Methods

I.1.General Definitions
linear regression is concerned with describing the relationship
between a dependent variable and one or more independent or
explanatory variable. We will speak of
- Univariate regression if we have a single dependent variable
and also one independent or explanatory variable.
- Simple Linear Regression is the least squares regression with a
single explanatory variable. Example: fitting a straight line
through a set of points, in a way that minimizes the sum of
squared residuals. Some authors refer to simple linear
regression as bivariate regression to indicate that two variables
are dealt with a single independent variable ( predictor,
explanatory variable, covariate, risk factor, cofounder)and a
dependent variable.
- We speak of multiple linear regression for a single dependent
variable, and two or more explanatory variables. 3/35
MAF3821 Mathematical Methods

I.1.General Definitions

- Multivariate linear regression is the modeling of multiple

correlated dependent variables using the least squares linear
predictor function, rather than a single scalar dependent
variable.
In linear regression the predictor function is linear. For a
single response variable Y and a single explanatory variable
X the simple linear regression model is:

Y = βo + β1 X + ε

β0 is the intercept
β1 is the slope
ε denote random errors

4/35
MAF3821 Mathematical Methods

Simple Linear Regression

Simple Linear Regression (Ordinary Least Squares;

Linear Least Squares)
Introduction
In curve-fitting the least squares is the most common method. It
aims to minimize the sum of squared deviations of the predictor or
fitted values from actual observations. Suppose a straight line is to
be fitted to the points (yi , xi ) i = 1, . . . , n.
y is the dependent variable, response variable,
x is the independent variable, explanatory variable, regressor, risk
factor, predictor variable, exposure variable.
The least squares method chooses the slope and intercept of the
line y = β0 + β1 x to minimize
X n
S(β0 , β1 ) = (yi − β0 − β1 xi )2
i=1
To find β0 and β1 we differentiate S(β0 , β1 )
5/35
MAF3821 Mathematical Methods

Simple Linear Regression

n
∂S X
= −2 (yi − β0 − β1 xi )
∂β0
i=1
n
∂S X
= −2 xi (yi − β0 − β1 xi )
∂β1
i=1
n
∂2S X
= −2 (−1) = 2n > 0
∂ 2 β0
i=1
n n
∂2S X X
= −2 xi (−xi ) = 2 xi2 > 0
∂ 2 β1
i=1 i=1

From the second derivative test for extreme values we will have
∂S ∂S
minima at ∂β 0
= 0 and ∂β 1
= 0.
We solve to find the minimizers βb0 and βb1 which are the values that
minimize the sum of squared deviations S(β0 , β1 )
6/35
MAF3821 Mathematical Methods

Simple Linear Regression

n
∂S X 
= 0 ⇔−2 yi − βb0 − βb1 xi = 0
∂β0
i=1
n 
X 
⇔ yi − β0 − β1 xi = 0
b b
i=1
n
X n 
X 
⇔ yi = βb0 + βb1 xi
i=1 i=1
n
X n
X
⇔ yi = nβb0 + βb1 xi
i=1 i=1

Likewise

7/35
MAF3821 Mathematical Methods

Simple Linear Regression

n
∂S X  
= 0 ⇔−2 xi yi − βb0 − βb1 xi = 0
∂β1
i=1
n 
X 
⇔ xi yi − βb0 xi − βb1 xi2 = 0
i=1
Xn n
X n
X
⇔ xi yi = βb0 xi + βb1 xi2
i=1 i=1 i=1

Putting the 2 equations together we have the simultaneous

equations:
(P
n b Pn xi
i=1 yi = nβ0 + β1
b
i=1
Pn Pn b Pn x 2
x y
i=1 i i = β
b 0 i=1 i + β1
x i=1 i

8/35
MAF3821 Mathematical Methods

Simple Linear Regression

We obtain after solving

Pn 2 ( n y ) − ( n x )( n x y )

P P P
i=1 x i i=1 i i=1 i i=1 i i
βb0 = Pn Pn 2
2
n i=1 xi − ( i=1 xi )
n ( i=1 xi yi ) − ( ni=1 xi ) ( ni=1 yi )
Pn P P
β1 =
b
n ni=1 xi2 − ( ni=1 xi )2
P P

which is equivalent to

βb0 = ȳ − βb1 x̄
Pn
i=1 (xi − x̄) (yi − ȳ )
β1 =
b Pn 2
i=1 (xi − x̄)

Prove the equivalence between the formulas!

9/35
MAF3821 Mathematical Methods

Standard model

Standard model

The simplest model ( standard statistical model )

yi = β0 + β1 xi + ei , i = 1, . . . , n
Assumptions


ei ∼ are normally distributed

E (ei ) = 0 for all i



Var (ei ) = σ 2 for all i ( homoscedasticity ) ≡ Var (yi ) = σ 2 for all i

cov (ei , ej ) = 0 for all i ̸= j ≡ cov (yi , yj ) = 0 for all i ̸= j




xi are assumed to be fixed is non stochastic constant


10/35
MAF3821 Mathematical Methods

Standard model

Theorem
Under the assumptions of the standard statistical model, the least
squares estimates are unbiased:

E (βbj ) = βj for j = 0, 1.

From the assumptions,

E (yi ) = β0 + β1 xi

thus from the equation for βb0

Pn 2 ( n E (y )) − ( n x ) ( n x E (y ))

P P P
i=1 x i i=1 i i=1 i i=1 i i
E (βb0 )= Pn Pn 2
2
n i=1 xi − ( i=1 xi )
Pn 2
Pn Pn Pn Pn 2


i=1 xi (nβ0 + β1 i=1 xi )− i=1 xi β0 i=1 xi + β1 i=1 xi
=
n ni=1 xi2 −( ni=1 xi )2
P P

11/35
MAF3821 Mathematical Methods

Standard model

nβ0 ni=1xi2 +β1( ni=1xi ) ni=1xi2 −β0( ni=1xi )2−β1( ni=1xi ) ni=1 xi2
P P P
P P P

=
n ni=1 xi2 − ( ni=1 xi )2
P P
 P 
β0 n ni=1 xi2 −( ni=1 xi )2
P
=
n ni=1 xi2 −( ni=1 xi )2
P P

= β0

The proof for βb1 is similar.

12/35
MAF3821 Mathematical Methods

Standard model

Theorem
Under the assumptions of the model
Pn
σ2 2
i=1 xi
Var (βb0 ) = Pn n 2
2
P
n i=1 xi −( i=1 xi )
nσ 2
Var (βb1 ) = Pn Pn 2
2
n i=1 xi −( i=1 xi )
−σ 2 ni=1 xi
P
cov (βb0 , βb1 ) =
n ni=1 xi2 −( ni=1 xi )2
P P

Proof
Pn Pn
i=1 (xi − x̄)(yi − ȳ ) (xi − x̄)yi
βb1 = Pn 2
= Pi=1
n 2
i=1 (xi − x̄) i=1 (xi − x̄)
σ2
we then have Var (βb1 ) = Pn 2
i=1 (xi − x̄)
13/35
MAF3821 Mathematical Methods

Assessing the fit

Assessing the fit

We use extensively the residuals which are the difference between

observed and fitted value

ebi = yi − βb0 − βb1 xi

The standard measure of closeness of the data points to the fitted

regression line is the R-squared, also called coefficient of
determination.
This number is between 0 and 1, which indicates the proportion of
variability in y about the sample mean ȳ , explained by the
regression on x
. If R 2 = 1 then all the variability in y is explained and the fit is
perfect.
If R 2 = 0 there is no explanation at all.

14/35
MAF3821 Mathematical Methods

Assessing the fit

Assessing the fit

For the variability in y before the regression we use the total sum
of squares
Xn
TSS = (yi − ȳ )2
i=1
For the variability in y after the regression we use the error sum of
squares
X n
ESS = (yi − yb)2
i=1
The R-squared is defined as :
Pn
2 ESS (yi − yb)2
R =1− = 1 − Pi=1
n 2
TSS i=1 (yi − ȳ )
n
(ybi − ȳ )2
P
Regression SS
= = Pi=1
n 2
.
TSS i=1 (yi − ȳ )
15/35
MAF3821 Mathematical Methods

Correlation and Regression

A standard measure of association between x and y , even before

considering the regression is the correlation coefficient. Let
n n
1X 1X
Sxx = (xi − x̄)2 ; Syy = (yi − ȳ )2
n n
i=1 i=1

and the sample covariance between x and y is

n
1X
Sxy = (xi − x̄)(yi − ȳ );
n
i=1

16/35
MAF3821 Mathematical Methods

Correlation and Regression

The correlation coefficient between xs′ and ys′ is:

Pn
Sxy (xi − x̄)(yi − ȳ )
r=p = pPn i=1 Pn
2 2
Sxx Syy i=1 (xi − x̄) i=1 (yi − ȳ )

The slope of the least square is

s
Sxy Sxx
βb1 = and therefore r = βb1 .
Sxx Syy

In particular the correlation is zero if and only if (iff) the slope is

zero. In the regression equation yb = βb0 + βb1 x the coefficients are
expressed as:

17/35
MAF3821 Mathematical Methods

Correlation and Regression

βb0 = ȳ − βb1 x̄
Pn
i=1 (xi − x̄) (yi − ȳ )
β1 =
b Pn 2
.
i=1 (xi − x̄)

Introducing βb1 in terms of r , and after some manipulations we

obtain
yb − ȳ x − x̄
p =r√
Syy Sxx

We can see that r and βb1 will always be of the same sign. A value
of r near or equal to 0 implies little or no relationship between y
and x. We have that −1 ≤ r ≤ 1. rxy = 1 means a perfect
positive correlation. 0 < rxy < 1 positive correlation; rxy = 0 no
correlation; −1 ≤ rxy ≤ 0 negative correlation

It can be shown that R 2 = r 2 . 18/35

MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

Matrix Approach to Linear Least Square/ Multiple

Regression

With problems more complex than fitting a straight line, it is very

useful to approach the linear least squares analysis via linear
algebra. Suppose that a model of the form

y = β0 + β1 x1 + . . . + βp−1 xp−1 + ε

is to be fitted to data, which we denote as

yi , xi1 , xi2 , . . . , xip−1 , i = 1, . . . , n. The observations yi where
i = 1, . . . , n will be represented by a vector Y . The unknowns
β0 , . . . βp−1 will be represented by a vector β. The errors
εi (i = 1, 2, 3, . . . , n) by the vector ε, and the independent variables
by the matrix X .

19/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

In matrix notation we can write the model as Y = X β + ε where
Y (n×1) ; β (p×1) ; ε(n×1)
     
y1 β0 ε1
 y2   β1   ε2 
Y =  . ; β =  . ; and ε =  . 
     
 ..   ..   .. 
yn βp−1 εn

The matrix X(n×p) is:

 
1 x11 x12 . . . x1p−1
1 x21 x22 . . . x2p−1 
X = .
 
.. .. .. 
 .. . . ... . 
1 xn1 xn2 . . . xnp−1
20/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

The vector of fitted values or predicted values Yb , can be written as

Yb = X βb where
   b 
yb1 β0
yb2   βb1 
Yb =  .  ; and βb =  . 
   
 ..  .
 . 
ybn βbp-1

Example
Let us consider fitting the straight line model y = β0 + β1 x + ε to
points (yi , xi ) where i = 1, 2, . . . , n in this case

21/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

Example
     
y1 1 x1 ε1
y2    1 x2   ε2 
β
Y =  .  ; β = 0 ; X =  . .  and ε =  .  then
     
 ..  β1  .. ..   .. 
yn 1 xn εn
      
y1 1 x1 y1 − β0 − β1 x1
y2  1 x2     y2 − β0 − β2 x2 
β  
Y − X β =  .  −  . .  × 0  =  .
    
. . . β1   ..
 .   . .  . 
yn 1 xn yn − β0 − βn xn

22/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

Example
In general for the model

Y (n×1) = X(n×p) β (p×1) + ε(n×1) .

The least squares problem is to find β, using say n × p

observations to minimize
n
X
(yi − β0 − β1 xi1 − β2 xi2 − . . . − βp−1 xip−1 )2


S β =
i=1
= ||Y − X β||2 = ||Y − Yb ||2
2
!
X
If u is a vector , ||u||2 = ui2
i=1
23/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

Theorem
Suppose the linear model

y = β0 + β1 x1 + β2 x2 + . . . + βp−1 xp−1 + ε

is to be fitted to data which we denote as

yi , xi1 , xi2 , xi3 , . . . , xip−1 (i = 1, 2, . . . , n), by using the method of
least squares.
The normal least squares equations can be written as

X ′ X βb = X ′ Y .

24/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

Proof.
The sum of squares of errors for n × p pairs of observations is
n
X n
X
ε2i (yi − β0 − β1 xi1 − β2 xi2 − . . . − βp−1 xip−1 )2


S β = =
i=1 i=1


The values βb0 , βb1 , . . . , βbp-1 that minimize S β are obtained by
setting the p partial derivatives

∂S(β) ∂S(β) ∂S(β)

, ,..., equal to zero.
∂β0 ∂β1 ∂βp−1

25/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

Proof.

n
∂S(β) X
=2 (yi − β0 − β1 xi1 − β2 xi2 − . . . − βp−1 xip−1 )(−1)=0
∂β0
i=1
n
∂S(β) X
=2 (yi − β0 − β1 xi1 − β2 xi2 − . . . − βp−1 xip−1 )(−xi1 )=0
∂β1
i=1
n
∂S(β) X
=2 (yi − β0 − β1 xi1 − β2 xi2 − . . . − βp−1 xip−1 )(−xi2 )=0
∂β2
i=1
..
.
n
∂S(β) X
=2 (yi − β0 − β1 xi1 − β2 xi2 − . . . − βp−1 xip−1 )(−xip-1 )=0
∂βp-1
i=1

26/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

Proof.
In general if we differentiate S with respect to each
βk , (0 ≤ k ≤ p − 1) we see that the minimizers βb0 , βb1 , . . . , βbp-1
satisfy the p linear equations
n
X n
X Xn n
X
nβb0 + βb1 xi1 + βb2 xi2 + . . . + βbp-1 xip-1 = yi
i=1 i=1 i=1 i=1
n
X n
X n
X Xn n
X
2
βb0 xi1 + βb1 xi1 + βb2 xi1 xi2 + . . . + βp-1 xi1 xip-1= xi1 yi
b
i=1 i=1 i=1 i=1 i=1
n
X Xn n
X Xn n
X
2
βb0 xi2 + β1 xi1 xi2 + β2
b b xi2 + . . . + βp-1 xi2 xip-1= xi2 y2
b
i=1 i=1 i=1 i=1 i=1
..
.
27/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

Proof.

..
.
n
X n
X Xn n
X n
X
2
βb0 xip-1 + βb1 xi1 xip-1 + β2 xi2 xip-1+. . .+βp-1 xip-1= xip-1 yi
b b
i=1 i=1 i=1 i=1 i=1
which could be also written as
n
X Xn X n n
X n
X
β0
b xik + β1 xi1 xik + β2 xi2 xik+. . .+βp-1 xip-1 xik = xik yi
b b b
i=1 i=1 i=1 i=1 i=1
k = 0, 1, 2, . . . , p-1; and in matrix form

X ′ X βb = X ′ Y .

28/35
 MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
The General Multiple Regression Model

General Multiple Regression Model

(X ′ X ) βb = X ′ Y .
Pn Pn  b 
. . . P ni=1xip-1
 P
β0
Pnn Pni=1xi1 n
i=1xi2
2 . . . Pni=1xi1 xip-1 
P
x x i=1xi1 xi2 β
 b 
i1  1 


 Pi=1
n Pn i=1 i1 P n 2 n

X X βb= i=1xi2
′
i=1xi1 xi2 i=1xi2 ... i=1xi2 xip-1 
  βb 
2 
.. .. .. ..  .

 .. 

Pn . Pn . Pn . ... P .
 

n 2
i=1xip-1 i=1xi1 xip-1 i=1xi2 xip-1 ... x
i=1 ip-1 β
bp-1
 Pn     b   
i=1 yi 1 1 1 ... 1 β0 y1
 Pn xi1 yi   x11 x21 x31 . . . xn1   β1  y2 
  b   
 Pi=1
 n xi2 yi  ′  x12
 
′ x x . . . xn2   y3 
 
X Y =  i=1 X = 22 32 b  βb2 Y =
β=

..   .. .. .. ..   ..  .
.   .   .. 
  
Pn .
   . . . ...

x
i=1 ip-1 i y x1p-1 x2p-1 x 3p-1 . . . xnp-1 βbp-1 yn

29/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
Existence Condition Solution of Normal Equations

Existence Condition Solution of Normal Equations

If (X ′ X ) is non-singular, the formal solution of the normal
equations is
b = X ′ X −1 X ′ Y .



β

Lemma
(X ′ X ) is non-singular, if and only if the rank of X is p.

Proof.
Suppose X ′ X is singular.
There exists a non zero vector u such that
X ′ Xu = 0
u ′ X ′ Xu = u ′ 0 = 0
(Xu)′ (Xu) = 0. So ( Xu=0 ) the columns of X are linearly
dependent and the rank of X is less than p.

30/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
Existence Condition Solution of Normal Equations

Existence Condition Solution of Normal Equations

Proof.
...
Suppose that the rank of X is less than p so that there exists a
non zero vector u, Xu = 0.
Then X ′ Xu = 0 and therefore X ′ X is singular.

31/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
Existence Condition Solution of Normal Equations

Existence Condition Solution of Normal Equations

Example
Returning to fitting
 a straight line to the points (yi , xi ), i = 1, 2, 3.
1 x1
Then X = 1 x2 
1 x3
If x1 = x2 = x3 ,the matrix is singular since the two columns are
proportional to each other. In this case we would be trying to fit a
line to a single point.
(Check that X ′ X is singular calculate it in this case ).
With n pair of observations: 
  1 x1  Pn 
1 1 . . . 1 1 x  n i=1 xi
2
X ′X =   .. ..  = P
 
. .  n Pn 2
x1 x2 . . . xn i=1 xi i=1 xi
1 xn
32/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
Existence Condition Solution of Normal Equations

Existence Condition Solution of Normal Equations

Example
 Pn 2
Pn 
i=1 xi − i=1 xi
(X ′ X )−1 = Pn
1 2

2−
( ni=1 xi )
P
n x
i=1 i
Pn
− i=1 xi n
 
y1  Pn 
y
. . . 1 y2   i=1 i 
  
1 1
X ′Y =  ..  = P Thus
x1 x2 . . . xn  .  n
x y
i=1 i i
yn
" #
b = β0 = X ′ X −1 X ′ Y
b


β
β1
b
 Pn 2 Pn   Pn 
1 i=1xi − i=1xi i=1 y i
= Pn Pn 2 −
P n Pn
i=1xi n i=1 xi yi
2
n i=1 xi −( i=1 xi )
33/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
Existence Condition Solution of Normal Equations

Existence Condition Solution of Normal Equations

Example
...

Pn 2
Pn Pn Pn 
i=1xi i=1 yi − i=1xi i=1 xi yi
1
= Pn  
n i=1 xi2 − ( ni=1 xi )2
P Pn Pn Pn
− i=1xi i=1 yi + n i=1 xi yi
 Pn 2 Pn Pn Pn 
x
i=1 i y
i=1 i − x
i=1 i i=1 i yi
x  
2
n i=1 xi2 −( ni=1 xi )  βb0
Pn P

=  =
  
P n P n P n
i=1 xi yi − i=1xi
 n
i=1 yi
 b
Pn Pn 2
β1
2
n i=1 xi −( i=1 xi )

34/35
MAF3821 Mathematical Methods
Matrix Approach to Linear Least Square
Existence Condition Solution of Normal Equations

Existence Condition Solution of Normal Equations

Example
...
Thus
Pn 2
Pn Pn
Pn
i=1xi i=1 yi −
i=1xi i=1 xi yi
βb0 = Pn 2 Pn 2
− ( i=1 xi )
n i=1 xi
Pn Pn Pn
n i=1 xi yi − i=1xi i=1 yi
βb1 =
n i=1 xi − ( i=1 xi )2
Pn 2 Pn

SSxy
=
SSxx

which agrees with earlier calculations.

35/35