Linear Regression

Mehul Motani
Electrical & Computer Engineering
National University of Singapore
Email: [email protected]

© Mehul Motani Linear Regression 1

B. Sikdar
11/17/10
Machine Learning Taxonomy
• Machine learning is function approximation
• Supervised learning – Access to labeled dataset
• Unsupervised learning – Dataset is not labeled
• Classification – The output is categorical
• Regression – The output is continuous
• Many different ML models are available
• Simplest model is Linear Regression
– Find the best fit line which goes through the data points
– Linear regression is supervised learning
© Mehul Motani Linear Regression 2
 Example – Tire tread vs Mileage
3
How are Tire tread wear
and Mileage related?

Clearly, tire tread wear

increases with mileage!
But how?

© Mehul Motani Linear Regression 3

Example – Tire tread vs Mileage

4 Mileage Groove Depth
(in 1000 miles) (in mils)
0 394.33
4 329.50
8 291.00
12 255.17
16 229.33
20 204.83
24 179.00
28 163.83
32 150.33
Tire tread wear vs. mileage. From: Statistics and Data
Analysis; Tamhane and Dunlop; Prentice Hall.
© Mehul Motani Linear Regression 4
A. Seyedi
11/17/10
Some Training Data

!
dependent
variable
(output)

x – independent variable (input)

© Mehul Motani Linear Regression 5

A. Seyedi
11/17/10
Linear Regression - Best Fit Line
• Linear Regression - Fit data with the “best”
hyperplane which goes through the data points

Best fit line

! = #! + #" %
!
dependent
variable ! = #$ + &
(output)
# is the slope of the line
& is the y-intercept
" – independent variable (input)

© Mehul Motani Linear Regression 6

A. Seyedi
11/17/10
Linear Regression - Best Fit Line
• For each point the differences between the predicted point
and the actual observation is the residue or error
• The “best” fit line is the one which minimizes the overall
error, which is defined as the sum of squared errors.

Best fit line

!
! = #! + #" %
dependent
variable
(output)

x – independent variable (input)

© Mehul Motani Linear Regression 7

Polynomial Curve Fitting

Degree M
polynomial

© Mehul Motani Linear Regression 8

 0th Order Polynomial (Degree 0)
9

© Mehul Motani Linear Regression 9

1st Order Polynomial (Degree 1)

10

© Mehul Motani Linear Regression 10

 3rd Order Polynomial (Degree 3)
11

© Mehul Motani Linear Regression 11

9th Order Polynomial (Degree 9)

12

© Mehul Motani Linear Regression 12

 How good is the fit?

Bad Fit

Good Fit
Over-Fitting

Root-Mean-Square(RMS) Error

© Mehul Motani Linear Regression 13

Linear Regression
14
• Response/outcome/dependent variable: !
• Predictor/explanatory/independent variable: "
• Example 1: Estimate electricity demand for home cooling
(!) from the average daily temperature (")
• Example 2: Relationship between the head size and body
size of a newborn
• Regression analysis: statistical methodology to estimate
the relationship between " and !
• Correlation analysis: statistical methodology used to
asses the strength of relationship between " and !

© Mehul Motani Linear Regression 14

 Linear Regression
15 • One response variable and one explanatory variable
• We denote the explanatory variable as # and response
variable as $
• % pairs of observations !! ; "! , ' = 1, ⋯ , %
• !! is the observed values of the random variable $! and is
related to "! by:
$! = ," + ,# "! + .!
• .! : random error with / .! = 0 and 123 .! = 4 $
• The “true regression line” models the true but unknown
mean of $!
/ $! = !5! = ," + ,# "!

© Mehul Motani Linear Regression 15

Linear Regression
16
• The error .! :
• Independent and identically distributed
• Variety of causes:
• Measurement errors
• Other variables affecting $! not included in the
model
• Assumption / .! = 0: implies there is no systematic
bias
• Usual model for .! : .! ~7 0, 4 $
• Justified by the Central Limit Theorem

© Mehul Motani Linear Regression 16

 Example – Tire tread vs Mileage
17
Inspect the data – Does the data have a linear trend?
Mileage Groove Depth
(in 1000 miles) (in mils)
# $
0 394.33
4 329.50
8 291.00
12 255.17
16 229.33
20 204.83
24 179.00
28 163.83
32 150.33
© Mehul Motani Linear Regression 17

Simple Linear Regression Model

18

• In simple linear regression, the data is represented as:

!! = ," + ,# "! + .! (1)
where .! ~7 0, 4 $
• The candidate / fitted model:
!5 = ," + ,# " (2)
where
• ," : intercept
• ,# : slope of regression line

© Mehul Motani Linear Regression 18

 Least Squares Fitting
19
• 8! : difference between
data and candidate line
Candidate Line 8! = !! − !5!
= !! − ," + ,# "!
ei
• Goal: Find the ," and
,# which minimize the
sum of squared errors:
&

: = ; !! − ," + ,# "! $

!%#

© Mehul Motani Linear Regression 19

Least Squares Fitting

20 • Obtain the values of ," and ,# that minimizes the sum of
squared errors
% % %
%&
Partial = −2 + $" − '! + '$ #" = 0 ⇒ /'! + '$ + #" = + $"
%'!
derivative "#$ "#$ "#$

% % % %
%&
= −2 + #" $" − '! + '$ #" = 0 ⇒ '! + #" + '$ + #"& = '$ + #" $"
%'$
"#$ "#$ "#$ "#$

∑%"#$ #"& ∑%"#$ $" − ∑%"#$ #" ∑%"#$ #" $"

'0! = &
After / ∑%"#$ #"& − ∑%"#$ #"
some
algebra / ∑%"#$ #" $" − ∑%"#$ #" ∑%"#$ $"
'0$ = &
/ ∑%"#$ #"& − ∑%"#$ #"
© Mehul Motani Linear Regression 20
 Least Squares Fitting
21
• To simplify:
& & & &
1
'!" = ( $# − $̅ !# − !+ = ( $# !# − ( $# ( !#
-
#$% #$% #$% #$%
& & & '
1
'!! = ( $# − $̅ '
= ( $#' − ( $#
-
#$% #$% #$%
& & & '
' ' 1
'"" = ( !# − !+ = ( !# − ( !#
-
#$% #$% #$%
Note:
1
$
1
$ 0/( = !+ − 0% $̅
"̅ = ' "! and !+ = ' !! ⇒ )'(
&
!"#
&
!"# 0/% = )''

© Mehul Motani Linear Regression 21

Example – Tire tread vs Mileage

22
/=9
Mileage Groove Depth
∑ #" = 144, ∑ #"& = 3264
(in 1000 miles) (in mils)
∑ $" = 2197.32, ∑ $"& = 589887.08 # $
∑ #" $" = 28167.72 0 394.33
#̅ = 16, $> = 244.15 4 329.50
?)* = −6989.40
8 291.00
?)) = 960
?)* 12 255.17
'0$ = = −7.281
?)) 16 229.33
'0! = $> − '$ #̅ = 360.64 20 204.83
@ = BCD. CE − F. GHIJ (I)
A 24 179.00
Exercise: Write code in python to compute the 28 163.83
best fit line for the tire tread example and plot
32 150.33
the best fit line overlaid with the data points.
© Mehul Motani Linear Regression 22
 Linear Regression – Tire tread vs Mileage
23 Mileage Groove Depth
(in 1000 miles) (in mils)
# $
0 394.33
4 329.50
8 291.00
12 255.17
16 229.33
20 204.83
24 179.00
28 163.83
32 150.33
Tire tread wear vs. mileage. From: Statistics and Data
Analysis; Tamhane and Dunlop; Prentice Hall.
© Mehul Motani Linear Regression 23

A. Seyedi
11/17/10
Anscombe's Quartet
A B

C D

The best fit line can be misleading!

© Mehul Motani Linear Regression 24
25

Linear regression is easy to explain!

• One nice advantage of linear regression models (and linear
classification) is the potential to look at the coefficients to give
insight into which input variables are most important in
predicting the output
• The variables with the largest magnitude have the highest
correlation with the output
– A large positive coefficient implies that the output will increase
when this input is increased (positively correlated)
– A large negative coefficient implies that the output will decrease
when this input is increased (negatively correlated)
– A small or 0 coefficient suggests that the input is uncorrelated
with the output (at least at the 1st order)
• Linear regression can be used to find best "indicators"
• However, be careful not to confuse correlation with causality

© Mehul Motani Linear Regression 25

Multivariate Regression
26
• We have explored problems with one response variable ! and one
explanatory variable "
• Sometimes a straight line (linear regression) is not adequate and
quadratic or cubic model is needed (polynomial regression)
• Sometimes there are more than one predictor variables and their
simultaneous effect needs to be modeled
• # pairs of observations !! ; "!", "!#, ⋯ , "!$ , ' = 1, ⋯ , #
• Multiple regression model: * = +% + +& -& + +' -' + ⋯ + +( -( + .
• Polynomial regression: Linear in / and not necessarily "’s: "" =
", "# = " #, "$ = " $
• Simple linear regression: !; "
• Multiple linear regression: !; "", "#, ⋯ , ")
• Multivariate regression: !", !#, ⋯ , !* ; "", "#, ⋯ , ")
© Mehul Motani Linear Regression 26
 Multiple Linear Regression
27
• Least squares fit: & = ∑%"#$ $" − '! + '$ #"$ + '& #"& + ⋯ + '+ #"+ &

• Taking the partial derivatives and equating to zero:

%&
= −2 + $" − '! + '$ #"$ + '& #"& + ⋯ + '+ #"+ = 0 for N = 1,2, ⋯ , O
%'!
%&
= −2 + $" − '! + '$ #"$ + '& #"& + ⋯ + '+ #"+ #", = 0 for N = 1,2, ⋯ , O
%',

• After simplification (for 1 = 1,2, ⋯ , 5):

/'! + '$ + #"$ + ⋯ + '+ + #"+ = + $"

/'! + #", + '$ + #"$ #", + ⋯ + '+ + #"+ #", = + $" #",

• These have to be solved simultaneously for 0% , 0' , ⋯ , 0*

© Mehul Motani Linear Regression 27

Multiple Linear Regression

28
• Matrix form: • Regression model:
Q$ T$ 6 = 78 + 9
Q T&
P= & , S= ⋮
⋮
Q% T% • Simultaneous linear
1 #$$ #$& ⋯ #$+ equations whose
1 #&$ #&& ⋯ #&+ solution gives the least
U=
⋮ ⋮ ⋮ ⋱ ⋮ square estimates:
1 #%$ #%& ⋯ #%+
7+ 78 = 7+ 6
'! '0!
'$ '0$ • Solution is given by the
W = '& , X
W = '0&
⋮ pseudoinverse:
⋮
'+ 0 : = 7+ 7 ,- 7+ 6
8
'+
© Mehul Motani Linear Regression 28