CSE 375 Machine Learning and Pattern

Recognition

3. Multiple Linear Regression

1
 Contents

• Formulate a machine learning model as a multiple linear regression

model.
– Identify prediction vector and target for the problem.
• Write the regression model in matrix form. Write the feature matrix
• Compute the least-squares solution for the regression coefficients on
training data.
• Derive the least-squares formula from minimization of the RSS

• Manipulate 2D arrays in python (indexing, stacking, computing

shapes, …)
• Compute the LS solution using python linear algebra and machine
learning packages
Slides Credit: Sundeep Rangan
#2
Pre-Requisites for this Lecture

• You should now know how to:

– Install python and run python notebooks
– Describe simple linear models mathematically
– Derive equations for fitting simple linear models
– Perform basic manipulations and plotting of data in
python

#3
Outline

• Motivating Example: Understanding glucose levels in

diabetes patients
• Multiple variable linear models
• Least squares solutions
• Computing the solutions in python
• Special case: Simple linear regression
• Extensions

#4
Example: Blood Glucose Level

• Diabetes patients must monitor

glucose level
• What causes blood glucose levels to
rise and fall?
• Many factors
• We know mechanisms qualitatively
• But, quantitative models are difficult
to obtain
– Hard to derive from first principles
– Difficult to model physiological
process precisely
• Can machine learning help?

#5
Data from AIM 94 Experiment

• Data collected as series of events

– Eating
– Exercise
– Insulin dosage
• Target variable glucose level monitored

#6
Demo on GitHub

• Demo:
demo2_glucose.ipynb

#7
Loading the Data

• Sklearn package:
– Many methods for machine
learning
– Datasets
– Will use throughout this class
• Diabetes dataset is one example

#8
Finding a Mathematical Model

Attributes Target
𝑥𝑥1 : Age 𝑦𝑦 =Glucose level
𝑥𝑥2 : Sex
𝑥𝑥3 : BMI
𝑦𝑦 ≈ 𝑦𝑦� = 𝑓𝑓(𝑥𝑥1 , … , 𝑥𝑥10 )
𝑥𝑥4 : BP
𝑥𝑥5 : S1
⋮
𝑥𝑥10 : S6

• Goal: Find a function to predict glucose level from the 10 attributes

• Problem: Several attributes
– Need a multi-variable function

#9
 Matrix Representation of Data
• Data is a matrix and a target vector
• 𝑛𝑛 samples: Attributes Target vector
– One sample per row
• 𝑘𝑘 features / attributes /predictors:
– One feature per column 𝑥𝑥11 ⋯ 𝑥𝑥1𝑘𝑘 𝑦𝑦1
𝑋𝑋 = ⋮ ⋱ ⋮ 𝑦𝑦 = ⋮ Samples

𝑥𝑥𝑛𝑛𝑛 ⋯ 𝑥𝑥𝑛𝑛𝑘𝑘 𝑦𝑦𝑛𝑛

• This example:
– 𝑦𝑦𝑖𝑖 = blood glucose measurement of i-th
sample
– 𝑥𝑥𝑖𝑖,𝑗𝑗 : j-th feature of i-th sample
– 𝒙𝒙𝑇𝑇𝑖𝑖 = [𝑥𝑥𝑖𝑖,1 , 𝑥𝑥𝑖𝑖,2 ,…, 𝑥𝑥𝑖𝑖,𝑘𝑘 ]: feature or predictor
vector
– i-th sample contains 𝒙𝒙𝑖𝑖 ,𝑦𝑦𝑖𝑖

#10
Outline

• Motivating Example: Understanding glucose levels in

diabetes patients
• Multiple variable linear models
• Least squares solutions
• Computing the solutions in python
• Special case: Simple linear regression
• Extensions

#11
Multivariable Linear Model for Glucose

Attributes 𝑦𝑦 ≈ 𝑦𝑦� = 𝑓𝑓(𝑥𝑥1 , … , 𝑥𝑥10 )

Target
Age, Sex, BMI,BP,S1, …, S6
𝑦𝑦 =Glucose level
𝒙𝒙 = [𝑥𝑥1 , … , 𝑥𝑥10 ]

• Goal: Find a function to predict glucose level from the 10 attributes

• Linear Model: Assume glucose is a linear function of the predictors:

glucose ≈ prediction
= 𝛽𝛽0 + 𝛽𝛽1 Age + ⋯ + 𝛽𝛽4 𝐵𝐵𝐵𝐵 + 𝛽𝛽5 S1 + ⋯ + 𝛽𝛽10 S6
• General form:
𝑦𝑦 ≈ 𝑦𝑦� = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥1 + ⋯ + 𝛽𝛽4 𝑥𝑥4 + 𝛽𝛽5 𝑥𝑥5 + ⋯ + 𝛽𝛽10 𝑥𝑥10
Target
Intercept 10 Features

#12
Multiple Variable Linear Model

• Vector of features: 𝒙𝒙 = [𝑥𝑥1 , … , 𝑥𝑥𝑘𝑘 ]

– 𝑘𝑘 features (also known as predictors, independent variable, attributes, covariates, …)
• Single target variable 𝑦𝑦
– What we want to predict
• Linear model: Make a prediction 𝑦𝑦�

𝑦𝑦 ≈ 𝑦𝑦� = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥1 + ⋯ + 𝛽𝛽𝑘𝑘 𝑥𝑥𝑘𝑘

• Data for training
– Samples are (𝒙𝒙𝑖𝑖 , 𝑦𝑦𝑖𝑖 ), i=1,2,…,n.
– Each sample has a vector of features: 𝒙𝒙𝑖𝑖 = [𝑥𝑥𝑖𝑖𝑖 , … , 𝑥𝑥𝑖𝑖𝑖𝑖 ] and scalar target 𝑦𝑦𝑖𝑖
• Problem: Learn the best coefficients 𝜷𝜷 = [𝛽𝛽0 , 𝛽𝛽1 , … , 𝛽𝛽𝑘𝑘 ] from the training data

#13
Example: Heart Rate Increase

• Linear Model: HR increase ≈ 𝛽𝛽0 + 𝛽𝛽1 [mins

exercise] + 𝛽𝛽2 exercise intensity
• Data:
Subject HR before HR after Mins on Speed Days exercise
number treadmill (min/km) / week
123 60 90 1 5.2 3

456 80 110 2 4.1 1

789 70 130 5 3.5 2

Measuring fitness of athletes
⋮ ⋮ ⋮ ⋮ ⋮ ⋮

https://www.mercurynews.com/2017/10/29/4851089/

#14
Why Use a Linear Model?

• Many natural phenomena have linear relationship

• Predictor has small variation
• Simple to compute
• Easy to interpret relation
– Coefficient 𝛽𝛽𝑗𝑗 indicates the importance of feature j for the
target.
• Advanced: Gaussian random variables:
– If two variables are jointly Gaussian, the optimal predictor
of one from the other is linear predictor
#15
Matrix Review

• Consider
1 2
2 0 2
𝐴𝐴 = 3 4 , 𝐵𝐵 = , 𝑥𝑥 = ,
3 2 3
5 6
• Compute (computations on the board):
– Matrix vector multiply: 𝐴𝐴𝐴𝐴
– Transpose: 𝐴𝐴𝑇𝑇
– Matrix multiply: 𝐴𝐴𝐴𝐴
– Solution to linear equations: Solve for 𝑢𝑢: 𝑥𝑥 = 𝐵𝐵𝐵𝐵
– Matrix inverse: 𝐵𝐵−1
#16
Slopes, Intercept and Inner Products
• Model with coefficients 𝜷𝜷: 𝑦𝑦� = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥1 + ⋯ + 𝛽𝛽𝑘𝑘 𝑥𝑥𝑘𝑘
• Sometimes use weight bias version:
𝑦𝑦� = 𝑏𝑏 + 𝑤𝑤1 𝑥𝑥1 + ⋯ + 𝑤𝑤𝑘𝑘 𝑥𝑥𝑘𝑘

– 𝑏𝑏 = 𝛽𝛽0 : Bias or intercept

– 𝒘𝒘 = 𝜷𝜷1:𝑘𝑘 = [𝛽𝛽1 , … , 𝛽𝛽𝑘𝑘 ]: Weights or slope vector
• Can write either with inner product:
𝑦𝑦� = 𝛽𝛽0 + 𝜷𝜷1:𝑘𝑘 ⋅ 𝒙𝒙 or 𝑦𝑦� = 𝑏𝑏 + 𝒘𝒘 ⋅ 𝒙𝒙

• Inner product:
– 𝒘𝒘 ⋅ 𝒙𝒙 = ∑𝑘𝑘𝑗𝑗=1 𝑤𝑤𝑗𝑗 𝑥𝑥𝑗𝑗
– Will use alternate notation: 𝐰𝐰 𝑇𝑇 𝒙𝒙 = 𝒘𝒘, 𝒙𝒙

#17
 Matrix Form of Linear Regression

• Data: 𝒙𝒙𝑖𝑖 , 𝑦𝑦𝑖𝑖 , 𝑖𝑖 = 1, … , 𝑛𝑛

• Predicted value for 𝑖𝑖-th sample: 𝑦𝑦�𝑖𝑖 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥𝑖𝑖𝑖 + ⋯ + 𝛽𝛽𝑘𝑘 𝑥𝑥𝑖𝑖𝑖𝑖
• Matrix form 𝑦𝑦�
1 1 𝑥𝑥11 ⋯ 𝑥𝑥1𝑘𝑘 𝛽𝛽0
𝑦𝑦�2 1 𝑥𝑥21 ⋯ 𝑥𝑥2𝑘𝑘 𝛽𝛽1 𝜷𝜷 with 𝑝𝑝 = 𝑘𝑘 + 1 coefficient vector
� a 𝑛𝑛 predicted values
𝒚𝒚 =
⋮ ⋮ ⋮ ⋯ ⋮ ⋮
𝑦𝑦�𝑛𝑛 1 𝑥𝑥𝑛𝑛1 ⋯ 𝑥𝑥𝑛𝑛𝑘𝑘 𝛽𝛽𝑘𝑘

𝑨𝑨 a 𝑛𝑛 × 𝑝𝑝 feature matrix

• Matrix equation: � = 𝑨𝑨 𝜷𝜷
𝒚𝒚
#18
In-Class Exercise

#19
Outline

• Motivating Example: Understanding glucose levels in

diabetes patients
• Multiple variable linear models
• Least squares solutions
• Computing the solutions in python
• Special case: Simple linear regression
• Extensions

#20
Least Squares Model Fitting

• How do we select parameters 𝜷𝜷 = (𝛽𝛽0 , … , 𝛽𝛽𝑘𝑘 )?

• Define 𝑦𝑦�𝑖𝑖 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥𝑖𝑖𝑖 + ⋯ + 𝛽𝛽𝑘𝑘 𝑥𝑥𝑖𝑖𝑖𝑖
– Predicted value on sample 𝑖𝑖 for parameters 𝜷𝜷 = (𝛽𝛽0 , … , 𝛽𝛽𝑘𝑘 )
• Define average residual sum of squares:
𝑛𝑛

RSS 𝜷𝜷 : = � 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 2

𝐼𝐼=1
– Note that 𝑦𝑦�𝑖𝑖 is implicitly a function of 𝜷𝜷 = (𝛽𝛽0 , … , 𝛽𝛽𝑘𝑘 )
– Also called the sum of squared residuals (SSR) and sum of squared
errors (SSE)
• Least squares solution: Find 𝜷𝜷 to minimize RSS.

#21
Variants of RSS

• Often use some variant of RSS

– Note: these are not standard
• Residual sum of squares: RSS = ∑𝑛𝑛𝑖𝑖=1 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 2

• RSS per sample or Mean Squared Error:

𝑛𝑛
RSS 1 2
MSE = = � 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖
𝑛𝑛 𝑛𝑛
𝑖𝑖=1

• Normalized RSS or Normalized MSE:

𝑛𝑛 2
𝑅𝑅𝑅𝑅𝑅𝑅⁄𝑛𝑛 𝑀𝑀𝑀𝑀𝑀𝑀 ∑𝑖𝑖=1 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖
2 = 2 = 𝑛𝑛 2
𝑠𝑠𝑦𝑦 𝑠𝑠𝑦𝑦 ∑𝑖𝑖=1 𝑦𝑦𝑖𝑖 − 𝑦𝑦�

#22
Finding Parameters via Optimization
A general ML recipe

General ML problem Multiple linear regression

• Pick a model with parameters Linear model: 𝑦𝑦� = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥1 + ⋯ + 𝛽𝛽𝑘𝑘 𝑥𝑥𝑘𝑘
• Get data
Data: 𝒙𝒙𝑖𝑖 , 𝑦𝑦𝑖𝑖 , 𝑖𝑖 = 1,2, … , 𝑛𝑛
• Pick a loss function
– Measures goodness of fit Loss function:
𝑅𝑅𝑅𝑅𝑅𝑅 𝛽𝛽0 , … , 𝛽𝛽𝑘𝑘 ≔ ∑ 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 2
model to data
– Function of the parameters

• Find parameters that minimizes Select 𝜷𝜷 = (𝛽𝛽0 , … , 𝛽𝛽𝑘𝑘 ) to minimize 𝑅𝑅𝑅𝑅𝑅𝑅 𝜷𝜷

loss

#23
RSS as a Vector Norm

• RSS is given by sum:

𝑛𝑛
RSS = � 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 2
𝑖𝑖=1
• Define norm of a vector:
– 𝒙𝒙 = 𝑥𝑥12 + ⋯ + 𝑥𝑥𝑟𝑟2 1/2
– Standard Euclidean norm.
– Sometimes called ℓ-2 norm. ℓ is for Lebesque

• Write RSS in vector form:

� 2
RSS = 𝒚𝒚 − 𝒚𝒚

#24
Least Squares Solution

• Consider cost function of the RSS:

𝑛𝑛 𝑝𝑝
RSS 𝜷𝜷 = � 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 2, 𝑦𝑦�𝑖𝑖 = � 𝐴𝐴𝑖𝑖𝑖𝑖 𝛽𝛽𝑗𝑗
𝑖𝑖=1 𝑗𝑗=0

– Vector 𝜷𝜷 that minimizes RSS called the least-squares solution

• Least squares solution: The vector 𝜷𝜷 that minimizes the RSS is:
� = 𝑨𝑨𝑇𝑇 𝑨𝑨
𝜷𝜷 −1 𝑨𝑨𝑇𝑇 𝒚𝒚

– Can compute the best coefficient vector analytically

– Just solve a linear set of equations
– Will show the proof below

#25
Proving the LS Formula

• Least squares formula: The vector 𝜷𝜷 that minimizes the RSS is:
� = 𝑨𝑨𝑇𝑇 𝑨𝑨
𝜷𝜷 −1 𝑇𝑇
𝑨𝑨 𝒚𝒚

• To prove this formula, we will:

– Review gradients of multi-variable functions

– Compute gradient 𝛻𝛻𝑅𝑅𝑅𝑅𝑅𝑅 𝜷𝜷

– Solve 𝛻𝛻𝑅𝑅𝑅𝑅𝑅𝑅 𝜷𝜷 = 0

#26
Gradients of Multi-Variable Functions

• Consider scalar valued function of a vector: 𝑓𝑓 𝜷𝜷 = 𝑓𝑓(𝛽𝛽1 , … , 𝛽𝛽𝑛𝑛 )

• Gradient is the column vector:
𝜕𝜕𝜕𝜕(𝜷𝜷)⁄𝜕𝜕𝛽𝛽1
𝛻𝛻𝑓𝑓 𝜷𝜷 = ⋮
𝜕𝜕𝜕𝜕(𝜷𝜷)⁄𝜕𝜕𝛽𝛽𝑛𝑛
• Represents direction of maximum increase
• At a local minima or maxima: 𝛻𝛻𝑓𝑓 𝜷𝜷 = 0
– Solve 𝑛𝑛 equations and 𝑛𝑛 unknowns
• Ex: 𝑓𝑓 𝛽𝛽1 , 𝛽𝛽2 = 𝛽𝛽1 sin 𝛽𝛽2 + 𝛽𝛽12 𝛽𝛽2 .
– Compute 𝛻𝛻𝑓𝑓 𝜷𝜷 try it! .

#27
Proof of the LS Formula

• Consider cost function of the RSS:

𝑛𝑛 𝑝𝑝
RSS = � 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 2, 𝑦𝑦�𝑖𝑖 = � 𝐴𝐴𝑖𝑖𝑖𝑖 𝛽𝛽𝑗𝑗
𝑖𝑖=1 𝑗𝑗=0

– Vector 𝜷𝜷 that minimizes RSS called the least-squares solution

𝜕𝜕𝜕𝜕𝜕𝜕𝜕𝜕 𝑛𝑛
• Compute partial derivatives via chain rule: = −2 ∑𝑖𝑖=1 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 𝐴𝐴𝑖𝑖𝑖𝑖 , 𝑗𝑗 = 1,2, … , 𝑘𝑘
𝜕𝜕𝛽𝛽𝑗𝑗
• Matrix form: RSS = 𝐴𝐴𝜷𝜷 − 𝒚𝒚 2 , 𝛻𝛻𝑅𝑅𝑅𝑅𝑅𝑅 = −2𝐴𝐴𝑇𝑇 (𝒚𝒚 − 𝑨𝑨𝑨𝑨)
• Solution: 𝐴𝐴𝑇𝑇 𝒚𝒚 − 𝐴𝐴𝜷𝜷 = 0 → 𝜷𝜷 = 𝐴𝐴𝑇𝑇 𝐴𝐴 −1 𝐴𝐴𝑇𝑇 𝒚𝒚 (least squares solution of equation
𝐴𝐴𝜷𝜷 = 𝐲𝐲)
• Minimum RSS: 𝑅𝑅𝑅𝑅𝑅𝑅 = 𝒚𝒚𝑇𝑇 𝐼𝐼 − 𝐴𝐴 𝐴𝐴𝑇𝑇 𝐴𝐴 −1 𝐴𝐴𝑇𝑇 𝒚𝒚
– Proof in previous lecture

#28
In-Class Exercise

#29
Outline

• Motivating Example: Understanding glucose levels in

diabetes patients
• Multiple variable linear models
• Least squares solutions
• Computing the solutions in python
• Special case: Simple linear regression
• Extensions

#30
Arrays and Vector in Python

• Python:
– Arrays can have 1, 2, 3, … dimension
– Vectors can be 1D arrays; matrices are generally 2D arrays
– Vectors that are 1D arrays are neither row not column vectors
– If x is 1D and A is 2D, then left and right multiplication are the
same: x.dot(A) and A.dot(x)

• Lecture notes: We will generally treat 𝑥𝑥 and 𝑥𝑥 𝑇𝑇 the same.

– Can write 𝑥𝑥 = (𝑥𝑥1 , … , 𝑥𝑥𝑁𝑁 ) and still multiply by a matrix on left or
right

#31
Fitting Using sklearn

• Return to diabetes data example

• All code in demo
• Divide data into two portions:
– Training data: First 300 samples
– Test data: Remaining 142
samples
• Train model on training data.
• Test model (i.e. measure RSS) on
test data
• Reason for splitting data discussed
next lecture.

#32
Manually Computing the Solution

• Use numpy linear algebra routine to

solve
𝛽𝛽 = 𝐴𝐴𝑇𝑇 𝐴𝐴 −1 𝐴𝐴𝑇𝑇 𝑦𝑦
• Common mistake:
– Compute matrix inverse 𝑃𝑃 = 𝐴𝐴𝑇𝑇 𝐴𝐴 −1 ,
– Then compute 𝛽𝛽 = 𝑃𝑃𝐴𝐴𝑇𝑇 𝑦𝑦
– Full matrix inverse is VERY slow. Not
needed.
– Can directly solve linear system: 𝐴𝐴 𝛽𝛽 =
𝑦𝑦
– Numpy has routines to solve this
directly

#33
Calling the sklearn Linear Regression method

• Construct a linear regression

object
• Run it on the training data
• Predict values on the test
data

#34
Outline

• Motivating Example: Understanding glucose levels in

diabetes patients
• Multiple variable linear models
• Least squares solutions
• Computing the solutions in python
• Special case: Simple linear regression
• Extensions

#35
Simple vs. Multiple Regression

• Simple linear regression: One predictor (feature)

– Scalar predictor 𝑥𝑥
– Linear model: 𝑦𝑦� = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥
– Can only account for one variable

• Multiple linear regression: Multiple predictors (features)

– Vector predictor 𝒙𝒙 = (𝑥𝑥1 , … , 𝑥𝑥𝑘𝑘 )
– Linear model: 𝑦𝑦� = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥1 + ⋯ + 𝛽𝛽𝑘𝑘 𝑥𝑥𝑘𝑘
– Can account for multiple predictors
– Turns into simple linear regression when 𝑘𝑘 = 1

#36
Comparison to Single Variable Models

• We could compute models for each variable separately:

𝑦𝑦 = 𝑎𝑎1 + 𝑏𝑏1 𝑥𝑥1
𝑦𝑦 = 𝑎𝑎2 + 𝑏𝑏2 𝑥𝑥2
⋮
• But, doesn’t provide a way to account for joint effects
• Example: Consider three linear models to predicting longevity:
– A: Longevity vs. some factor in diet (e.g. amount of fiber
consumed)
– B: Longevity vs. exercise
– C: Longevity vs. diet AND exercise
– What does C tell you that A and B do not?

#37
Special Case: Single Variable

• Suppose 𝑘𝑘 = 1 predictor.
• Feature matrix and coefficient vector:
1 𝑥𝑥1
𝛽𝛽0
𝐴𝐴 = ⋮ ⋮ , 𝛽𝛽 =
𝛽𝛽1
1 𝑥𝑥𝑛𝑛
1 𝑇𝑇 −1 1
• LS soln: 𝛽𝛽 = 𝐴𝐴 𝐴𝐴 𝐴𝐴𝑇𝑇 𝑦𝑦
= 𝑃𝑃−1 𝑟𝑟
𝑁𝑁 𝑁𝑁
1 𝑥𝑥̅ 𝑦𝑦�
𝑃𝑃 = , 𝑟𝑟 =
𝑥𝑥̅ 𝑥𝑥 2 𝑥𝑥𝑦𝑦
• Obtain single variable solutions for coefficients (after some algebra):
𝑠𝑠𝑥𝑥𝑥𝑥 𝑠𝑠 2
𝑥𝑥𝑥𝑥
𝛽𝛽1 = 2 , 𝛽𝛽0 = 𝑦𝑦� − 𝛽𝛽1 𝑥𝑥,̅ 𝑅𝑅2 = 2 2
𝑠𝑠𝑥𝑥 𝑠𝑠𝑥𝑥 𝑠𝑠𝑦𝑦

#38
Simple Linear Regression for Diabetes Data

• Try a fit of each variable individually

• Compute 𝑅𝑅𝑘𝑘2 coefficient for each variable
• Use formula on previous slide
• “Best” individual variable is a poor fit
– 𝑅𝑅𝑘𝑘2 ≈ 0.34
Best individual variable

#39
Scatter Plot

• No one variable explains glucose well

• Multiple linear regression is much better

#40
Outline

• Motivating Example: Understanding glucose levels in

diabetes patients
• Multiple variable linear models
• Least squares solutions
• Computing in python
• Extensions

#41
Next Lecture

• Model Selection

#42