1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

Machine Learning
Machine Learning Resources ▾
Machine Learning and Econometrics ▾
Supervised Learning Theory ▾
Overview (/machine-learning/)

One Variable Linear Regression (/one-variable-linear-regression/)

Linear Algebra (/linear-algebra-machine-learning/)

Multiple Variable Linear Regression (/multi-variable-linear-regression/)

Logistic Regression (/logistic-regression/)

Neural Networks (Representation) (/neural-networks-representation/)

Neural Networks (Learning) (/neural-networks-learning/)

Applying Machine Learning (/applying-machine-learning/)

Machine Learning Systems Design (/machine-learning-systems-design/)

Support Vector Machines (/machine-learning-svms-support-vector-machines/)

Unsupervised Learning Theory ▾

Reinforcement Learning Theory ▾
Deep Learning Theory ▾
Deep Learning with TensorFlow ▾
Machine Learning with Scikit-Learn ▾
Machine Learning Projects ▾

Linear Regression with Multiple Variables

Summary: Linear Regression with Multiple Variables.

Table of Contents
1. Multivariate Linear Regression
– 1a. Multiple Features (Variables)
– 1b. Gradient Descent for Multiple Variables

http://www.ritchieng.com/multi-variable-linear-regression/ 1/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

– 1c. Gradient Descent: Feature Scaling

– 1d. Gradient Descent: Checking
– 1e. Gradient Descent: Learning Rate
– 1f. Features and Polynomial Regression

2. Computing Parameters Analytically

– 2a. Normal Equation
– 2b. Normal Equation Non-invertibility

1. Multivariate Linear Regression

I would like to give full credits to the respective authors as these are my personal python notebooks taken from
deep learning courses from Andrew Ng, Data School and Udemy :) This is a simple python notebook hosted
generously through Github Pages that is on my main personal notes repository on
https://github.com/ritchieng/ritchieng.github.io . They are meant for my personal review but I have open-source
my repository of personal notes as a lot of people found it useful.

1a. Multiple Features (Variables)

X1, X2, X3, X4 and more

http://www.ritchieng.com/multi-variable-linear-regression/ 2/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

New hypothesis

Multivariate linear regression

Can reduce hypothesis to single number with a transposed theta matrix multiplied by x matrix

1b. Gradient Descent for Multiple Variables

http://www.ritchieng.com/multi-variable-linear-regression/ 3/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

Summary

http://www.ritchieng.com/multi-variable-linear-regression/ 4/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

New Algorithm

1c. Gradient Descent: Feature Scaling

Ensure features are on similar scale

Gradient descent will take longer to reach the global minimum when the features are not on a similar
scale

Feature scaling allows you to reach the global minimum faster

http://www.ritchieng.com/multi-variable-linear-regression/ 5/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

So long they’re close enough, need not be between 1 and -1

Mean normalization

1d. Gradient Descent: Checking

Can you a graph

x-axis: number of iterations

http://www.ritchieng.com/multi-variable-linear-regression/ 6/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

y-axis: min J(theta)

Or use automatic convergence test

Tough to gauge epsilon

Gradient descent that is not working (large learning rate)

1e. Gradient Descent: Learning Rate

Alpha (Learning Rate) too small: slow convergence

http://www.ritchieng.com/multi-variable-linear-regression/ 7/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

Alpha (Learning Rate) too large:

J(theta) may not decrease on every iteration

May not converge (diverge)

Start with 0.001 and increase x3 each time until you reach an acceptable alpha

Choose a slightly smaller number than that acceptable alpha value

1f. Features and Polynomial Regression

Ensure the features capture the pattern

Doesn’t make sense to choose quadratic equation for house prices

Use cubic or square root

There are automatic algorithms, and this will be discussed later

http://www.ritchieng.com/multi-variable-linear-regression/ 8/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

2. Computing Parameters Analytically

2a. Normal Equation

Method to solve for theta analytically

If theta is real number

Minimise J(theta) is to take the derivative and equate to zero

Solve for theta

If theta is not

Take partial derivative and equate to zero

http://www.ritchieng.com/multi-variable-linear-regression/ 9/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

Solve for all thetas

Minimise Cost Function: Specific Example

X: m x (n + 1)

m: number of training examples

n: number of features

X_transpose: (n + 1) x m

X_transpose * X: (n + 1) x m * m x (n + 1) = (n + 1) x (n + 1)

(X_transpose * X)^-1 * X_transpose: (n + 1) x (n + 1) * (n + 1) x m = (n + 1) x m

http://www.ritchieng.com/multi-variable-linear-regression/ 10/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

theta = (n + 1) x m * m x 1 = (n + 1) x 1

Minimise Cost Function: General

Minimise Cost: Octave Code

No need for feature scaling using normal equation

http://www.ritchieng.com/multi-variable-linear-regression/ 11/12
1/21/2018 Linear Regression with Multiple Variables | Machine Learning, Deep Learning, and Computer Vision

pinv (X' * X) * X' * y

Gradient Descent vs Normal Equation

Gradient Descent Normal Equation

Need to choose No need to choose alpha

alpha

Needs many Don’t need to iterate

iterations

Works with large n Slow if n is large (100, 1000 is fine)

(10,000)

Number of features So long number features < 1000

> 1000

2b. Normal Equation Non-invertibility

What happens if X_transpose * X is non-invertible (singular or degenerate)

pinv (X' * X) * X' * y

This works regardless if it is non-invertible

Intuition of non-invertibility

Causes of non-invertibility

http://www.ritchieng.com/multi-variable-linear-regression/ 12/12