UET

Since 2004

ĐẠI HỌC CÔNG NGHỆ, ĐHQGHN

VNU-University of Engineering and Technology

INT3405 - Machine Learning

Lecture 3: Linear Regression

Hanoi, 09/2024
 Outline
● Supervised Learning
● Linear Regression with One Variable
○ Model Representation
○ Cost Functions
○ Gradient Descent
● Linear Regression with Multiple Variables
○ Learning rate
○ Normal Equation

FIT-CS INT3405 - Machine Learning 2

 Recap: Random Variables

FIT-CS INT3405 - Machine Learning 3

 Recap: Inductive Learning

FIT-CS INT3405 - Machine Learning 4

 Recap: Supervised Learning Workflow

FIT-CS INT3405 - Machine Learning 5

 Supervised Learning
● Supervised (Inductive) Learning
● Formalization
○ Input:

○ Output:

○ Target function: (unknown)

○ Training Data:

○ Hypothesis:

○ Hypothesis space:
FIT-CS INT3405 - Machine Learning 6
 A Learning Problem

Unknown
Function
Input Output

FIT-CS INT3405 - Machine Learning 7

 Hypothesis Spaces
● Linear models

○ Infinite possible hypotheses!

○ Any choices of coefficient a and b will result in a possible hypothesis
● Polynomial models

● Any nonlinear models

FIT-CS INT3405 - Machine Learning 8

 Two Views of Learning
● Learning is the removal of our remaining uncertainty.
○ If we are know that x and y are linearly dependent, then we could
use the training data to infer the linear function
● Learning requires guessing a good, small hypothesis class.
○ We could start with a very small / simple class, and enlarge it until it
contains a hypothesis that fits the data
● But we could be wrong
○ Our prior knowledge might be wrong
○ Our guess of the hypothesis class could be wrong
■ The smaller the hypothesis class, the more likely we are wrong
FIT-CS INT3405 - Machine Learning 9
 Two Strategies for Machine Learning
● Develop Languages for Expressing Prior Knowledge
○ Rule grammars and stochastic models

● Develop Flexible Hypothesis Spaces

○ Nested collections of hypotheses, rules, linear models, decision trees,
neural networks, etc.

● For either case, the key is to

○ Developing efficient algorithms for finding a Hypothesis that best
approximates the target function for fitting the data
FIT-CS INT3405 - Machine Learning 10
 Key Issues in Machine Learning
● What are good hypothesis spaces?
○ Which spaces have been useful in practical applications and why?
● What algorithms can work with these spaces?
○ Are there general design principles for machine learning algorithms?
● How can we find the best hypothesis in an efficient way?
○ How to find the optimal solution efficiently (“optimization” question)
● How can we optimize accuracy on future data?
○ Known as the “overfitting” problem (i.e., “generalization” theory)
● How can we have confidence in the results?
○ How much training data is required to find accurate hypothesis? (“statistical” question)
● Are some learning problems computationally intractable? (“computational” question)
● How can we formulate application problems as machine learning problems? (“engineering”
question)
FIT-CS INT3405 - Machine Learning 11
 Regression with One Variable (1)
Housing Prices
(Portland, OR)
Price
(in 1000s of dollars)

Size (feet2)

Supervised Learning Regression Problem

Given the “right answer” for each Predict real-valued output
example in the data.

FIT-CS INT3405 - Machine Learning 12

 Regression with One Variable (2)

Training set of Size in feet2 (x) Price ($) in 1000's (y)

housing prices 2104 460
(Portland, OR) 1416 232
1534 315
852 178
… …
Notation:
m = Number of training examples
x’s = “input” variable / features
y’s = “output” variable / “target” variable

FIT-CS INT3405 - Machine Learning 13

 Model Representation
Training Set How do we represent h ?

Learning Algorithm y

Size of h Estimated x
house price
x Hypothesis y
Linear regression with one variable.
“Univariate Linear Regression”

How to choose parameters ?

FIT-CS INT3405 - Machine Learning 14
 Formulation: Cost Function (1)
Hypothesis:

Parameters:

y
Cost Function: mean squared error (MSE)

x
Goal:

FIT-CS INT3405 - Machine Learning 15

 Formulation: Cost Function (2)
Simplified
Hypothesis:

Parameters:

Cost Function:

Goal:

FIT-CS INT3405 - Machine Learning 16

 Cost Function: Example (1)

For fix this is a function of x function of the parameter

FIT-CS INT3405 - Machine Learning 17

 Cost Function: Example (2)

For fix this is a function of x function of the parameter

FIT-CS INT3405 - Machine Learning 18

 Cost Function: Example (3)

For fix this is a function of x function of the parameter

FIT-CS INT3405 - Machine Learning 19

 Cost Function (1)

Hypothesis:

Parameters:

Cost Function:

Goal:

FIT-CS INT3405 - Machine Learning 20

 Cost Function (2)

(for fixed , this is a function of x) (function of the parameters )

Price ($)
in
1000’s

Size in feet2 (x)

FIT-CS INT3405 - Machine Learning 21

 Cost Function (3)
● Contour plots

FIT-CS INT3405 - Machine Learning 22

 Cost Function (4)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 23

 Cost Function (5)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 24

 Gradient Descent for Optimization (1)

Given some objective function

Want to optimize

Outline:
• Start with some
• Keep changing to reduce
until we hopefully end up at a minimum

FIT-CS INT3405 - Machine Learning 25

 Gradient Descent for Optimization (2)

FIT-CS INT3405 - Machine Learning 26

 Gradient Descent for Optimization (3)

FIT-CS INT3405 - Machine Learning 27

 Gradient Descent Algorithm

Gradient descent algorithm

learning rate parameter

(rule of thumb: 0.1)

FIT-CS INT3405 - Machine Learning 28

 Gradient Descent for Linear Regression (1)
Gradient descent algorithm Linear Regression Model

FIT-CS INT3405 - Machine Learning 29

 Gradient Descent for Linear Regression (2)

Gradient descent algorithm

update
and
simultaneously

FIT-CS INT3405 - Machine Learning 30

 Gradient Descent Example (1)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 31

 Gradient Descent Example (2)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 32

 Gradient Descent Example (3)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 33

 Gradient Descent Example (4)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 34

 Gradient Descent Example (5)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 35

 Gradient Descent Example (6)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 36

 Gradient Descent Example (7)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 37

 Gradient Descent Example (8)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 38

 Gradient Descent Example (9)

(for fixed , this is a function of x) (function of the parameters )

FIT-CS INT3405 - Machine Learning 39

 Batch Gradient Descent

“Batch”: Each step of gradient descent uses all the

training examples.

FIT-CS INT3405 - Machine Learning 40

 Multivariate Linear Regression (1)
Multiple features (variables).

Size (feet2) Number of Number of Age of home Price ($1000)

bedrooms floors (years)

2104 5 1 45 460
1416 3 2 40 232
1534 3 2 30 315
852 2 1 36 178
… … … … …

Notation:
= number of features
= input (features) of training example.
= value of feature in training example.
FIT-CS INT3405 - Machine Learning 41
 Multivariate Linear Regression (2)
Hypothesis:

Previously:

For convenience of notation, define .

FIT-CS INT3405 - Machine Learning 42

 Gradient Descent for Multivariate LR
Hypothesis:

Parameters:

Cost function:

Gradient descent:
Repeat

(simultaneously update for every )

FIT-CS INT3405 - Machine Learning 43

 Univariate LR vs Multivariate LR

Gradient Descent
Previously (n=1): New algorithm :
Repeat Repeat

(simultaneously update )

FIT-CS INT3405 - Machine Learning 44

 Convergence and Learning Rate

Example automatic convergence test:

Declare convergence if
decreases by less than
in one iteration.

No. of iterations
For sufficiently small , should decrease on every iteration.
But if is too small, gradient descent can be slow to converge.
If is too large: may not decrease on every iteration; may not converge.
FIT-CS
SML– Term 1 2020-2021 INT3405 - Machine Learning 45
45
 Learning Rate

divergenc
e

gradually decreased
too small too
constant large

FIT-CS INT3405 - Machine Learning 46

 Normal Equation (1)
Gradient Descent
• Iterative approach
Normal Equation
• Analytical method to solve
Intuition Example: If 1D

Solve equation to find w

FIT-CS INT3405 - Machine Learning 47

 Normal Equation (2)

FIT-CS INT3405 - Machine Learning 48

 Normal Equation (3)
● Matrix-vector formulation

● Analytical solution

FIT-CS INT3405 - Machine Learning 49

 The Pseudo-inverse

FIT-CS INT3405 - Machine Learning 50

 Normal Equation: Example
Examples:
Size (feet2) Number of Number of Age of home Price ($1000)
bedrooms floors (years)

1 2104 5 1 45 460
1 1416 3 2 40 232
1 1534 3 2 30 315
1 852 2 1 36 178

is inverse of matrix .

FIT-CS INT3405 - Machine Learning 51

 Gradient Descent vs Normal Equation
training examples, features.
Gradient Descent Normal Equation
• Need to choose . • No need to choose .
• Needs many iterations. • Don’t need to iterate.
• Works well even • Need to compute
when is large.
• Slow if is very large.

FIT-CS INT3405 - Machine Learning 52

 Summary
● Supervised Learning
● Linear Regression with One Variable
○ Model Representation
○ Cost Functions
○ Gradient Descent
● Linear Regression with Multiple Variables
○ Learning rate
○ Normal Equation

Duc-Trong Le
FIT-CS INT3405 - Machine Learning 53
 Reading
To understand this lecture better: 3.1 to 3.2, ESL; 8.4, 11.2 PML
For next lectures: 5, 10 PML

FIT-CS INT3405 - Machine Learning 54

UET
Since 2004

ĐẠI HỌC CÔNG NGHỆ, ĐHQGHN

VNU-University of Engineering and Technology

Thank you