Linear regression

with one variable

Supervised Learning
Regression Problem: Predict real-valued output

Andrew Ng
Supervised Learning
The computer is presented with example inputs and
their desired outputs, and the goal is to learn a general
rule that maps inputs to outputs.

Andrew Ng
Supervised Learning: Regression
• Goal: Determine the function, which maps x to y
• Function: Approximated using the dataset
• The machine learns, for what value of x, what value
of y is usually obtained
• Formulated as a function
• Any unseen x as input provides an expected y
Andrew Ng
 Training set of housing Size in feet2 (x) Price ($) in 1000's (y)
prices
(Portland, OR) 2104 460
Linear regression with one variable.
1416 232
Univariate linear regression. 1534 315 m
One variable 852 178
… …
Notation:
m = Number of training examples
x’s = “input” variable / features
x(1) = 2104
y’s = “output” variable / “target” variable
x(2) = 1416
(x, y) – one training example y(1) = 460
(x(i), y(i)) – ith trainingg example
Andrew Ng
Andrew Ng
Andrew Ng
Andrew Ng
 How do we represent h ?
Training Set

Learning Algorithm

Size of h Estimated
house price
Degree 7 polynomial
x hypothesis y Linear hypothesis
hypothesis
h maps from x’s to y’s

Restriction bias: Consider only

linear functions

Andrew Ng
 Size in feet2 (x) Price ($) in 1000's (y)
Training Set
2104 460
1416 232
1534 315
852 178
… …
Hypothesis:

‘s: Parameters
How to choose ‘s ?
Andrew Ng
3 3 3
h(x) = 1.5 + 0·x h(x) = 0.5·x
2 2 2

1 1 1
h(x) = 1 + 0.5·x
0 0 0
0 1 2 3 0 1 2 3 0 1 2 3

Andrew Ng
 (x(i), y(i))
1 𝑚 (i)) −y(i) )2
y minimize 𝑖=1 (hΘ(x
Θ0, Θ1 Θ0 Θ1 2𝑚

h(x(i)) = Θ0+ Θ1x(i)

x
1 𝑚 (i)) −y(i) )2
J(Θ0, Θ1) = 𝑖=1 (hΘ(x
2𝑚

Idea: Choose so that

Minimize J(Θ0, Θ1) : Cost Function
is close to for our Θ0 Θ1

training examples Squared error function

Andrew Ng
 Linear regression
with one variable
Cost function
intuition I
Machine Learning

Andrew Ng
 Simplified
Hypothesis:

Parameters:
h(x) Θ0 = 0
h(x)

Cost Function:

Goal:

Andrew Ng
 (for fixed , this is a function of x) (function of the parameter )

3 3

2 2
y
1 1

0 0
0 1 x 2 3 -0.5 0 0.5 1 1.5 2 2.5
1 𝑚 (i)) − y(i))2
J(Θ1) = 𝑖=1 (hΘ(x
2𝑚
1 𝑚 (i)) − y(i)) 2
= 𝑖=1 (Θ1x 𝐽 1 =0
2𝑚
1
= 02 + 02 + 02
2𝑚 Andrew Ng
 (for fixed , this is a function of x) (function of the parameter )

3 3

2 2
y y(i)
1 1
hΘ(x(i))
0 0
0 1 2 3 -0.5 0 0.5 1 1.5 2 2.5
x
1 3
J(0.5) = 𝑖=1 [(0.5−1)2 +(1−2)2+(1.5−3)2]
2∙3
1
= ∙(3.5) = 0.58
6
Andrew Ng
 (for fixed , this is a function of x) (function of the parameter )

3 3

2 2
y
1 1

0 0
0 1 2 3 -0.5 0 0.5 1 1.5 2 2.5
x
1 3 2
J(0) = 𝑖=1 [1 +22+32]
2∙3
1
= 6
∙ 14 = 2.3
Andrew Ng
 Linear regression
with one variable
Cost function
intuition II
Machine Learning

Andrew Ng
Hypothesis:

Parameters:

Cost Function:

Goal:

Andrew Ng
 (for fixed , this is a function of x) (function of the parameters )

500

400
Price ($) 300
in 1000’s
200
Θ0 = 50
100
Θ1 = 0.06
0
0 1000 2000 3000
Size in feet2 (x)

Andrew Ng
Contour plots

Andrew Ng
 (for fixed , this is a function of x) (function of the parameters )

Is the slope positive or negative What is the value of

Do you agree?
Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

h(x) = 360 + 0·x

Θ0 = 360
Θ1 = 0
Andrew Ng
 (for fixed , this is a function of x) (function of the parameters )

Is the slope positive or negative What is the value of

Do you agree?
Andrew Ng
 (for fixed , this is a function of x) (function of the parameters )

Is the slope positive or negative What is the value of

Do you agree?
Andrew Ng
 Linear regression
with one variable

Gradient
Machine Learning
descent
Andrew Ng
Have some function
Want

Outline:
• Start with some
• Keep changing to reduce
until we hopefully end up at a minimum

Andrew Ng
J(0,1)

1
0

Andrew Ng
J(0,1)

1
0

Andrew Ng
 Linear regression
with one variable
Gradient descent for
linear regression
Machine Learning

Andrew Ng
J(0,1)

1
0

Andrew Ng
Convex function

Bowl-shaped

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng
(for fixed , this is a function of x) (function of the parameters )

Andrew Ng