High Impact Skills Development Program

in Artificial Intelligence, Data Science, and Blockchain

Module 1: AI Fundamentals
Lecture 4: Linear Regression

Instructor: Dr Syed Imran Ali

Assistant Professor, SEECS, NUST

Courtesy: Dr. Faisal Shafait and Dr. Adnan ul Hasan 1

Supervised Learning
- Regression
- Classification

2
What is Regression Problem?
• We have a pile of data and we want to use this data to
make quantitative predictions
• Which means we want to use Machine Learning to do
Regression
• For example, we want to model relationship between
heights and weights of 5 different people
• Goal: Given weight of a person, predict his/her height!

3
 Solution!
• Fit a line to the original data
and use the line to make
quantitative predictions

• Models need Training Data.

Using machine learning
lingo, we say that we build
models by training machine
learning algorithms. The StatQuest Illustrated Guide To Machine Learning by Josh Starmer

4
 Solution!

• Models, or equations, can

tell us about people we
haven’t measured yet.

The StatQuest Illustrated Guide To Machine Learning by Josh Starmer

5
 Solution!

• Because models are only

approximations, it’s
important that we’re able to
measure the quality of their
predictions.

The StatQuest Illustrated Guide To Machine Learning by Josh Starmer

6
 Nature of Relationship
• Independent variables
can be manipulated or Regression Line
changed
• Dependent variable

Dependent Variable
registers an outcome

• Relationship:
• Positive
• Negative
Independent Variable(s)

7
 Positive Relationship

Regression Line

• With the increase/decrease

Dependent Variable
of independent variable the
+ve slope
dependent variable also
increases/decreases

Independent Variable(s)

8
 Negative Relationship

Regression Line
• With the increase/decrease

Dependent Variable
of independent variable the
-ve slope
dependent variable changes
in the opposite direction

Independent Variable(s)

9
 Fitting Line to Observations

Regression Line
• Method: Least Squares

Dependent Variable
X
• Objective: to minimize
difference between the
estimated values and
the actual values

Independent Variable(s)

10
Fitting Line to Observations
𝑦ො = 𝑏0 + 𝑏1 𝑥 (estimated regression line equation)
where 𝑏0 is y-intercept,
𝑏1 is slope of the line

x y 𝒙
x-ෝ 𝒚)
(y-ෝ 𝒙)2
(x-ෝ 𝒙)(y-ෝ
(x-ෝ 𝒚)
1 2 -2 -2 4 4
2 4 -1 0 1 0
3 5 0 1 0 0
4 4 1 0 1 0
5 5 2 1 4 2
Mean: 3 4 Sum: 10 6

11
 Fitting Line to Observations

x y 𝒙
x-ෝ 𝒚)
(y-ෝ 𝒙)2
(x-ෝ 𝒙)(y-ෝ
(x-ෝ 𝒚)
1 2 -2 -2 4 4
2 4 -1 0 1 0
3 5 0 1 0 0
4 4 1 0 1 0
5 5 2 1 4 2
Mean: 3 4 Sum: 10 6

σ 𝑥−𝑥ො (𝑦−𝑦)
ො 6
𝑏1 = = = 0.6
𝑥−𝑥ො 2 10
𝑦ො = 𝑏0 + 𝑏1 𝑥
𝑏0 = ?
12
 Fitting Line to Observations

x y 𝒙
x-ෝ 𝒚)
(y-ෝ 𝒙)2
(x-ෝ 𝒙)(y-ෝ
(x-ෝ 𝒚)
1 2 -2 -2 4 4
2 4 -1 0 1 0
3 5 0 1 0 0
4 4 1 0 1 0
5 5 2 1 4 2
Mean: 3 4 Sum: 10 6

σ 𝑥−𝑥ො (𝑦−𝑦)
ො 6
𝑏1 = = = 0.6
𝑥−𝑥ො 2 10
𝑦ො = 𝑏0 + 𝑏1 𝑥 𝑏0 = 2.2
4 = 𝑏0 + 0.6 3
13
 Fitting Line to Observations

x y 𝒙
x-ഥ 𝒚)
(y-ഥ 𝒙)2
(x-ഥ 𝒙)(y-ഥ
(x-ഥ 𝒚)
1 2 -2 -2 4 4
2 4 -1 0 1 0
3 5 0 1 0 0
4 4 1 0 1 0
5 5 2 1 4 2
Mean: 3 4 Sum: 10 6

σ 𝑥−𝑥ො (𝑦−𝑦)
ො 6
𝑏1 = = = 0.6
𝑥−𝑥ො 2 10
𝑦ො = 2.2 + 0.6 𝑥 𝑏0 = 2.2
14
 R2 Evaluation
• R2 tell us how well a regression line
predicts or estimates actual values

• Following are the steps for R2

Calculation:
• Take actual values and determine the
mean of those values
• Look at the distance from the actual
values to the mean

15
 R2 Evaluation
• Following are the steps for R2
Calculation:
• Take actual values and determine the
mean of those values
• Look at the distance from the actual
values to the mean
• Then draw a linear regression line and
come up with estimated values
• Take the distance from the estimated
values to the mean
• Compare (distance: actual-mean) to
(distance: estimated-mean)
16
 R2 Evaluation
x y 𝒙
x-ഥ 𝒚
y-ഥ 𝒚)2
(y-ഥ 𝒙)2
(x-ഥ ෝ = 𝟐. 𝟐 + 𝟎. 𝟔𝒙
𝒚 ෝ-y
𝒚 𝒚 − 𝒚)𝟐
(ෝ
1 2 -2 -2 4 4 2.8 -1.2 1.44
2 4 -1 0 0 1 3.4 -0.6 0.36
3 5 0 1 1 0 4 0 0
4 4 1 0 0 1 4.6 0.6 0.36
5 5 2 1 1 4 5.2 1.2 1.44
Sum: 6 Sum: 3.6

2
σ 𝑦−
ො 𝑦ത 3.6 𝑅2 = 1 𝑝𝑒𝑟𝑓𝑒𝑐𝑡 𝑚𝑎𝑡𝑐ℎ, 𝑛𝑜 𝑒𝑟𝑟𝑜𝑟
𝑅2 = σ 𝑦−𝑦ത 2 = 6
= 0.6
𝑅2 = 0 (𝑛𝑜 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠ℎ𝑖𝑝 )
17
Happy
Learning!