Regression

■ Duration: 2 hrs
■ Outline:
1. What is regression?
2. Simple linear regression
3. Example
 Regression

■ Duration: 2 hrs
■ Outline:
1. What is regression?
2. Simple linear regression
3. Example
 Regression
 What is regression analysis?

 A basic and commonly used predictive analysis (predictive analysis:

the use of data, statistical algorithms and machine learning
techniques to identify the likelihood of future outcomes based on
historical data)
 Goals:

 To establish the possible relationship between variables

 To predict a real-valued label (target) given an unlabeled example

 Regression
 What is regression analysis?

 A basic and commonly used predictive analysis (predictive analysis:

the use of data, statistical algorithms and machine learning
techniques to identify the likelihood of future outcomes based on
historical data)
 Goals:

 To establish the possible relationship between variables

 To predict a real-valued label (target) given an unlabeled example

Regression
 Regression
 Applications: finance, investing, and other disciplines  to
determine the strength and character of the relationship between
one dependent variable (y) and a series of other independent
variables.

 Ex: estimating house price valuation based on house features

[area, the number of bedrooms, location,…]
 Types of regression

 Regression describes the relationship between variables by fitting a

line to the examples.

 Linear regression: line is straight

 Simple linear regression: using one variable to predict the
outcome

 Multiple linear regression: using two or more independent

variables to predict the outcome

 Nonlinear regression: line is curved

 Regression

■ Duration: 2 hrs
■ Outline:
1. What is regression?
2. Simple linear regression
3. Example
 Objectives

 Learning a linear model that summarizes the statistical relationship

between two continuous (quantitative) variables

 Independent variable X, or predictor variable

 Dependent variable Y, or response

 Ex: income and spending, student IQ and exam scores

 Linear model: straight line of best fit through the data points

 Note: the relationship is statistical, i.e., not perfect, not exact

Simple linear regression model
Simple linear regression model
Simple linear regression model
 Simple linear regression model

y-axis

(x2, y2)

(x1, y1)
x-axis
0 Give two point (x1, y1) and (x2, y2)
How to "develop" an equation connecting these two points?
13
 Simple linear regression model

y-axis

(x2, y2)

(x1, y1)
x-axis
0 • Find gradient (slope):
• Find (intercept) (initial coefficient of y when x=0)
16 14
 Simple linear regression model

intercept y  f x   1 x  0

slope

Figure 3.1 , ISL 2013

15
Simple linear regression model
 Assumptions of simple linear regression

 Homogeneity of variance (homoscedasticity): The size of the

error in the prediction doesn’t change significantly across the
values of the independent variable.

 Independence of observations: the observations in the dataset

were collected using statistically valid methods, and there are no
hidden relationships among variables.

 Normality: the data follows a normal distribution (Gaussian

distribution).

 Linearity: the line of best fit through the data points is a straight
line, rather than a curve or some sort of grouping factor.
Assumptions of simple linear regression
 Simple linear regression problem

Y^ = predicted dependent variable

(output)
X = independent variable (input)
β0 = intercept (the value of Y^ when
X=0),
β1 = slope (the rate of change of Y^
with respect to X).
Simple linear regression problem
Basic idea
Basic idea
Basic idea
 Regression

■ Duration: 2 hrs
■ Outline:
1. What is regression?
2. Simple linear regression
3. Example
 Restaurant tipping 1

 Predicting the amount of tip a waiter is expected to earn from the

restaurant.
Example (cont)
Example (cont)
 Restaurant tipping 2

 Developing a simple linear regression model to predict the amount

of tip a waiter can earn given the total bill paid.
Step 1: scatter plot
Step 2: look for a visual line
Step 3: correlation check (optional)
Step 4: centroid determination
Step 5: calculation
Estimated simple linear regression model
Estimated error
Estimated error
Estimated error
 SSE comparison

Tip amount only Tip amount as a function of

bill amount

amount
SSE = 120 = SST SSE = 30.075
Solving Linear Regression Using Gradient
Descent
Gradient Descent is an optimization algorithm used to find the
best values of β0 (intercept) and β1 (slope) by minimizing the
Mean Squared Error (MSE).

The goal is to minimize MSE, which is equivalent to minimizing

the Sum of Squared Errors (SSE):

It can be used gradient descent to iteratively adjust β0

and β1 to minimize this cost.
 Basic idea

Compute the Gradients

The gradients (partial derivatives) of the cost function

with respect to β0 and β1 are.

These gradients tell us the direction in which we should update β0 and β1.
 Basic idea

Gradient Descent Update Rules

Using a learning rate α\alphaα, the update rules for β0

and β1 are.

which expands to:

Basic idea • Initialize β1
• Repeat until convergence
 Gradient Descent

• Initialize β1
• Repeat until convergence
simultaneous update
β1 ➛ β1 β1
β1 for j = 0 ... d

learning rate (small)

e.g., α = 0.05 3

2
J(β1)
1

0
-0.5 0 0.5 1 1.5 2 2.5
β1
 How to improve the model?

 Including more predictors

 The bills were paid by male or female?

 The customers are smokers or non-smokers?

 The meals are on Monday, Tuesday,… or Sunday?

 The meals took place during lunchtime or dinner?

 Multiple linear regression

 Multiple Linear Regression

f x   w0  w1 x1  w2 x2    w p x p

w  w0 , w1 , w2 , , w p 
T


x  1, x1 , x2 , , x p 
f x   xw

Loss
function
Polynomial Regression
Overfitting

Training, true, & test error vs. model complexity

Overfitting if:
Error

y Model complexity y

2 x x
Underfitting and overfitting illustrated
Underfitting and overfitting illustrated
 Exercise 1
Predicting weight from height.

Note: remove outlier

Height (cm) Weight (kg) Height (cm) Weight (kg)

 Exercise 2

Predicting job performance from IQ scores.

If an applicant’s IQ score = 100  estimate his performance?