REGRESSION

MODELLING
-statistical approach-
 Regression
Models
Types of
Regression
Models
 Regression
1 Explanatory Variable Models
Types of
Regression
Simple
Models
 Regression
1 Explanatory Variable Models 2 Explanatory Variable
Types of
Regression
Simple Multiple
Models
 Regression
1 Explanatory Variable Models 2 Explanatory Variable
Types of
Regression
Simple Multiple
Models

Linear
 Regression
1 Explanatory Variable Models 2 Explanatory Variable
Types of
Regression
Simple Multiple
Models

Non-
Linear
Linear
 Regression
2 Explanatory Variable Models 2 Explanatory Variable
Types of
Regression
Simple Multiple
Models

Non-
Linear Linear
Linear
 Regression
1 Explanatory Variable Models 2 Explanatory Variable
Types of
Regression
Simple Multiple
Models

Non- Non-
Linear Linear
Linear Linear
Linear Regression
Model

9
 Regression
1 Explanatory Variable Models 2 Explanatory Variable
Types of
Regression
Simple Multiple
Models

Non- Non-
Linear Linear
Linear Linear
Y
Y = mX + b
Change
m = S lo p e in Y
C h a n g e in X
b = Y -in te rc e p t
X
 Relationship Between Variables is a Linear
Function
Linear
Regression Population Populati Random
Y-Intercept on Slope Error
Model
Yi  0  1Xi   i
Dependent Independent (Explanatory)
(Response) Variable
Variable
coefficient
  When a coefficient becomes zero, it effectively
removes the influence of the input variable on the
Linear model and therefore from the prediction made from
the model (0 × x = 0).
Regression  Learning a linear regression model means estimating
the values of the coefficients used in the
representation with the data that we have available
  With simple linear regression
when we have a single input,
we can use statistics to
Simple estimate the coefficients.
Linear  statistical properties from the
Regression data
 means
 standard deviations
 correlations and covariance
  Is used when we have more than one
input
 seeks to minimize the sum of the
squared residuals.
 given a regression line through the
Ordinary
data, we calculate the distance from
Least each data point to the regression line,
Squares square it, and sum all of the squared
errors together.
 all of the data must be available and
must have enough memory to fit the
data and perform matrix operations.
 Linear Regression

x x- mean (x)
1 -2
 mean (x) = 3 2 -1

 mean (y) = 3 0
2.8 4 1
5 2 y y- mean (y)
1 -1.8
3 0.2
2 -0.8
3 0.2
5 2.2
  optimizing the values of the coefficients by iteratively
minimizing the error of the model on the training data
 starts with zero values for each coefficient
 sum of the squared errors are calculated for each
pair of input and output values.
 a learning rate is used as a scale factor and the
Gradient coefficients are updated in the direction towards
minimizing the error.
Descent  the process is repeated until a minimum sum
squared error is achieved or no further improvement
is possible.
 must select a learning rate (alpha) parameter that
determines the size of the improvement step to take
on each iteration of the procedure.
 Best used on very large dataset
 Preparing Data for Linear
Regression
oLinear assumption
oRships between input and oput is linear
oRemove noise
oInput & oput are not noise
oRemove collinearity
oRemove the most correlated variables
oGaussian distributions
oInput & oput have Gaussian
distribution
oRescale inputs
oUse normalization on input
Input (x) Output (y)
1 1
2 3
3 2
4 3
5 5

EXAMPLE
Linear Regression
Yi  0  1Xi
 x – mean (x) y- mean (y) multiplication x- mean (x) squared
-2 -1.8 3.6 -2 4
-1 0.2 -0.2 -1 1
0 -0.8 0 0 0
1 0.2 0.2 1 1
2 2.2 4.4 2 4
Sum = 8 Sum = 10

1 = 0.8
Y i  0.4  0.8 Xi

0 2.8 – 0.8 x 3 = 0.4

 Linear Regression : Making
Predictions
Y i  0.4  0.8 Xi
x Predicted y
1 1.2
2 2
3 2.8
4 3.6
5 4.4
 Linear Regression : Estimating
Error
Predicted y Predicted - y Squared
error
1.2 1 0.2 0.04
2 3 -1 1
2.8 2 0.8 0.64
3.6 3 0.6 0.36
4.4 5 -0.6 0.36
Sum = 2.4

2.4 /5 = 0.69282 Each prediction is on average wrong by

about 0.69
Number of observation or data
 Pearson Correlation

SHORTCUT
(IMPLEMENTATION
)

2.8 – 0.8 x 3 = 0.4