Simple Linear Regression

Goals, Aims, and Requirements

• Understand the concept of Regression.

• Simple Linear Regression.
• The Least Squares Method
• The Coefficient of Determination
• Using the Estimated Regression
Equation for Estimation and Prediction
• t test
• F test
.
 Regression Analysis
In regression analysis we analyze the relationship between two or
more variables.

The relationship between two or more variables could be linear or

non linear.

This week we only talk about relationship between only two

variables. We also restrict our topic to Linear Regression.

Simple Linear Regression : Linear Regression Between Two

Variables

How we could use available data to investigate such a relationship?

If there exist a relationship, how could we use this relationship to

forecast future.
 Simple Linear Relationship
Linear relationship between two variables is stated as

y = 0 +  1 x

This is the general equation for a line

0 : Intersection with y axis

1 : The slope

x : The independent variable

y : The dependent variable

 Example
For example, advertising could be the independent variable
and sales to be the dependent variable.

We first implement available data to develop a relationship

between sales and advertising.

Sales = 0 + 1 (Advertising)

After estimating 0 and 1 , then we implement this relationship

to forecast sales given a specific level of advertising.

How much sales we will have if we spent a specific amount on

advertising.
 Example
A nationwide chain Pizza restaurant.

The most successful branches are close to college campuses.

Managers believe that quarterly sales is closely related to the

size of the student population.

That is : restaurants near campus with large population tend to

have more sales than those located near campus with small
population.

X : student population
y : quarterly sales
 Simple Linear Regression Model
y = 0 +  1 x + 

Y is a linear function of x plus 

0 and 1 are parameters of the model,

 is a random variable.

The linear term of 0 + 1 x is the variations in y that can be

explained by x

The error term of  is variations in y that can not be explained

by the liner relationship between x and y.
 Simple Linear Regression Model
One of the assumptions of the simple linear model is  has
normal distribution with mean of 0 and a standard deviation of σ
referred to as standard error.  : N(0, σ)

Mean ( ) = 0

Therefore,

Mean (y ) = 0 + 1 x

In words Mean of y is a linear function of x

The actual y for any given x has normal distribution with mean
of  +  x and a standard deviation of standard error.
 Simple Linear Regression Equation
For the time being forget  . The following equation describes how
the mean value of y is related to x.

E (y ) = 0 + 1 x

0 is the intersection with y axis, 1 is the slope.

1 > 0 1 < 0 1 = 0
 Estimated Linear Regression Equation
If we knew the values of 0 and 1 then given any campus
population, we could plug it into the equation and find the mean
value of the sales.

E (y ) = 0 + 1 x

But we do not know the values for 0 and 1

We have to estimate them.

We estimate them using past data.

We estimate 0 by b0 and 1 by b1
 Estimated Linear Regression Equation
Simple Linear Equation
E (y ) = 0 + 1 x

Estimate of the Simple Linear Equation

^
y = b0 + b1 x
The Estimating Process in Simple Linear Regression

Regression model X y
y = 0 + 1 x +  x1 y1
Regression equation x2 y2
E (y ) = 0 + 1 x . .
Unknown . .
0 and 1 xn yn

b0 and b1 Estimated regression

provide estimates for equation
^
y = b0 + b1 x
0 and 1
Sample statistics
b0 and b1
 Pizza Restaurant Example
We collect a set of data from random stores of our Pizza
restaurant example

Restaurant Student population Quarterly Sales

(1000s) ($1000s)
i xi yi
1 2 58
2 6 105
3 8 88
4 8 118
5 12 117
6 16 137
7 20 157
8 20 169
9 22 149
10 26 202
Scatter Diagram
 Estimated Linear Regression Equation
We want to estimate the relationship between
? ?

Student population Mean value of sales

We may rely on own judgement, and draw a line to fit them.

Then we measure the intersection with y axis and that is b0, and
the slope is b1
Graphical - Judgmental Solution

Y = 55+6x
1 b1
b0
 The Least Square Method
Our judgmental approach and our eyes tried to minimize the
difference between observed values and values obtained on the
estimated regression line.

We may implement the same approach in algebra by

Minimize Sum of the square of the difference between observed

values and values on the regression line.

The Least Square Method.

Graphical - Judgmental Solution
Graphical - Judgmental Solution
The Least Square Method

yi xi yˆ i
y1 x1 b0  b1 x1
y2 x2 b0  b1 x2
y3 x3 b0  b1 x3
. . .
. . .
yn xn b0  b1 xn

n

Min Z   (y i  y i ) 2
i 1

n
Min Z   (y i  b0  b1 x i ) 2
i 1
 Classic Minimization

n
Min Z   (y i  b0  b1 x i ) 2
i 1

We want to minimize this function with respect to b0 and b1

This is a classic optimization problem.

We may remember from high school algebra that to find the

minimum value we should get the derivative and set it equal to
zero.
 The Least Square Method

Note : Our unknowns are b0 and b1 .

xi and yi are known. They are our data

yi xi yi
y1 x1 b0  b1 x 1
y2 x2 b0  b1 x 2
y3 x3 b0  b1 x 3
. . .
. . .
yn xn b0  b1 x n

n
Z   (y i  b0  b1 x i ) 2
i 1

Find the derivative of Z with respect to b0 and b1 and set them

equal to zero
Derivatives
n
Z   ( y i  b0  b1 x i ) 2

i 1

Z n
  2( 1 )( y i  b0  b1 x i )  0
b0 i  1

Z n
  2(  x i )( y i  b0  b1 x i )  0
b1 i  1
B0 and b1

b1 
 xy  (  x  y ) / n
 x ( x ) / n
2 2

b0  y  b1 x
 Example

Restaurant i xi yi xi yi xi2
1 2 58 116
2 6 105 630 4
3 8 88 704 36
4 8 118 944 64
5 12 117 1404 64
6 16 137 2192 144
7 20 157 3140 256
8 20 169 3380 400
9 22 149 3278 400
10 26 20 5252 484
676

Total 140 1300 21040 2528

b1

b1 
 xy  (  x  y ) / n
 x ( x ) / n
2 2

21 ,040  ( 140 )( 1300 ) / 10

b1  2
2528  ( 140 ) / 10

2840
b1  5
568
b0

y  b0  b1 x
1300
y  130
10
140
x  14
10
130  b0  5 ( 14 )

b0  60
Estimated Regression Equation

y  60  5 x

Now we can predict.

For example, if one of restaurants of this Pizza Chain is close to
a campus with 16,000 students.
We predict the mean of its quarterly sales is

y  60  5 ( 16 )
y  140 thousand dollars
Summary ; The Simple Linear Regression Model

• Simple Linear Regression Model

y =  0 + 1 x + 

• Simple Linear Regression Equation

E(y) = 0 + 1x

• Estimated Simple Linear Regression Equation

ŷ = b0 + b1x
Summary ; The Least Square Method

• Least Squares Criterion

min S(yi - ŷi)2

where
yi = observed value of the dependent variable
for the i th observation
ŷi = estimated value of the dependent variable
for the i th observation
Summary ; The Least Square Method

• Slope for the Estimated Regression Equation

b1 
 x y ( x  y )/ n
i i i i

 x ( x ) / n
2
i i
2

• y -Intercept for the Estimated Regression Equation

b0  y  b1 x

xi = value of independent variable for i th observation

yi = value of dependent variable for i th observation
_
x = mean value for independent variable
_
y = mean value for dependent variable
n = total number of observations
Example : Read Auto Sales

Reed Auto periodically has a special week-long sale.

As part of the advertising campaign Reed runs one or
more television commercials during the weekend
preceding the sale. Data from a sample of 5 previous
sales showing the number of TV ads run and the
number of cars sold in each sale are shown below.
Number of TV Ads Number of Cars Sold
1 14
3 24
2 18
1 17
3 27
Example : Read Auto Sales

30

25

20

15 X

10

0
0 0.5 1 1.5 2 2.5 3 3.5
Example : Read Auto Sales
Y X
14 1
24 3
18 2
17 1
27 3

Y X XY X2
14 1 14 1
24 3 72 9
18 2 36 4
17 1 17 1
27 3 81 9
 Example : Read Auto Sales

Y X XY X2
14 1 14 1
24 3 72 9
18 2 36 4
17 1 17 1
27 3 81 9
Sum 100 10 220 24
Example : Read Auto Sales

• Slope for the Estimated Regression Equation

b1 = 220 - (10)(100)/5 = 5
24 - (10)2/5
• y -Intercept for the Estimated Regression Equation
b0 = 20 - 5(2) = 10
• Estimated Regression Equation
ŷ = 10 + 5x
Practice
Pages 562
Problems 1 (d,e)
Problems 3
Problems 4 (b,c)
 Coefficient of Determination
Question : How well does the estimated regression line fits the
data.

Coefficient of determination is a measure for Goodness of Fit.

Goodness of Fit of the estimated regression line to the data.

Given an observation with values of yi and xi.

^
We put xi in the equation and get yi
.
ŷi = b0 + b1xi

(yi – ŷi) is called residual.

It is the error in using ŷi to estimate yi.

SSE =  (y - ŷ )2
SSE : Pictorial Representation

y10 - ^y10

Y = 60+5x
 SSE Computations
i xi yi
1 2 58
2 6 105
3 8 88
4 8 188
5 12 117
6 16 137
7 20 157
8 20 169
9 22 149
10 26 202
 SSE Computations
i xi yi ŷi = 60 + 5xi
1 2 58 70
2 6 105 90
3 8 88 100
4 8 188 100
5 12 117 120
6 16 137 140
7 20 157 160
8 20 169 160
9 22 149 170
10 26 202 190
 SSE Computations
i xi yi ŷi = 60 + 5xi (yi - ŷi ) (yi- ŷi )2
1 2 58 70 -12 144
2 6 105 90 15 225
3 8 88 100 -12 144
4 8 188 100 18 324
5 12 117 120 -3 9
6 16 137 140 -3 9
7 20 157 160 -3 9
8 20 169 160 9 81
9 22 149 170 -21 441
10 26 202 190 12 144
 SSE Computations
i xi yi ŷi = 60 + 5xi (yi - ŷi ) (yi- ŷi )2
1 2 58 70 -12 144
2 6 105 90 15 225
3 8 88 100 -12 144
4 8 118 100 18 324
5 12 117 120 -3 9
6 16 137 140 -3 9
7 20 157 160 -3 9
8 20 169 160 9 81
9 22 149 170 -21 441
10 26 202 190 12 144

Total SSE = 1530

SSE = 1530 measures the error in using estimated equation to

predict sales
 SST Computations
Now suppose we want to estimate sales without using the level of
advertising. In other words, we want to estimate y without using x.

If y does not depend on x, then b1 = 0.

Therefore y = b0 + b1x ===> b0 = y
Here we do not take x into account, we simply use the average of
y as our sales forecast.
 SST Computations
Now suppose we want to estimate sales without using the level of
advertising. In other words, we want to estimate y without using x.

If y does not depend on x, then b1 = 0.

Therefore y = b0 + b1x ===> b0 = y
Here we do not take x into account, we simply use the average of
y as our sales forecast.

y = ( yi) / n
y = 1300/10 =
130
This is our estimate for the next value of y.
Given an observation with values of yi and xi.
(yi –y ) is the error in using x to estimate yi.

SST =  (y - y )2
SST : Pictorial Representation

y10 - y

yi = 130
 SST Computations
i xi yi (yi - ) (yi - y)2 y
1 2 58 -72 5184
2 6 105 -25 625
3 8 88 -42 1764
4 8 188 -12 144
5 12 117 -13 169
6 16 137 7 49
7 20 157 27 729
8 20 169 39 1521
9 22 149 19 361
10 26 202 72 5184

Total SST = 15730

SST = 15730 measures the error in using mean of y values to

predict sales
SSE , SST and SSR

SST : A measure of how well the observations cluster around y

SSE : A measure of how well the observations cluster around ŷ

If x did not play any role in vale of y then we should

SST = SSE

If x plays the full role in vale of y then

SSE = 0

SST = SSE + SSR

SSR : Sum of the squares due to regression

SSR is explained portion of SST

SSE is unexplained portion of SST
Coefficient of Determination for Goodness of Fit

SSE = SST - SSR

The largest value for SSE is

SSE = SST

SSE = SST =======> SSR = 0

SSR/SST = 0 =====> the worst fit

SSR/SST = 1 =====> the best fit

Coefficient of Determination for Pizza example

In the Pizza example,

SST = 15730
SSE = 1530
SSR = 15730 - 1530 = 14200

r2 = SSR/SST : Coefficient of Determination

1  r2  0

r2 = 14200/15730 = .9027

In other words, 90% of variations in y can be explained by the

regression line.
SST Calculations

SST   ( y  y ) 2

SST   y 2  (  y ) 2 / n
SST Calculations

SST   y 2  (  y ) 2 / n
Observation Xi yi yi^2
1 2 58 3364
2 6 105 11025
3 8 88 7744
4 8 118 13924
5 12 117 13689
6 16 137 18769
7 20 157 24649
8 20 169 28561
9 22 149 22201
10 26 202 40804
1300 184730
SST  184730  ( 1300 ) 2 / 10  15730
SSR Calculations

SSR 
 xy  (  x  y ) / n
2

 x   x  / n
2 2

Observation X Y XY Y^2 X^2

1 2 58 116 3364 4
2 6 105 630 11025 36
3 8 88 704 7744 64
4 8 118 944 13924 64
5 12 117 1404 13689 144
6 16 137 2192 18769 256
7 20 157 3140 24649 400
8 20 169 3380 28561 400
9 22 149 3278 22201 484
10 26 202 5252 40804 676
10 140 1300 21040 184730 2528
[ 21040  ( 140 )( 1300 ) / 10 ] 2
SSR  2
 14200
2528  ( 140 ) / 10
SSR Calculations

r 2  SSR / SST  14200 / 15730

r 2  .9027

SSE  SST  SSR

SSE  15730  14200  1530

Example : Read Auto Sales

Reed Auto periodically has a special week-long sale.

As part of the advertising campaign Reed runs one or
more television commercials during the weekend
preceding the sale. Data from a sample of 5 previous
sales showing the number of TV ads run and the
number of cars sold in each sale are shown below.
Number of TV Ads Number of Cars Sold
1 14
3 24
2 18
1 17
3 27
Example : Read Auto Sales

SSR 
 xy  (  x  y ) / n
2

SST   y 2  (  y ) 2 / n
 x   x  / n
2 2

We need to calculate x, y, xy , x2, y2

x y xy x2 y2
1 14 14 1 196
3 24 72 9 576
2 18 36 4 324
1 17 17 1 289
3 27 81 9 729

10 100 220 24 2114

Example : Read Auto Sales

SSR 
 xy  (  x  y ) / n
2

SST   y 2  (  y ) 2 / n
 x   x  / n
2 2

 x = 10
 y = 100
 xy = 220
 x2 = 24
 y2 = 2114

SST  2114  ( 100 ) / 5 2

SSR 
220  ( 10 )( 100 ) / 5 
2

24  10  / 5
2
SST  114
SSR 
220  200 
2
 100
24  20
 Example : Read Auto Sales
Alternatively; we could compute SSE and SST and then find
SSR = SST -SSE

y x y^2 yhat=10+5x y-yhat (y-yhat)^2

14 1 196 15 -1 1
24 3 576 25 -1 1
18 2 324 20 -2 4
17 1 289 15 2 4
27 3 729 25 2 4
Sy Sx Sy^2 SSE
100 10 2114 14
SST = Sy^2-[(Sy)^2]/n
114
SSR = SST - SSE
100 R2=100/114
0.877193
Example : Read Auto Sales
• Coefficient of Determination
r 2 = SSR/SST = 100/114 = .88

The regression relationship is very strong since

88% of the variation in number of cars sold can be
explained by the linear relationship between the
number of TV ads and the number of cars sold.
 The Correlation Coefficient
Correlation Coefficient = Sign of b1 times Square Root of the
Coefficient of Determination)

rxy  ( sign of b1 ) r 2

Correlation coefficient is a measure of the strength of a linear

association between two variables. It has a value between -1
and +1
Rxy = +1 : two variables are perfectly related through a line with
positive slope.

Rxy = -1 : two variables are perfectly related through a line with

negative slope.

Rxy = 0 : two variables are not linearly related.

 The Correlation Coefficient : example
IN our Pizza example, r2 = .9027 and sign of b1 is positive

rxy  ( sign of b1 ) r 2

rxy   .9027

rxy  .9501

There is a strong positive relationship between x and y.

Correlation Coefficient and Coefficient of Determination
Coefficient of Determination and Correlation Coefficient are both
measures of associations between variables.

Correlation Coefficient for linear relationship between two

variables.

Coefficient of Determination for linear and nonlinear

relationships between two and more variables.
Practice

Page 573
Problems 15
Problems 17
Problems 18