Topic 11.

Simple Linear Regression

Any questions?

During the
Lecture
E6-2004
Skype: L2ahmadi
Office Ext. 37160
L2ahmadi @uw...
hour
After the
Lecture

In Person Message Call

1
Course Topics
1. Role of Statistics in Engineering
2. Probability
3. Treatment of Data
4. Discrete Probability Distributions
5. Continuous Probability Distributions
6. Point Estimation of Parameters
7. Statistical Intervals for a Single Sample
8. Tests of Hypothesis for a Single Sample
9. Statistical Intervals for Two Samples
10. Topic 6 to 9 in one Picture
11. Simple Linear Regression (Sections 11.1 - 11.3)
12. Multiple Linear Regression
2
13. Process Quality Improvement (If time permits)
Topic Outline
Part 1:
11-1. Empirical Models
11-2. Simple Linear Regression
11-3. Properties of the Least Squares Estimators
11-4. Minitab
Part 2:
11-5. Hypothesis Tests in Simple Linear Regression
11-6. Confidence Intervals
11-7. Prediction of New Observations
11-8. Adequacy of the Regression Model (by R2)

3
Learning Objectives

4
 Previous Topic
General Procedure for Hypothesis Tests:
1. From the problem context, identify the
parameter of interest.
2. State the null hypothesis, H0 .
3. Specify an appropriate alternative
hypothesis, H1. +

4. Choose a significance level, .

5. Determine an appropriate test statistic.
6. State the rejection region for the statistic.
7. Compute any necessary sample quantities,
−∆◦
substitute these into the equation for
the test statistic, and compute that value.
8. Decide whether or not H0 should be rejected
and report that in the problem context.
Reject the null hypothesis
Conclusion Statement:
Fail to reject the null hypothesis 5
 PART 1
11-1. Empirical Models
11-2. Simple Linear Regression
11-3. Properties of the Least Squares Estimators
11-4. Minitab

6
 11-1 Empirical Models
• Many problems in engineering and science involve
exploring the relationships between two or more
variables.
• Regression analysis is a statistical technique that is
very useful for these types of problems.
• For example, in a chemical process, suppose that the
yield of the product is related to the process-operating
temperature.
• Regression analysis can be used to build a model to
predict yield at a given temperature level. 7
11-1 Empirical Models

8
11-1 Empirical Models

Figure 11-1 Scatter Diagram of oxygen purity versus 9

hydrocarbon level from Table 11-1.
11-1 Empirical Models
Based on the scatter diagram, it is probably reasonable to
assume that the mean of the random variable Y is related to x
by the following straight-line relationship:

where the slope and intercept of the line are called regression
coefficients.
The simple linear regression model is given by

where  is the random error term.

10
11-1 Empirical Models
We think of the regression model as an empirical model.
Suppose that the mean and variance of  are 0 and 2,
respectively, then

The variance of Y given x is

11
11-1 Empirical Models
• The true regression model is a line of mean values:

where 1 can be interpreted as the change in the

mean of Y for a unit change in x.
• Also, the variability of Y at a particular value of x is
determined by the error variance, 2.
• This implies there is a distribution of Y-values at
each x and that the variance of this distribution is the
same at each x.
12
11-1 Empirical Models

Figure 11-2 The distribution of Y for a given value of

x for the oxygen purity-hydrocarbon data.
13
11-2 Simple Linear Regression

• The case of simple linear regression considers

a single regressor or predictor x and a
dependent or response variable Y.
• The expected value of Y at each level of x is a
random variable:

• We assume that each observation, Y, can be

described by the model

14
11-2 Simple Linear Regression

• Suppose that we have n pairs of observations

(x1, y1), (x2, y2), …, (xn, yn).

Figure 11-3
Deviations of the
data from the
estimated
regression model.
15
11-2 Simple Linear Regression
• The method of least squares is used to
estimate the parameters, 0 and 1 by minimizing
the sum of the squares of the vertical deviations in
Figure 11-3.

Figure 11-3
Deviations of the
data from the
estimated
regression model.
16
11-2 Simple Linear Regression
• Using Equation 11-2, the n observations in the
sample can be expressed as

• The sum of the squares of the deviations of the

observations from the true regression line is

17
11-2 Simple Linear Regression

18
11-2 Simple Linear Regression

19
11-2 Simple Linear Regression

Definition

20
11-2 Simple Linear Regression

21
11-2 Simple Linear Regression

Notation

22
11-2 Simple Linear Regression
Example 11-1

23
11-2 Simple Linear Regression
Example 11-1

24
 11-2 Simple Linear Regression
Example 11-1

Figure 11-4 Scatter

plot of oxygen
purity y versus
hydrocarbon level x
and regression
model ŷ = 74.20 +
14.97x.
25
11-2 Simple Linear Regression
Estimating 2
The error sum of squares is

It can be shown that the expected value of the

error sum of squares is E(SSE) = (n – 2)2.

26
11-2 Simple Linear Regression
Estimating 2
An unbiased estimator of 2 is

where SSE can be easily computed using

27
11-3 Properties of the Least Squares
Estimators

• Slope Properties

• Intercept Properties

28
11-4 Minitab Output

29
30
STEP I: BUILD THE MODEL

Fitted or estimated regression line:

STEP II: GOOD ENOUGH? Each pair of observation

satisfies the relation:

Residual:

Sum of Squares of Error:

The Sum of Squares Regression:

SSR is the sum of the squared differences between

the prediction for each observation and the population mean.
Total Sum of Squares:

31
The Coefficient of Determination:
 PART 2
11-5. Hypothesis Tests in Simple Linear
Regression
11-6. Confidence Intervals
11-7. Prediction of New Observations
11-8. Adequacy of the Regression Model (by R2)

32
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.1 Use of t-Tests
Suppose we wish to test

An appropriate test statistic would be

33
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.1 Use of t-Tests
The test statistic could also be written as:

We would reject the null hypothesis if

34
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.1 Use of t-Tests
Suppose we wish to test

An appropriate test statistic would be

35
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.1 Use of t-Tests

We would reject the null hypothesis if

36
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.1 Use of t-Tests
An important special case of the hypotheses of
Equation 11-18 is

These hypotheses relate to the significance of regression.

Failure to reject H0 is equivalent to concluding that there
is no linear relationship between x and Y.
37
11-5 Hypothesis Tests in Simple Linear
Regression

Figure 11-5 The hypothesis H0: 1 = 0 is not rejected.

38
11-5 Hypothesis Tests in Simple Linear
Regression

Figure 11-6 The hypothesis H0: 1 = 0 is rejected.

39
11-5 Hypothesis Tests in Simple Linear
Regression
Example 11-2 (Tutorial)

40
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.2 ANalysis Of VAriance Approach to Test
Significance of Regression

The ANalysis Of VAriance identity is

Symbolically,

41
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.2 Analysis of Variance Approach to Test
Significance of Regression

If the null hypothesis, H0: 1 = 0 is true, the statistic

follows the F1,n-2 distribution and we would reject if

f0 > f,1,n-2.
* The quantities, MSR and MSE are
called mean squares. 42
11-5 Hypothesis Tests in Simple Linear
Regression
11-5.2 ANalysis Of VAriance Approach to Test
Significance of Regression

ANalysis Of VAriance (ANOVA) Table:

43
ANOVA TABLE

44
45
11-5 Hypothesis Tests in Simple Linear
Regression
Example 11-3 (Tutorial)

46
11-5 Hypothesis Tests in Simple Linear
Regression

47
Previous Lecture
- ANOVA Table

If the null hypothesis, H0: 1 = 0 is true, the statistic

follows the F1,n-2 distribution and we would reject if
f0 > f,1,n-2.

48
11-6 Confidence Intervals

11-6.1 Confidence Intervals on the Slope and Intercept

Definition

49
11-6 Confidence Intervals

Example 11-4 (Tutorial)

50
11-6 Confidence Intervals

11-6.2 Confidence Interval on the Mean Response

Definition

51
11-6 Confidence Intervals

Example 11-5

52
11-6 Confidence Intervals

Example 11-5

53
11-6 Confidence Intervals

Example 11-5

54
11-6 Confidence Intervals

Example 11-5

Figure 11-7
Scatter diagram of
oxygen purity data
from Example 11-1
with fitted
regression line and
95 percent
confidence limits
on Y|x0.
55
11-7 Prediction of New Observations

If x0 is the value of the regressor variable of interest,

is the point estimator of the new or future value of the

response, Y0.

56
11-7 Prediction of New Observations

Definition

57
11-7 Prediction of New Observations

Example 11-6

58
11-7 Prediction of New Observations

Example 11-6

59
11-7 Prediction of New Observations

Example 11-6

Figure 11-8 Scatter

diagram of oxygen
purity data from
Example 11-1 with
fitted regression line,
95% prediction limits
(outer lines) , and
95% confidence
limits on Y|x0.

60
11-8 Adequacy of the Regression Model

• Fitting a regression model requires several

assumptions.
1. Errors are uncorrelated random variables with
mean zero;
2. Errors have constant variance; and,
3. Errors be normally distributed.
• The analyst should always consider the validity of
these assumptions to be doubtful and conduct
analyses to examine the adequacy of the model
61
11-8 Adequacy of the Regression Model

11-8.1 Residual Analysis

• The residuals from a regression model are ei = yi - ŷi , where yi

is an actual observation and ŷi is the corresponding fitted value
from the regression model.
• Analysis of the residuals is frequently helpful in checking the
assumption that the errors are approximately normally distributed
with constant variance, and in determining whether additional
terms in the model would be useful.

62
11-8 Adequacy of the Regression Model

11-8.1 Residual Analysis

Figure 11-9 Patterns
for residual plots. (a)
satisfactory, (b)
funnel, (c) double
bow, (d) nonlinear.
[Adapted from
Montgomery, Peck,
and Vining (2001).]

63
11-8 Adequacy of the Regression Model

Example 11-7

64
11-8 Adequacy of the Regression Model

Example 11-7

65
11-8 Adequacy of the Regression Model

Example 11-7

Figure 11-10 Normal

probability plot of
residuals, Example
11-7.

66
11-8 Adequacy of the Regression Model

Example 11-7

Figure 11-11 Plot of

residuals versus
predicted oxygen
purity, ŷ, Example
11-7.

67
11-8 Adequacy of the Regression Model
11-8.2 Coefficient of Determination (R2)

• The quantity

is called the coefficient of determination and is often

used to judge the adequacy of a regression model.
• 0  R2  1;
• We often refer (loosely) to R2 as the amount of
variability in the data explained or accounted for by the
regression model. 68
11-8 Adequacy of the Regression Model
11-8.2 Coefficient of Determination (R2)

• For the oxygen purity regression model,

R2 = SSR/SST
= 152.13/173.38
= 0.877
• Thus, the model accounts for 87.7% of the
variability in the data.

69
Concept Map

Next:
Ho
H1 Simple
R2
. CI
r
Multiple
Linear
ANOVA Regression

70