Regression: Data Analysis

Quick survey
Case Studies:
1. Office Trip Study
2. Does an Increasing Crime Rate
Decrease House Prices?
3. Analysis of Car Mileage Data

Motivating Examples
Suppose we have data on sales of houses in some area.
For each house, we have complete information about its size, the number of
bedrooms, bathrooms, total rooms, the size of the lot, the corresponding
property tax, etc., and also the price at which the house was eventually sold.
Can we use this data to predict the selling price of a house currently on the
market?
The first step is to postulate a model of how the various features of a house
determine its selling price.
A linear model would have the following form:
selling price = 0 + 1(sq.ft.) + 2 (no. bedrooms) + 3 (no. bath)
+ 4 (no. acres) + 5 (taxes) + error
In this expression, 1 represents the increase in selling price for each additional
square foot of area: it is the marginal cost of additional area.
2 and 3 are the marginal costs of additional bedrooms and bathrooms, and so on.
The intercept 0 could in theory be thought of as the price of a house for which all
the variables specified are zero; of course, no such house could exist, but including
0 gives us more flexibility in picking a model.

Sales of Houses
The error reflects the fact that two houses with exactly the same characteristics
need not sell for exactly the same price.
There is always some variability left over, even after we specify the value of a large
number variables.
This variability is captured by an error term, which we will treat as a random
variable.

Regression analysis is a technique for using data to identify

relationships among variables and use these relationships to make
predictions.

Growth of the economy

Consider a simplified version of economic forecasts using
regression models.
Consider the problem of predicting growth of the economy in the
next quarter.
Some relevant factors might be last quarter's growth, this quarter's growth, the
index of leading economic indicators, total factory orders this quarter,
aggregate wholesale inventory levels, etc.
A linear model for predicting growth would then take the following form:

next qtr growth = 0 + 1(last qtr growth) + 2(this qtr growth)

+ 3(index value) + 4(factory orders)
+ 5(inventory levels) + error
Estimate 0 and the coefficients 1,, 5 from historical data, in
order to make predictions.

Levels of advertising
Determine appropriate levels of advertising and promotion for a
particular market segment.
Consider the problem of managing sales of beer at large college
campuses.
Sales over, say, one semester might be influenced by ads in the college paper,
ads on the campus radio station, sponsorship of sports-related events,
sponsorship of contests, etc.

Use data on advertising and promotional expenditures at many

different campuses to tell us the marginal value of dollars spent in
each category.
A marketing strategy is designed accordingly.
Set up a model of the following type:
sales = 0 + 1(print budget) + 2(radio budget)
+ 3(sports promo budget) + 4(other promo) + error

Simple Linear Regression

Observe the data recorded as the pairs (Xi, Yi), i = 1,,n.
Xi is called independent variables or explanatory variables
X is used to explain part of the variability in the dependent variable Y .

Look at some scatterplots of samples with varying degrees of

correlation.

Purpose of Modeling
Prediction: The fitted regression line is a prediction rule!
The regression equation is Price = 38.9 + 35.4 Size

What is the definition of a Prediction Rule?

Put in a value of X for a Y we havent yet seen, and out comes a prediction (a
function or black box).
f(X)= 0 + 1X
You give me any new value of X and I can predict Y.

Forecast Accuracy
Our forecast is not going to be right on the money every time and
we need to develop the notion of forecast accuracy.
Two things we want:
What kind of Y can we expect for a given value of X?
How sure are we about this forecast?
How different could y be from what we expect?

Goal: Provide a measure of the accuracy of forecasts or

equivalently how much uncertainty is there in our forecasts.
Proposal: Provide a range of possible Y values that are likely given
this x value.

Prediction Interval
Prediction Interval: range of possible Y values that are likely given
X
What influences the length of the prediction interval?
Intuitively, the answer must lie in observed variation of the data points about
the prediction rule or fitted line.

Key Insight: To construct a prediction interval, we have to assess

the the likely range of residual values which will occur for an as
yet unobserved Y value!
How can we achieve this?
Develop a probability model for distribution of these residuals values.
If the residuals were normally distributed with a given standard deviation,
then we could make formal probability statements about the range of likely
residuals!!
With 95% probability, the residuals will be between -$28,000 and $28,000.

Simple Linear Regression Model

Once we come to the view that the residuals might come from a
probability distribution, we must also acknowledge that the fitted
line might be fooled by the particular realizations of the residuals.
The model will enable us to think about uncertainty and which uses
a particular distribution for the deviations from the line.
The power of statistical inference comes from our ability to make
very precise probability statements about the accuracy of estimates
and forecasts.
There is no free lunch, in order to make these statements and to understand the
output from the regression procedure, we must invest in a probability model.

Simple Linear Regression Model

Y = 0 +1X +, ~ N(0,2)
Part of Y related to X (What we can expect): 0 +1X
Part of Y independent of X (How different can Y be):
Note that is a random variable which is called the error term.
This can be thought of as a sort of trash can which contains all of the
omitted influences on the Y variable. As an example, it represents the other
omitted factors which change the price of the house other than the house
size.
E()=0.
standard deviation: The size of is measured by [Var( )]1/2 .

The systematic part is given by the term 0 +1X.

The conditional expectation of Y given X, E(Y|X), is0 +1X.

Linear Regression
A regression model species a relation between a dependent variable
Y and certain independent variables X1,,XK.
A simple linear regression refers to a model with just one
independent variable, K=1.
Y = 0 +1X +
Independent variables are also called explanatory variables; in the equation
above, we say that X explains part of the variability in the dependent variable
Y.

Example: A large corporation is concerned about maintaining

parity in salary levels across different divisions.
As a rough guide, it determines that managers responsible for comparable
budgets in different divisions should have comparable compensation.
Data Summary

Example
The following is a list of salary levels ($1000s) for 20 managers and
the sizes of the budgets ($100,000s) they manage: (59.0,3.5),
(67.4,5.0), (50.4,2.5), (83.2,6.0), (105.6, 7.5), (86.0,4.5), (74.4,6.0),
(52.2,4.0), (59.0,3.5), (67.4,5.0), (50.4,2.5), (83.2,6.0), (105.6,7.5),
(86.0,4.5), (74.4,6.0), (52.2, 4.0)

Best Line
Want to fit a straight line to this data.
The slope of this line gives the marginal increase in salary with respect to
increase in budget responsibility.

We need to define what we mean by the best line.

Regression uses the least squares criterion, which we now explain.
Any line we might come up with has a corresponding intercept b0 and a slope
b1.
This line may go through some of the data points, but it typically does not go
through all of them.

The least squares criterion chooses b0 and b1 to minimize the sum of

squared errors 1in (yi - b0 - b1xi)2 where n is the number of data
points.

Least Squares
For the budget level Xi, the least squares line predicts the salary
level
SALARY = 31.9 + 7.73 BUDGET or PYi = 31.9 + 7.73Xi
Unless the line happens to go through the point (Xi; Yi), the predicted value
PYi will generally be different from the observed value Yi.
Each additional $100,000 of budget responsibility translates to an expected
additional salary of $7,730.
The average salary corresponding to a budget of 6.0, we get a salary of 31:9 +
7:73(6:0) = 78:28.
The difference between the two is the error or residual ei = Yi - PYi.

The least squares criterion chooses b0 and b1 to minimize the sum of

squared errors 1in ei2 .
A consequence of this criterion is that the estimated regression line always
( X ,.Y )
goes through the point

Questions
Q1: Why is the least squares criterion the correct principle to
follow?
Q2: How do we evaluate and use the regression line?
Assumptions Underlying Least Squares

The errors 1,, n are independent of the values of X1,,Xn.

The errors have expected value zero; i.e., E[ i] = 0.
All the errors have the same variance: Var[ i] = 2, for all i = 1,,n.
The errors are uncorrelated; i.e., Corr[i, j] = 0 if i j.

Q1: What are the angle between (1,,1) and (1,, n) and that of
(1,, n) and (X1,,Xn)?

Discussion on Assumptions
The first two are very reasonable: if the ei's are indeed
random errors, then there is no reason to expect them to
depend on the data or to have a nonzero mean.
The second two assumptions are less automatic.
Do we necessarily believe that the variability in salary levels
among managers with large budgets is the same as the variability
among managers with small budgets? Is the variability in price
really the same among large houses and small houses?
These considerations suggest that the third assumption may not
be valid if we look at too broad a range of data values.
Correlation of errors becomes an issue when we use regression
to do forecasting. If we use data from several past periods to
forecast future results, we may introduce correlation by
overlapping several periods and this would violate the fourth
assumption.

Linear Regression
We assume that the outcome we are predicting depends
linearly on the information used to make the prediction.
Linear dependence means constant rate of increase of one
variable with respect to another (as opposed to, e.g., diminishing
returns).
E(Y|X) is the population average value of Y for any given
value of X. For example, the average house price for a house
size = 1,000 sq ft.

Regression models are really all about modeling the

conditional distribution of Y given X.
Distribution of House Price given Size
Distribution of Portfolio return given return on market
Distribution of wages given IQ or educational attainment
Distribution of sales given price

Evaluating the Estimated Regression Line

Feed data into the computer and get back estimates of the model
parameters 0 and 1.
Is this estimated line any good?
Does it accurately reflect the relation between the X and Y variables?
Is it a reliable guide in predicting new Y values corresponding to new X
values?
predicting the selling price of a house that just came on the market, or
setting the salary for a newly defined position

Intuitively, the estimated regression line is useful if the

points (Xi,Yi) are pretty well lined up. The more they look
like a cloud of dots, the less informative the regression
will be.

Reduction of Variability

Our goal is to determine how much of the

variability in Y values is explained by the X values.
We measure variability using sums of squared
quantities.
Consider the salary example. The Yi's (the salary levels)
exhibit considerable variability |-not all managers have
the same salary.
We conduct the regression analysis to determine to
what extent salary is tied to responsibility as measured
by budget: the 20 managers have different budgets as
well as different salaries.

What extent the differences in salaries are

explained by differences in budgets?

Analysis of Variance Table

s = 12.14, R2 = 72.2%, R2 (adj) = 70.7%
SOURCE DF
SS
MS
Regression 1
6884.7 6884.7 . . .
Error
18 2651.1
147.3
Total
19 9535.8
DF stands for degrees of freedom, SS for sum of squares, and MS for
mean square. The mean squares are just the sum of squares divided by
the degrees of freedom: MS = SS/DF.

A sum of squares measures variability.

The Total SS (9535.8) measures the total variability in the salary levels.
The Regression SS (6884.7) is the explained variation. It measures how
much variability is explained by differences in budgets.
The Error SS (2651.1) is the unexplained variation.

The Error SS
This reflects differences in salary levels that cannot be attributed
to differences in budget responsibilities.
The explained and unexplained variation sum to the Total SS.
How much of the original variability has been explained?
The answer is given by the ratio of the explained variation to the total
variation, which is
R2 =SSR/SST=6884.7/9538.8= 72.2%
This quantity is the coefficient of determination, though everybody calls it
R-squared.

Normal Distribution
The following figure depicts two different normal distributions
both with mean 0 one with =.5 one with =2.
one : 68%, two : 95.44%, two : 99.7%

How does determine the dispersion of points about

the true regression line?
small/small

large/large

Assessed Property Values

A data set which is consist of 228 assessed property values along
with some information about the houses in question.
VALUE

LO
C

LOTSZ

BED RM

BATH

ROOMS

AGE

GARG

EMEADW

LVTTWN

190.00

6.90

38

215.00

6.00

2.0

30

160.00

600

2.0

35

195.00

6.00

2.0

35

163.00

7.00

1.0

39

159.90

6.00

1.0

38

160.00

6.00

1.0

35

195.00

6.00

2.0

38

165.00

9.00

1.0

32

180.00

11.20

1.0

32

181.00

6.00

2.0

10

35

Description of this Data Set

The first column gives the assessed value in thousands of
dollars. The second encodes the location: 1 for East Meadow,
3 for Levittown, and 4 for Islip. The next gives lot size in
thousands of square feet, then bedrooms, bathrooms, total
rooms, age, and number of garage units.
The last two columns encode location in dummy variables.
We discuss these in more detail later. For now, just note that a
1 under EMEADW indicates a house in East Meadow, a 1
under LVTTWN indicates a house in Levittown, and 0's in
both columns indicate a house in Islip.
Our goal is to use this data to predict assessed values from
characteristics of a house. The best model need not include
all variables.

Conditional Distributions vs. Marginal Distributions

The conditional distribution is obtained by slicing the
point cloud in the scatter diagram to obtain the distribution
of Y conditional on various ranges of X values.
Try this with a lot of housing data.

Plot the conditional distributions for each of these slices

Boxplot: Median is center line. "box" goes from 25 to75th
percentile. "whiskers" from 1 to 99th.

Marginal distribution of price

Key Observations from these Plots:

Conditional distributions give us the answer to forecasting
problem if I know that a house is between 1 and 1.5 K , then
the conditional distribution (depicted in the second boxplot)
gives me a point forecast (the mean) and any prediction
interval I want.
The conditional means (medians) seem to line up along the
regression line.
Do we always have E(Y|X) = 0 +1X?
The conditional distributions have much smaller dispersion
than the marginal distribution.

Suggestion
If X has no forecasting power, then the marginal and conditionals
will be the same.
If X has some forecasting information or power, then
the conditional means will be different than the marginal or overall mean and
the conditional standard deviations will be less than the marginal variances.

Compare the marginal and conditional moments.

Class Interval Marg [1, 1.5] [1.5, 2] [2, 2.5] [2.5, 3] [3,
3.5]
Mean
116
84
102
119
135
154
Standard Dev
27
9.8
12.9
10.0
10.2
10.9

Confirm our intuition by looking at the situation where there

is little or no relationship between X and Y.
Suppose we had collected information on the number of
stop signs with a two block radius of each house.
Look at the relationship between this variable and Y.

Compare marginal and conditionals

Not much different!

Multiple Regression
A regression model specifies a relation between a dependent
variable Y and certain independent variables X1, ,XK.
Here independence is not in the sense of random variables; rather, it
means that the value of Y depends on - or is determined by - the Xi
variables.)

A linear model sets

Y = 1 + 1X1 + + kXK + ,
where is the error term.
To use such a model, we need to have data on values of Y
corresponding to values of the Xi's.
selling prices for various house features, past growth values for various
economic conditions
beer sales corresponding to various marketing strategies


Sir Francis Galton (1822-1911)
Galton mark: ,

?
:
?

:

?
:
:


Regression ( ) to the mean

( )

Regression ( ) to the mean

Office Trip Study

Traffic Planners often refer to results from a classic study done in
1989 for the Maryland Planning Commission by Douglas &
Douglas Inc.
The study was done in Montgomery County, MD.
Goal: Predict the volume of traffic associated with office buildings.
Such information is useful for several purposes.

For example if a new office building of 350,000 sq. ft. were being planned,
planners and zoning administrators, etc., would need to know how much
additional traffic to expect after the building was completed and occupied .

Data AM: traffic counts over a period of time at 61 office building

sites within the county.
X-variable: size of the building measured in occupied gross square feet of
floor space (in 1000 sq. ft. units).
Y-variable: average daily number of vehicle trips originating at or near the
building site during the AM hours.

Data PM: Similar data for PM hours was also measured, and some
other information about the size, occupancy, location, etc., of the
building was also recorded.

Scatterplot: AM Trips By Occup. Sq. Ft. (1000)

Fit AM Trips = -7.939 + 1.697 Occup. Sq. Ft. (1000)
Summary of Fit R2 = 0.800

Residual Plot
How do you know that a correct model is being fitted?
Prediction: For a 350,000 sq. ft. bldg, it generates -7.939 + 1.697350 =
586.0 trips. The 95% confidence interval for this prediction is 535.8 to 636.1.

Noticeable
heteroscedasticity
by looking at scatter plot.

undesirable histogram of
residuals

Transformation Attempt #1
Since the (vertical) spread of the data increases as the y-value
increases a log transformation may cure this heteroscedasticity.
Scatter plot after transforming y to Log(y)
Residual Plot

Linear Fit: Log Trips = 1.958 + 0.00208 Occup. Sq. Ft. (1000)
Summary of Fit: R2 = 0.761

New analysis introduces a new problem!

An undesirable degree of non-linearity: It is evident in both the
residual plot and the scatterplot.
Transformation Attempt #2
Try to fix nonlinearity with an additional transformation from x to
log(x).
Residual Plot

Linear Fit: log Trips = 0.639 + 0.803 log(OccSqFt)

Summary of Fit: R2 = 0.827

Standard Assumptions
After log-log transformation: Linearity, homoscedasticity, and
normality of residuals are all quite OK.

Prediction
If a zoning board were in the process of considering the zoning application from a proposed 350,000 sq. ft. office bldg, what is their
primary concern?
Proposal I: Find the 95% confidence limits for 350,000 sq. ft. office buildings.
Lower 95% Pred Upper 95% Pred
Log (Trips)
2.6262
2.7399
Number of Trips:
422.9 = 102.6262
549.4
Compare this to the confidence interval of 535.8 to 636.1 from the initial model.
These CIs are very different. The latter one, based on the log-log analysis, is the valid
one, since the analysis leading to it is valid.

Proposal II: Consider 95% Individual Prediction intervals - that is, in 95%
intervals for the actual traffic that might accompany the particular proposed
building. These are
Lower 95% Indiv Upper 95% Indiv
Log (Trips)
2.3901
2.9760
Number of Trips:
245.5
946.2

Comparison of the two analyses on a single plot

Does an Increasing Crime Rate Decrease House Prices?

This data was gathered in 1996.
For each community in the Philadelphia area, it gives the crime rate (reported
crimes/1000 population/year) and the average sale price of a single family
home.
Center City Philadelphia is not included on the following plot and data
analyses.
House Price ($10,000) versus Crime Rate

Least squares straight line fit to this

data
Summary of Fit
R2 = 0.184
R2 Adj = 0.176
Root Mean Square
Error = 7.886
Observations=98

Linear Fit
Hs Prc ($10,000) = 22.53 0.229 Crime
Rate

(1) Linear fit with all data

(2) Linear fit with five points removed
Linear Fit number (2)
Hs Prc ($10,000) = 19.91 - 0.184 Crime
Rate

Linear Fit Number (1)

Hs Prc ($10,000) = 22.52 0.229Crime Rate
Summary of Fit
R2 =0.184, RMSE = 7.886

Analysis of Variance
Term
Estimate
Prob>|t|
Intercept
22.52
<.0001
Crime Rate
-0.229
<.0001

Std Error
1.649
0.0499

t Ratio
13.73
-4.66

Linear Fit Number (2)

Hs Prc ($10,000) = 19.91 0.184Crime Rate
Summary of Fit
R2 =0231, RMSE = 5.556

Analysis of Variance
Term
Estimate
Prob>|t|
Intercept
19.91
<.0001
Crime Rate
-0.184
<.0001

Std Error
1.193
0.0351

t Ratio
16.69
-5.23

Residuals Hs Prc
($10,000); Analysis 1

Normal Quantile Plot

Residuals Hs Prc
($10,000); Analysis 2

Normal Quantile Plot

Analysis of Car Mileage Data

Data set: It gives mileage figures (in MPG (City)) for various
makes of cars, along with various characteristics of the car engine
and body as well as the base price for the car model.
Objective:
Create a multiple regression equation that can be used to predict the MPG for
a car model having particular characteristics.
Get an idea as to which characteristics are most prominent in relation to a car
mileage.

Build a regression model:

Step 1: Examine the data
a. Look for outliers and influential points.
b. Decide whether to proceed by predicting Y or some function of Y.
Preliminary analysis: Consider X to be weight.
Linear Fit: MPG City = 40.266964 - 0.006599 Weight(lb)
Transformed Fit to Log: MPG City = 171.42143 - 18.898537 Log(Weight(lb))

R2 changes from 0.75093 to 0.803332.

RMSE changes from 2.136611 to 1.898592.

Log(Weight) version provides a somewhat better

fit.
A slight curve is evident in the residual plot (and
in the original scatter plot if you look carefully).
No outliers or influential points that seem to
warrant special attention.

Another Fit
Fit of MPG City By Horsepower
MPG City = 32.057274 - 0.0843973 Horsepower
MPG City = 76.485987 - 11.511504 Log(Horsepower)

--- Linear Fit

--- Transformed Fit to Log

This suggests that we might

want to try also transforming
the Y-variable somehow, in
order to remove the remaining
curved pattern.

Step 1b: Try transformations of Y

One transformation that has been suggested is to
transform Y to 1/Y = Gallons Per Mile.

Since this is a rather small decimal, we consider Y* = Gallons

Per 1000Miles.
Linear Fit: GP1000M City = 9.2452733 + 0.0136792
Weight(lb)
Another Fit: GP1000M City = 25.559124 + 0.1806558
Horsepower
R2 = 0.774344
RMSE = 4.151475

R2 = 0.705884
RMSE = 4.739567

Transformation
The first analysis looks nicely linear, but there is some
evident heteroscedasticity.
The second analysis seems to be slightly curved;
maybe we could try using log(HP) as a predictor.
It seems reasonable to also try transforming to Log(Y)
= Log10 (MPG).
Since MPG was nearly linear in Wt, it seems more reasonable
to try Log10 (Wt) as a predictor here, and similarly for Log10
(HP).

Log10(MPG) = 4.2147744
- 0.8388428
Log10(Wt)

Transformation
Linearity looks fine on these plots.
There may be a hint of heteroscedasticity - but not
close to enough to worry about.
Again, there are no leverage points or outliers that
seem to need special care.
Log10(MPG) = 2.3941295 - 0.5155074 Log10(HP)

Step 2: Choose predictor variables for the

Multiple Regression
Use the chosen form of Y variable and of X variables - and
perhaps other X-variables as well.
Multivariate Correlations
Log_10(MPG)
Log_10(HP)

Log_10(MPG) Log_10(HP) Log (Displ) Log_10(Wt) Log_10(Cargo) Log_10(Price) cylinders length

1.0000
-0.8791
-0.8624
-0.9102
-0.1361
-0.8208
-0.7572
-0.8791
1.0000
0.8530
0.8260
-0.0440
0.8335
0.7976

Log (Displ)
-0.8624
Log_10(Wt)
0.7483
0.8768
Log_10(Cargo)
-0.1361
Log_10(Price)
-0.8208
0.6860
0.6221
cylinder
-0.7572
length
-0.7516

0.8530
-0.9102
-0.0440
0.7976
0.6592

1.0000
0.8260

0.1060
1.0000

0.6757
0.1449

0.8700
0.8073
0.8055

0.1060
0.1449
1.0000
0.8335
0.6757
0.8055

-0.0684
-0.0684

-0.0225
0.0020
1.0000

0.8700
0.8073

0.8809
0.8809

-0.7516
0.6592

0.7483
0.8768

-0.0225
-0.0020

0.6860
0.6221

1.0000
0.6724

0.6724
1.0000

The biggest correlation is with Log(Wt). Therefore this variable

gives the best fitting linear regression.

Note that MPG City = 171.42143 - 18.898537 Log(Weight(lb))

In that analysis, the SSE was 0.1327.
Add Log(HP) to this model as another predictor. This creates a multiple
regression with two predictor variables.
MPG City = 3.61 0.513 Log(Weight(lb)) 0.25Log(HP)
Choose one (or sometimes more) variables as the most important among
the predictor variables. The best single choice is that having the largest
correlation with the y-variable.

Scatterplot Matrix

Plots
Actual by Predicted
Plot

Residual by Predicted
Plot