ECON 370: Weighted Least Squares Estimation 1

Weighted Least Squares (WLS) Estimation Given Heteroscedasticity

Econometric Methods, ECON 370

We have learned that our OLS estimator remains unbiased in the face of heteroskedasticity. To
address the problem the variance of the parameters are no longer B.L.U.E, we know that all we need
to do is to use robust standard errors to ensure our inferences are not affected.
We can however also adopt another method of estimation in the face of heteroskedasticity call the
Weighted Least Squares Method, which yields greater efficiency then OLS, if the form of the variance
is correctly specified.

1 When Heteroskedasticity is Known up to a Multiplicative

Constant
Let x = x1 , x2 , ..., xk , and
V ar(²|x) = σ 2 h(x)

where h(x) > 0, is some function of the independent variables that determines the particular het-
eroskedasticity. And assume that h(.) is known. Then for each observation in the population we can
express the variance of the parameter in the regression equation as,

σi2 = V ar(²|xi ) = σ 2 h(xi ) = σ 2 hi

Where hi varies with each observation since xi varies with each observation. Given this structure, we
could then transform the OLS regression equation so that the error term would become homoskedastic
and satisfy the Gauss-Markov assumptions, or the OLS assumptions. How can we do that? Will it
really work?
√
Consider the transformation where we divide the error term by hi . That is the new error becomes
²i
hi . What happens to variance with this transformation?

²i ²2 σ 2 hi
V ar( √ |xi ) = E( i |xi ) = = σ2
hi hi hi

Viola! The new error term is homoskedastic. This then suggests that we could divide the entire
√
original OLS regression by hi , so that we get
y β β β β ²
√ i = √ 0 + √ 1 x1,i + √ 2 x2,i + ... + √ k xk,i + √ i
hi hi hi hi hi hi
or more simply as
yi∗ = β0 x∗0,i + β1 x∗1,i + β2 x∗2,i + ... + βk x∗k,i + ²∗i

Although this new regression equation looks peculiar, and seem to complicate interpretation, you
have to keep in mind that this transformation is performed so as to achieve efficiency (obtain the
ECON 370: Weighted Least Squares Estimation 2

correct variance). At the end of the day, when we interpret the results, we are concerned with the
form of the original, untransformed regression, and not the weighted least squares regression equation.
Lastly, note that the WLS equation is still linear in the parameters, and that it satisfies all of the
Gauss-Markov Assumptions of the OLS as long as besides heteroskedasticity, all other assumptions
are not violated.

Note that the two estimators, that from the original OLS, and that from the transformed regression
equation which is a Generalized Least Squares (GLS) Estimator. However, what we actually need to
do after transformation is to perform the same OLS procedure. Although in the interpretation part
of the analysis, we are using the linear OLS, you have to keep in mind that the estimators, standard
errors, t statistics, F statistic are all from the transformed equation. As a final note, although the
goodness of fit measure is useful for calculation of the F statistic, it is not informative to us here since
as noted before, we are only concerned with the original regression relationship.

To see why all we need to do is to perform the OLS procedure is to recall that when finding the
estimators, we are in effect finding the estimator that minimizes the square of the error terms.
X X¡ ¢2
min (²∗i )2 = min yi∗ − d0 x∗0,i − d1 x∗1,i − d2 x∗2,i ... − dk x∗k,i

where dj is just the sample version of the population parameters δj for j ∈ {1, 2, ..., k}. The GLS
estimators that minimizes the above problem are called weighted least squares (WLS) estimators.
1
The name is derived from the fact that the estimators are found by minimizing the weighted, hi ,
sum of the the square of the residuals/errors. The idea behind this method is to give less weight to
observations with the greatest variance (note that the weight is the inverse of hi ). You should be able
to see now that OLS is a special case of the WLS since it gives all observations the same weight. You
should also realize why we said earlier that the estimators are not the same. To see the reason why
the OLS estimator is not the same as the WLS estimator, first note that the OLS estimator in the
case of the simple regression is, P
(xi − x)yi
βb1 = P
(xi − x)2
whereas the estimator from the WLS is,
P ∗ P xi
(xi − x∗ )yi ( hi − x∗ ) hyii
β̃1 = P ∗ = P
(xi − x∗ )2 ( hxii − x∗ )2

Consequently, we cannot use the OLS estimates in making inference. This however is not the efficient
²2 by the conditional variance for each observation, that is V ar(²i |xi ).
GLS procedure which weights b
The command to do this in stata is vwls. Note lastly that the WLS estimator can be defined for any
positive weight.

The above discussion predicates on us knowing the relationship between the variance of the errors
and the independent variable. First of all, this need not be so. If we don’t, but use an arbitrary
weight, just like the OLS estimator, the WLS estimator remains unbiased and consistent. However,
ECON 370: Weighted Least Squares Estimation 3

just like the OLS estimator, the variances are no longer correct. And just as in the OLS case, we
can always use a robust variance estimator. This has led to some criticism that if we had the robust
variances for OLS, what is the need in performing a WLS. Although correct, we must realize that in
the face of strong heteroskedasticity, it is always better to account for it than not, even though the
manner in which we account for it may be incorrect.

Read your text, pages 289-290 on an example of when the weights needed for WLS
occurs naturally

2 When the Heteroskedasticity Function must be Estimated:

Feasible GLS
As mentioned earlier, usually the exact form heteroskedasticity is not known, and we have to estimate
di , and the use of the latter yields the Feasible
h(xi using the data first, which would then give h(x
GLS (FGLS) Estimator.

A particularly flexible manner in which we can account for the unknown functional form of het-
eroskedasticity is to apply

V ar(²|x) = σ 2 exp(δ0 + δ1 x1 + δ2 x2 + ... + δk xk )

Of course the use of the exponential function is not unique, and depends on your imagination. Why
do we use the exponential function in the first place? Principally, recall that we need hi to be strictly
positive, and the fact of the matter is that the predicted values under linear functional form does not
guarantee us that, whereas exponential function does recalling the range of that functional form.

The next question is how do we estimate it? Realize that if we perform a log transformation, we
would get in return a log-levels model noting first that the estimator of the variance of error is the
expected value of the square of the errors. That is we get,

²2 = α0 + δ1 x1 + δ2 x2 + ... + δk xk + ν

which can be easily estimated using OLS, and where of course e has a mean of 0, and is independent
of x by assumption. Where do we get ². Well, simple, perform a OLS on the original (very first model
that started this whole fracas) first to get it. Upon estimation of the above, we next have to produce
b
hi which is just,
b
hi = exp(b gi ) = exp(bα0 + δb1 x1 + δb2 x2 + ... + δbk xk )
ECON 370: Weighted Least Squares Estimation 4

We will now summarize the entire procedure,

1. Run OLS for y = β0 + β1 x1 + β2 x2 + ... + βk xk + ².

²2 by squaring b
2. Create log(b ² obtained from step 1.

3. Run OLS on ²2 = α0 + δ1 x1 + δ2 x2 + ... + δk xk + ν and obtain b

hi .
1
4. Estimate y = β0 + β1 x1 + β2 x2 + ... + βk xk + ² again, but this time using the weight b
hi
and
WLS.

Although the FGLS procedure is no longer unbiased (since we are using b

hi instead of hi ), it is
consistent and asymptotically more efficient than OLS. Also, if we suspect that the model of variance
used above yields heteroskedasticity in ν, we could always use the robust variance and its consequent
test statistics in the final regression.

Another alternative to the functional form suggested is that we could include the square of the
independent variables just as was suggested in the Breusch-Pagan test using the predicted dependent
variable, and its squared value. As a final note, you cannot test for heteroskedasticity with the WLS
estimation, though you can account for it using robust variances.

When testing the usual multiple hypothesis using the F test, we must apply the WLS estimation
for all the models in the sense that we have to use the same weights (Weights are from the unrestricted
model. Why? Recall first what the null hypothesis is.)

In practice, as you will find out, the OLS and WLS estimators will produce different estimates (due
not only to the formulation) due to sampling error. It is fine if for statistically significant parameters,
the differences pertain to slight differences in magnitude. If however, the magnitudes differ by large
amount, or that the sign changes by for statistically significant and not significant parameters in the
OLS, or the parameters that were not significant becomes significant (or the other way around), we
should be concerned as it is be a signal that there are violations of other OLS assumptions such as
E(²|x) 6= 0 (which would bring about bias and inconsistency in both the OLS and WLS estimators).