Optimal Selection of Ridge Biasing Parameter for Linear Regression Model

.
Summary

The choice of ridge parameter plays a pivotal role in achieving the desirable property i.e. the
smaller variance. Generally, this is achieved at the cost of introducing a bias in the estimates.
In this work, we will propose a new method for selection of optimal value of ridge parameter.
The main objective in this new proposal is to achieve the smaller variance without a
substantial increase in the bias. The proposed estimators will be compared with the ordinary
least square and existing ridge estimators. Extensive simulations will be performed to
evaluate the performance for small to large samples. Real-life examples will be provided to
illustrate the use of proposed methods.

Keywords: Multicollinearity, linear regression, Monte Carlo simulation, ridge estimators,

MSE

1. Introduction

The multiple linear regression (MLR) model is defined as:

where represents vector of dependent variable vector, represents the design

matrix of fixed independent variables, denotes the vector of unknown regression
coefficients and represents the vector of error term given that . The
ordinary least square (OLS) estimator of regression coefficient ) is given as:

and

( ̂)

The OLS estimator is unbiased and may have smaller variance under usual conditions
but its covariance matrix is heavily dependent upon the characteristics of matrix.
When the independent variables are correlated, the OLS remains unbiased but it leads to a
substantial increase in the variance.

2. Background of the Proposed Research Project

1
To overcome the problem of multicollinearity, Hoerl and Kennard (1970) proposed ridge
regression (RR) estimator by introducing a biasing parameter, , in the objective function of
the Lagrange multiplier as follows:

where represents a constant value, i.e., . The normal equations for regression
parameter are obtained as:

( )

The solution of equation (5) provides the RR estimator of as:

̂ ( )

where represent the biasing parameter, denotes the number of independent variables and
denotes the identity matrix of order . The biasing parameter plays a vital role in
tradeoff between the bias and consistency of the estimator. When ̂ approaches to
OLS estimator and we obtain the unbiased estimator but inconsistent estimator of . Further,
when , ̂ we obtain consistent, stable but biased estimator of For positive
value of biasing parameter ( ), the estimator provides smaller mean squared error compared
to the OLS estimator (Muniz & Kibria, 2009).

Multicollinearity is a common practical problem that requires an optimal solution. In

ridge regression, the key idea is to introduce a biased estimator that decreases the overall
variance. Thus, the optimal value of biasing parameter which can yield the best results is still
an open issue in the literature. The ridge regression (RR) estimator is a function of biasing
parameter and many researchers have proposed different methods for its estimation (see
e.g., Kibria 2003; Muniz & Kibria, 2009; Khalaf et al., 2013; Dorugade 2014; Suhail, et al.,
2020; Kibria & Lukman, 2020).

Consider the canonical form of the MLR model in (1) as follows:

where , ( ) and represents orthogonal matrix given

that and  where  ( ) consist the eigen

2
values obtained from the matrix. The OLS estimator ( ̂), and the ridge regression
estimator ̂ in the canonical form are defined as follows:

̂ 

where ( ) and , for The MSE of the OLS estimator

and ridge regression estimator is defined respectively as:

̂ ∑( ⁄ )

̂ ∑ ∑

To deal with the problem of multicollinearity, Hoerl and Kennard (1970) introduced an
alternative technique to OLS using (4). To use ridge regression, a researcher needs to
determine the value of biasing parameter ( ). Thus, for the estimation of many methods
have been recommended by the investigators in terms of minimum MSE of ridge estimator
which depends on parameter vector of regression coefficient ( ) and the variance of MLR
model ( ). Since and are unknown, the researcher needs to employ their estimates. In
past twenty years, the researchers have been focusing on the estimation of biasing parameter
( ) by several ways and under different situations and then compare the results with the OLS
estimator. Much discussion concerning ridge regression go around finding an optimal value
of biasing parameter .

To estimate the biasing parameter ( ), Hoerl and Kennard (1970) defined the ridge
estimator as follows:

̂ ̂ ( )
̂

where ̂ represents the residual mean square estimate and can be computed using ̂
̂
∑ , ̂ are the OLS residuals and ̂ (̂ ̂ ).

Hoerl, et al., (1975) suggested a new ridge estimator as:

3
 ̂
̂
∑ ̂

Some new methods to estimate were suggested by Kibria (2003) by taking the
arithmetic mean, geometric mean and median of ̂ ⁄ ̂ respectively are defined as:

̂
̂ ∑( )
̂

̂
̂
√∏ ̂

̂
̂ ( )
̂

3. Justification / Rationale of the Project:

In general, biasing parameter ( ) in ridge regression leads to reduced variance but the cost
may be high in terms of bias. To deal with this problem, we will propose some new
estimators that may achieve smaller MSE with no or very little bias.

4. Research Objectives:

i. To propose a new ridge estimator to address the problem of multicollinearity with smaller
variance and bias.

ii. To address the estimation issue specifically for ill-conditioned problems.

iii. To make the estimator least affected by the signal-to-noise ratio.

5. Methodology:

To evaluate the performance of the proposed estimators in this study, the MSE criterions will
be used. Many researchers e.g., Kibria (2003), Suhail et al., (2020) have used the MSE
criterion for performance analysis of estimators. The MSE is described as follows:

̂ ̂ ̂ ∑ ̂

6. Proposed Analysis:

4
Following Kibria (2003) and Suhail et al., (2021), we have generated the independent
variables equation as follows:

where represents the specified degree of correlation between the independent variables
and is generated using the standard normal distribution. The response variable is obtained
by:

where represents the pseudo-random numbers i.e., and denotes the

number of observations.

Algorithm: MSE Computation

i. Generate the independent variables ( ) matrix using (18).

ii. Obtain the eigen values (i.e., ), and eigen vectors ( ) of the matrix
given that ∑
iii. Obtain the regression parameters in canonical form ( ) by using where
denotes the eigen vector of the maximum eigen value ( ) and
.
iv. Generate the random error term ( ).
v. Obtain the dependent variables ( ) by using the model in (19).
vi. Compute the OLS and RR estimates using (8) and (9) respectively.
vii. Repeat the above steps (ii) to (vi) for each of Monte Carlo runs.
viii. Calculate the MSE using:

̂ ∑ ̂ ̂

References
Dorugade, A., 2014. New ridge parameters for ridge regression. Journal of the Association of
Arab Universities for Basic and Applied Sciences, 15(1), pp. 94-99.

Hoerl, A. E. & Kennard, R. W., 1970. Ridge regression: Biased estimation for nonorthogonal
problems. Technometrics, 12(1), pp. 55-67.

5
Hoerl, A. E., Kennard, R. W. & Baldwin, K. F., 1975. Ridge regression: Some simulations.
Communications in Statistics-Theory and Methods, 4(2), pp. 105-123.

Khalaf, G., Mansson, K. & Shukur, G., 2013. Modified ridge regression estimators.
Communications in Statistics-Theory and Methods, 42(8), pp. 1476-1487.

Kibria, B. G., 2003. Performance of some new ridge regression estimators. Communications
in Statistics-Simulation and Computation, 32(2), pp. 419-435.

Kibria, B. & Lukman, A., 2020. A new ridge-type estimator for the linear regression model:
Simulations and applications. Scientifica, Volume 2020, pp. 1-16.

Muniz, G. & Kibria, B. G., 2009. On some ridge regression estimators: An empirical
comparison. Communications in Statistics - Simulation and Computation, 38(3), pp. 621-630.

Suhail, M., Chand, S. & Aslam, M., 2021. New quantile based ridge M-estimator for linear
regression models with multicollinearity and outliers. Communications in Statistics-
Simulation and Computation, pp. 1-25.

Suhail, M., Chand, S. & Kibria, . B. M. G., 2020. Quantile based estimation of biasing
parameters in ridge regression model. Communications in Statistics-Simulation and
Computation, 49(10), pp. 2732-2744.

Suhail, M., Chand, S. & Kibria, G. B., 2019. Quantile-based robust ridge m-estimator for
linear regression model in presence of multicollinearity and outliers. Communications in
Statistics-Simulation and Computation, pp. 1-13.