Multiple Linear Regression Model

Presented by: Nivita Jain Course instructor Debasis Patnaik

BITS Pilani Course: Topics in Econometrics
Course Code: ECON G546
K K Birla Goa Campus
Why Multiple Linear Regression ?
• In CLRM, we studied the relationship between the one dependent variable and one explanatory
variable.
• The primary drawback in using CLRM is that it is very difficult to draw ceteris paribus conclusions about
how x affects y; that all other factors affecting y are uncorrelated with x—is often unrealistic
• To make the model more realistic we include multiple independent (explanatory) variables in the
regression.
• This will also lead to Model Misspecification in form of omitted variable bias as explained in further
slides.
• The simplest possible multiple regression model is three-variable regression, with one dependent
variable and two explanatory variables.
• We may write the three-variable PRF as
Yi = β1 + β2X2i + β3X3i + ui ...(1.1)
where
Y is the dependent variable,
X2 and X3 the explanatory variables
u the stochastic disturbance term, and
i the ith observation
• The coefficients β2 and β3 are called the partial regression coefficients in eq. (1.1) – the meaning will be explained
in further slide.
Estimation – Three Variable Regression Funtion
• The three –variable PRF is as:
Yt = B1 + B2X2t + B3X3t + ut ...(1.1.1)
• The sample regression function (SRF) corresponding to the PRF Eq. (1.1.1), as follows:
Yt = b1 + b2X2t + b3X3t + et ...(1.1.2)
• The OLS principle chooses the values of the unknown parameters in such a way that the residual sum
of squares (RSS) is as small as possible. To do this, we first write Equation (1.1.2) as:
et = Yt - b1 - b2 X2t - b3X3t ...(1.1.3)
• Squaring equation (1.1.3)on both sides and summing over the sample observations,we obtain
RSS: ∑et2 = ∑(Yt - b1 - b2 X2t - b3X3t)2 ...(1.1.4)
• Minimizing the eq. (1.1.4) using calculus technique of differentiation we obtain the three OLS
estimators as follows:
Estimation – Kth Variable
Partial Regression Coefficients
• Let us consider the below models,
Wage (Y) = β1 + β2X2i + ui ...(1.2)
Wage (Y) = β1 + β2X2i + β3X3i + ui ...(1.3)
where,
X2 = Education
X3 = Experience
• Thus, wage is determined by the two explanatory or independent variables, education and experience, and by
other unobserved factors, which are contained in u.
• We are still primarily interested in the effect of education on wage, holding fixed all other factors affecting wage;
that is, we are interested in the parameter β2
• Multiple regression analysis is more amenable to ceteris paribus analysis because it allows us to explicitly
control for many other factors which simultaneously affect the dependent variable.
• Compared with a CLRM analysis relating wage to education, equation (1.2) effectively takes experience out of the
error term and puts it explicitly in the equation.
• Since eq.(1.3) contains experience explicitly, we will be able to measure the effect of education on wage, holding
experience fixed where as in a simple regression analysis—which puts experience in the error term—we would
have to assume that experience is uncorrelated with education, a tenuous assumption - causing the OLS
estimator of β2 in the two-variable model to be biased.
• Not surprisingly, just as with simple regression, we will have to make assumptions about how u is related to the
independent variables, which we will see in the further slides.
• The meaning of partial regression coefficient is as follows: β2 measures the change in the mean value of Y,
E(Y), per unit change in X2, holding the value of X3 constant.
• Put differently, it gives the “direct” or the “net” effect of a unit change in X2 on the mean value of Y, net of
any effect that X3 may have on mean Y.
• Likewise, β3 measures the change in the mean value of Y per unit change in X3, holding the value of X2
constant.
• Now the Question is, How do we actually go about holding the influence of a regressor constant?
• Let us understand this with the example,
Yi = β1 + β2X2i + β3X3i + ui
where,
Yi = child mortality (CM),
X2 = per capita GNP (PGNP), and

X3 = female literacy rate (FLR)

CMi =1+ 2PGNPi + 3FLRi + i ...(1.4)

• Let’s say our objective is to get the net impact of PGNP on the CM
• For this, we need to remove influence of FLR from both CM & PGNP by running the regression of CM on FLR
and of PGNP on FLR separately
• Then regressing the residuals obtained these regression, we would obtain the net effect of PGNP on CM
• Step 1 : Regress CM on FLR,
CMi = λ1 + λ2FLRi + 1 …(1.5)
If we regress eq. (1.5) we will get 1 (residual portion) after removing the impact of FLR on CM
• Step 2: We will run another regression PGNP on FLR,
PGNPi = δ1 + δ2 FLRi + 2 …(1.6)
If we regress eq. (1.6) we will get 2 (residual portion) after removing the impact of FLR on PGNP
• Step 3: We now regress 1 on 2 , which are “purified” of the influence of FLR, we will obtain the net effect of
PGNP on CM
1 = γ1 +γ2 2 + ε …(1.7)
If we regress eq. (1.7) we will get γ2 (net impact / effect of PGNP on CM)
• The slope coefficient γ2 now gives the “true” or net effect of a unit change in PGNP on CM or the true slope
of CM with respect to PGNP. That is, it gives the partial regression coefficient of CM with respect to PGNP, β2.
• In the eq. (1.7) will have no intercept (γ1) because ,
• one of the assumption of OLS is that the regression line (Sample regression function) should pass through the
sample mean and
• one assumption of CLRM is E(ui|xi) = 0
• Following the above assumption we can say, u̅1 = 0 and u̅2 = 0
• Thus, (u̅1 ,u̅2 ) = (0,0)
• That means the line pass through the origin and thus eq. (1.7) will have no intercept term
Assumptions
• The regression model is linear in the parameters
• There is no specification bias. The model is correctly specified.
• X2 and X3 are uncorrelated with the disturbance term u
• The error term u has a zero mean value;
that is, E (ui) = 0
• Homoscedasticity, that is, the variance of u, is constant:
var (ui) = σ2
• No autocorrelation exists between the error terms ui and uj:
cov (ui, uj) = 0 i ≠j
• The error term u follows the normal distribution with mean zero and (homoscedastic) variance σ2
that is, ui ~ N(0, σ2)
Assumptions
• No exact collinearity exists between X2 and X3; that is, there is no exact linear relationship between the
two explanatory variables.
• no collinearity means that there exists no set of numbers, β2 and β3, not both zero such that,
β2X2i + β3X3i = 0
• If such an exact linear relationship exists, then X2 and X3 are said to be collinear or linearly dependent.
• For e.g., Y = Child Mortality, X2 = per capita GNP (PGNP) and X3 = female literacy rate (FLR)
• The economic theory presumes that PGNP and FLR may have some independent influence Child
Mortality.
• If there is an exact linear relationship between PGNP and FLR, we have only one independent variable,
not two, and there is no way to assess the separate influence of PGNP and FLR on Child Mortality.
• In practice, when we collect data for empirical analysis there is no guarantee that there will not be
correlations among the regressors. As a matter of fact, in most applied work it is almost impossible to
find two or more (economic) variables that may not be correlated to some extent
• What we require is that there be no exact linear relationships among the regressors.
Specification Bias : Under Specified Model
… (1.4.1)

… (1.4.2)

Estimate from the underspecified model is denoted by ̃ (tilde)

• Assume that Eq. (1.4.1) is the “true” model explaining the behavior of child mortality in relation to per capita
GNP and female literacy rate (FLR).
• But suppose we disregard FLR and estimate the simple regression as in eq. (1.4.2) would constitute a
specification error; the error here consists in omitting the variable X3, the female literacy rate.
• Now will β̃ 1 provide an unbiased estimate of the true impact of PGNP, which is given by 1 in model (1.4.1)?
• In other words, will the coefficient of PGNP in Eq. (1.4.2) provide an unbiased estimate of the true impact of
PGNP on CM, knowing that we have omitted the variable X3 (FLR) from the model?
• In absolute terms, if we compare the results of this (mis-specified) regression with the “true” multiple
regression, we will see that the results are different.
• No, as β̃ 1 is not same as 1
Allocating R2 among Regressors
• In eq (1.4) CMi = 1+ 2PGNPi + 3FLRi + i the two regressors explain the 0.7077 variation in the
e.g. of child mortality that is:
R12 = 0.7077
• In eq (1.4) when we dropped PGNP, CMi = 1 + 2FLRi + 1 ,the goodness of fit is:
R22 = 0.6696
• In eq (1.4) when we dropped FLR, CMi =1+ 2PGNPi + i ,the goodness of fit is:
R32 = 0.1662
• From the above we can say that,
• FLR & PGNP , these two variables together explains 70.77% of variation in CM
• Out of 70.77%, FLR explains 66.96 % & PGNP explains 16.62 %
• But, R22 + R32 ≠ R12 - that means we cannot allocate R2 of 0.7077 between two regressors, PGNP and FLR,
in this manner.
• Because, the two regressors are correlated, the correlation coefficient between the two being 0.2685
Adjusted R2
• From the previous example we can also observe that as the number of regressors in the model
increases, the R2 value increases.
• An important property of R2 is that it is a nondecreasing function of the number of explanatory
variables or regressors present in the model, unless the added variable is perfectly collinear with the
other regressors; as the number of regressors increases, R 2 almost invariably increases and never
decreases.
• To see this, recall the definition of R 2

• Now ∑ y2 is independent of the number of X variables in the model because it is simply ∑(Yi − Y ̄ )2.
The RSS, ∑ 2 , however, depends it is clear that as the number of X variables increases, on the number
of regressors present in the model, ∑ 2 is likely to decrease; hence R2 will increase.
• To compare two R2 terms, one must take into account the number of X variables present in the
model. R2 measures the proportion of the variation in the dependent variable accounted for by the
explanatory variable(s).
Adjusted R2
• If you do want to compare the two R2 values, we need to calculated adjusted R2.
• The term adjusted means adjusted for the df associated with the sums of squares, which is as follows:

• It is observed from above equation,

• for k > 1, R ̄2 < R2 which implies that as the number of X variables increases, the adjusted R 2 increases
less than the unadjusted R2; and
• R ̄2 can be negative, although R2 is necessarily non-negative. In case R ̄2 turns out to be negative in an
application, its value is taken as zero.
RESEARCH PAPER
TITLE OF PAPER YEAR AUTHORS OBJECTIVES METHODOLOGY FINDINGS
USED

A comparison of 2008 Yu Cen , Empirical studies of the variation This study adopts Results of this study
neural network and Jinpeng Yin in debt ratios across firms have multiple linear show that the
multiple regression used statistical models singularly regressions and artificial determinants of capital
analysis to analyze the important neural networks (ANN) structure are different in
in modeling capital determinants of capital structure models. both industries.
structure This study attempts to examine
the usefulness of ANNs as Basic statistics are The major different
analyses and predictions of capital estimated to describe determinants are
structure and to compare these each variable collected business-risk and growth
ANNs with multiple linear and t-tests are opportunities.
regression results. conducted to determine
if variables of high-tech
corporations are
different from that of
traditional corporations.
References
• Pao, H. T. (2008). A comparison of neural network and multiple
regression analysis in modeling capital structure. Expert Systems with
Applications, 35(3), 720–727.
https://doi.org/10.1016/j.eswa.2007.07.018
• Gujarati, D. N. (2015). Basic econometrics (5th ed.). McGraw-Hill
Education.
• GMK Madnani. (2024). Introduction to Econometrics, Principles and
Applications (8th ed.)
Thank you