Comprehensive Examination

Item 3

1.1. The issue that Amie didn’t address in her research is called Perfect Collinearity among

Predictors or “violation of no perfect multicollinearity assumption” in multiple linear regression

analysis (MLRA hereafter). According to R. Warner, in MLRA, there should be no perfect

linear relationship among predictors (2013, p. 462). High correlation (Pearson’s rho ≥ .8)

between two or more linearly related predictors creates a perfect linear correlation between

them. This, in turn, creates a high or a perfect multicollinearity when one variable is completely

predictable from one or more other variables. Hence, although we want each of our predictors

to be significantly correlated to the outcome, we want our predictors to correlate to each other

as little as possible for regression analysis to be unbiased. If only two out of five predictors of

Amie’s model are correlated at rho = .8, it would have jeopardized her results. Five highly

correlated predictors are likely to result in perfect multicollinearity and make the results of the

analysis highly questionable: even if the overall multiple R is statistically significant, no one

predictor may be judged as statistically significant (Warner, 2013).

High multicollinearity received a great deal of attention in classical MLRA in 1990s. The work

of N. Morrow-Howell (1994) outlines two major problems that it may create: inflated standard

errors and reduced magnitude of parameter estimates. High level of correlation between

predictors overlaps their influences and limits the size of multiple R. It makes the information

on their unique contributions quite small and difficult to estimate. Morrow-Howell also cites

evidence that high multicollinearity may distort regression coefficients even at as low values of

Pearson’s rho as 0.4. (1994, p. 249).

1.2. To effectively separate the unique predictive contributions of correlated predictors via

MLRA we must avoid having highly correlated predictors. An effective method for highly

correlated variables (Pearson’s rho ≥ .8) is to combine two or three highly correlated variables

(Pearson’s rho ≥ .8) into one index. For instance, J. Smits (2011, p. 16) applied this method and

calculated a single index that captured all predictor variables of the aspects of governance, that

Page 1 of 8
Comprehensive Examination
Item 3

were highly correlated. The same author applied a method to deal with the perfectly correlated

predictors (Pearson’s rho ≥ .9) considering two variables as a single predictor and using only

one of them in analysis. Dependency ratios (DR) in Smits data were highly correlated with

working-age share (WA) of the population, violating the assumption of no perfect

multicollinearity in regression analysis. Following theory, Smits judged that WA influences the

youth and old age dependency ratios, which in turn influence savings and investments, which in

their turn affect productivity growth. She chose to exclude WA from her regression analysis,

while keeping DR and testing direct effects of productivity and behavior.

A. Field states that a perfect multicollinearity among predictors makes impossible to get unique

estimates of the regression coefficients because there is “an infinite number of combinations of

coefficients that would work equally well” (Field, 2013, p.324). He also warns that low

multicollinearity is almost unavoidable and that even lower than .8 correlation might affect the

analysis. To prevent this, Field suggest obtaining Variance inflation factors (VIF) for every

predictor. If the average VIF is substantially greater than 1, or if the largest VIF is greater than

10, the regression may be biased. The variables with highest VIF are those that should be of

concern. Even more robust method is to scan eigenvalues of the scaled, uncentred cross-

products matrix. The predictors that make the results biased load heavily on a single small

eigenvalue, close to the end of the matrix (Field, 2013, p. 342).

2.1. Ying may ruin his efforts to run a rigorous study by entering all six variables from his data

into regression analysis in random order. 313 According to Field, predictor variables should be

selected with a great deal of care because the values of the regression coefficients depend upon

the variables in the model and the way in which they are entered (2013, p. 385). Researchers

decide what data to collect based on past research and substantive theoretical importance of

particular variables. However, some of collected variables may turn unrelated to the outcome, in

contrary what the researcher hypothesized. Large number of predictors greatly affects the power

Page 2 of 8
Comprehensive Examination
Item 3

of analysis (Field, 2013, p. 313). Including too few predictors, on the other hand, can lead to

inaccurate predictions because one or more important predictors are missed (Halinski & Feldt,

1970). To eliminate unnecessary variables that do not contribute to explained variance in

outcome but decrease precision, Halinski and Feldt suggested running several preliminary

MLRA(s) that will help to identify which predictors are statistically relevant to an outcome

(1970).

2.2. Furnival, Wilson, and Robert (1974) summarized the research on the most approved by

academia but tedious all-possible-subsets approach, which is doable, but will require running of

68 regression analyses (2 at power of the number of predictors) to test all possible combinations

of 6 predictors of Ying’s data set and to select the one with the best fit. The best fit can be

determined by the smallest Akaike Information Criterion (AIC) or by Bayesian information

criteria (BIC) (Field, 2013, p. 802), which penalize the model for having more variables.

Furnival et al. suggested a “Leaps and bounds” algorithm, that can help to find several

candidate models with the best fit without actually computing all possible models, but it works

well for up to 4 predictors (Furnival et al, 1974). While there are many books that outline other

(stepwise) variable-selection procedures for a univariate MLRA, A. Field provided a short and

to-the-point critique (2013, p. 322). Stepwise methods have three basic variations. Forward

Selection initially selects one predictor that maximally and significantly correlates with the

outcome and enters it into the regression equation. At each following stage, the predictor

providing the maximum significant increase in the coefficient of multiple determination R2

(which is the explained variance, multiple R2 hereafter) is added to previously selected

predictors. Stepwise Regression differs from the Forward Selection by allowing for the deletion

of predictors after their entry if its contribution to multiple R2 becomes nonsignificant.

Backward Elimination starts with building the regression equation for the full set of predictors

(six in Ying’s data set). Then, the reduction in multiple R2 is calculated for each predictor if it

Page 3 of 8
Comprehensive Examination
Item 3

were deleted and the F ratio associated with that decrease. The variable with the smallest

nonsignificant F ratio is removed. this process is repeated until all F ratios are significant or

until the removal of another variable produces a significantly lower multiple R2 than the

multiple R2 for the full set. According to Field, being influenced by random variation in the

data, all three stepwise methods are not guaranteed to find an optimal candidate subset, and

seldom give replicable results if the model is retested, but provide useful results (2013, p. 322).

He suggests choosing Backward Elimination out of three, because it is less likely to exclude

predictors involved in suppressor effects, hence reduces the risk of type II error (2013, p. 322).

For the final or main regression analysis, Field recommends a Forced Entry (also called

Simultaneous or Standard Regression), when all thoroughly selected predictor variables are

entered forcefully in one step (Field, 2013, p. 322).

3.1. Morrissey and Ruxton described the issue that emerged in Cindy’s and numerous other

papers as a “superficial similarity of simple and multiple regression that leads to confusion in

their interpretation” (2018). They state that a simple linear regression uses regression line to

describe associations between two variables, while a multiple linear regression describes the

partial (or direct) effects of several variables conditioned on one another. Although the focus of

Morrissey and Ruxton’s paper is on misunderstanding of collinearity, its main idea directly

applies to Cindy’s problem: reporting two F-values under the title of “multiple regression.”

Pedhazur and Schmelkin describe F-test in MLRA as the test of significance of squared

multiple correlation (multiple R2) of the outcome with the predictors (1991). In other words, F

statistic shows whether the coefficient of multiple determination R2 is significantly different

from zero (or whether the variance explained by the model is larger than the variance

unexplained by the model). The term “model” is crucial here, because during the procedure of

variable selection we may run several models and obtain a pair of multiple R2 and F-statistic for

Page 4 of 8
Comprehensive Examination
Item 3

each of them. But in the results section, we report only one model and, therefore, a single F-

statistic as an evidence of model’s significance.

3.2 Two F-ratios were reported in Cindy’s paper, one per each regression coefficient; which

suggests that she ran two simple regressions: regressing the dependent variable on the first

predictor only, then regressing the DV on the second predictor only, and reporting both F-ratios.

B. Cronk (2017, p. 55) provides a detailed explanation how to report MLRA while meeting all

requirements of American Psychological Association (APA, 2020). First of all, Cindy has to run

a multiple linear regression rather than two simple linear regressions. Further, in addition to

standardized betas and p-values for their t-tests, Cindy has to report 1) degrees of freedom for a

single F-test for the multiple R2, and 2) the whole regression equation (including the intercept

and two regression coefficients). Although APA does not require to report individual t-tests for

every regression coefficient, D. Martin provided justification for reporting significance levels of

the t-tests in addition to the F-test for the significance of the whole model. According to Martin,

there are six possible combinations of outcomes for the F-test and the two t-tests for MLRA

with two predictors (2008, p. 9). Most importantly, the F-test allows the researcher to assert that

regression coefficients are jointly significant, while two t-tests provide the information whether

they are individually significant in relation to the outcome (Martin, 2008, p. 9).

4.1. Using a squared multiple correlation coefficient (or a coefficient of multiple determination)

R2 as a goodness-of-fit estimate for his linear model, Terrel most likely exaggerated the model

fit of his regression equation. Hutcheson and Sofroniou expressed a common among

statisticians opinion that R2 is a biased estimator of the goodness of fit: adding an additional

variable to a model inflates the R2, increases the extent to which the model fits the actual data,

and decreases generalizability of the model (1999, p. 76). H. Reisinger empirically analyzed the

influence of research designs on R2 in linear models (1997). His exploratory meta-analysis study

confirmed that R2 is a measure of the predictive ability of a model but cannot measure the

Page 5 of 8
Comprehensive Examination
Item 3

goodness of fit since it is sensitive to both the number of predictors and the sample size. Even

with fixed predictor variables, among reviewed research articles, there are many examples of

poorly fitted models with high R2 values and perfectly fitted models with low R2 values

(Reisinger, 1997).

4.2. C. Lewis-Beck and M. Lewis-Beck (2016) as well as other textbooks on multiple linear

regression analysis (MLRA) recommend an adjusted coefficient of multiple determination as a

goodness-of-fit estimate to prevent overstating true explanatory power. The criteria of < 0.1;

0.11-0.3; 0.31- 0.5; and > 0.5 have been interpreted as a poor, modest, moderate, and strong fit

respectively (Muijs, 2004, p. 165). The adjusted R2 accounts for degrees of freedom and is

widely recognized as almost unbiased estimator of the multiple correlation parameter in

population (ρ2): “the adjusted R2 ‘penalizes’ for adding extraneous variables to the model”

(Lewis-Beck & Lewis-Beck, 2016). In other words, while unadjusted R2 assumes that all

predictors add to explained variance in the outcome, the adjusted R2 assumes that only

statistically significant predictors contribute to the explained variance and shrinks if a non-

significant predictor is added to the model.

However, there is ongoing discourse in academia about the necessity to develop a better

estimate of the model fit for MLRA which would have accounted not only for the number of

predictors but for the sample size as well. As H. Bar-Gera states, the extent of the inflation of

R2 depends primarily on the sample size, and secondarily on the number of predictors.

Controlling for the number of predictors, the larger sample size results in a smaller difference

between adjusted and unadjusted R2 values (Bar-Gera, 2017). Via a true R2 parameter in

population, ρ2, Bar-Gera demonstrated that “the unadjusted R2 overestimates ρ2, while the

traditional R2Adj underestimates it” (2017). He suggests an alternative bi-adjusted estimator of

ρ2, but I would suggest using adjusted R2 for MLRA goodness-of-fit estimation until a new

estimator is approved by the academia.

Page 6 of 8
Comprehensive Examination
Item 3

References

American Psychological Association. (2020). Publication manual of the American

Psychological Association : the official guide to APA style. (7th ed.).

Bar-Gera, H. (2017). The Target Parameter of Adjusted R-Squared in Fixed-Design

Experiments. The American Statistician, 71(2), 112–119.
https://doi.org/10.1080/00031305.2016.1200489

Cronk B.C. (2017). How to Use SPSS: A Step-By-Step Guide to Analysis and Interpretation (9th
ed.). Routledge

Field, A. (2013). Discovering statistics using IBM SPSS statistics : and sex and drugs and rock
“n” roll (4th ed.). Sage.

Furnival, G. M., Wilson, J., and Robert W. (1974). Regressions by leaps and bounds.
Technometrics, 16, 499–511. URL: http://www.jstor.org/stable/1267601.

Halinski, R. S., & Feldt, L. S. (1970). The Selection of Variables in Multiple Regression
Analysis. Journal of Educational Measurement, 7(3), 151–157.
https://doi.org/10.1111/j.1745-3984.1970.tb00709.x

Hutcheson, G., & Sofroniou, N. (1999). The multivariate social scientist. London: Sage.

Lewis-Beck, C. & Lewis-Beck, M. (2016). Applied regression. Thousand Oaks, CA: SAGE
Publications, Inc doi: 10.4135/9781483396774

Martin, D. (2008). A Spreadsheet Tool for Learning the Multiple Regression F-test, T-tests, and
Multicollinearity. Journal of Statistics Education, 16(3). DOI:
10.1080/10691898.2008.11889573

Morrissey, M., & Ruxton, G. (2018). Multiple Regression Is Not Multiple Regressions: The
Meaning of Multiple Regression and the Non-Problem of Collinearity. Philosophy,
Theory, and Practice in Biology, 10(20200624).
https://doi.org/10.3998/ptpbio.16039257.0010.003

Morrow-Howell, N. (1994). The M word: Multicollinearity in multiple regression. Social Work

Research, 18(4), 247–251. https://doi.org/10.1093/swr/18.4.247

Muijs, D. (2004) Doing Quantitative Research in Education with SPSS. London: Sage.

Page 7 of 8
Comprehensive Examination
Item 3

Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated
approach. Hillsdale, N.J.: Lawrence Erlbaum Associates.

Reisinger, H. (1997). The impact of research designs on R2 in linear regression models: an

exploratory meta-analysis. Journal of Empirical Generalisations in Marketing Science,
2Sm(1)

Smits, J. (2011). The demographic window of opportunity: age structure and sub-national
economic growth in developing countries. NiCE Working Paper 11-102. Nijmegen
Center for Economics (NiCE), Institute for Management Research Radboud University
Nijmegen

Warner, R. (2013). Applied statistics : from bivariate through multivariate techniques (2nd ed.).
SAGE Publications.

Page 8 of 8