Organizational Research Methods

15(3) 413-435
ª The Author(s) 2012
The Illusion of Statistical Reprints and permission:
sagepub.com/journalsPermissions.nav
Control: Control Variable DOI: 10.1177/1094428111428817
http://orm.sagepub.com

Practice in Management
Research

Kevin D. Carlson1 and Jinpei Wu2

Abstract
The authors extend previous recommendations for improved control variable (CV) practice in
management research by mapping the objectives for using statistical control to recommendations
for research practice. Including CVs in research designs to permit statistical control of ‘‘nuisance’’
variance is a common research practice that is subject to well-documented and potentially serious
problems. Yet because CVs are frequently weakly related to focal variables, they rarely influence the
interpretation of results. As a result, current practice offers an illusion of statistical control when in
fact little control actually occurs. The authors extend the growing literature on CV practice by
examining the ambiguity of researchers’ stated purposes for using statistical control that makes it
difficult to determine whether common CV practice accomplishes any of these intents effectively.
Guidelines for improving research practice are offered, including adopting a conservative stance
toward the inclusion of CVs in the analysis of quasiexperimental and correlational designs guided
by the principle ‘‘When in doubt, leave them out.’’

Keywords
control variable, multiple regression, statistical control

This article examines problems with the alignment of control variable (CV) practice with stated
purposes for statistical control in management and organization research and how this affects
research findings and interpretations of results. Previous work in psychology (e.g., Dohrenwend,
Dohrenwend, Dodson, & Shrout, 1984; Hull, Tedlie, & Lehn, 1992; Meehl, 1970, 1971; Newcombe,
2003), medicine (e.g., Christianfeld, Sloan, Carrol, & Greenland, 2004), and management (e.g.,
Atinc, Simmering, & Kroll, 2012; Becker, 2005; Breaugh, 2005, 2008; Breaugh & Arnold, 2007;
Spector & Brannick, 1995, 2011; Spector, Zapf, Chen, & Frese, 2000) describes how using CVs can
lead to conceptually ambiguous relationships, inappropriate interpretations of findings, faulty

1
Department of Management, Pamplin College of Business, Virginia Tech, Blacksburg, VA, USA
2
School of Business, Minnesota State University Moorhead, Moorhead, MN, USA

Corresponding Author:
Kevin D. Carlson, Department of Management, Pamplin College of Business, Virginia Tech, 2094 Pamplin Hall, Blacksburg, VA
24061
Email: [email protected]
414 Organizational Research Methods 15(3)

inferences, and incorrect policy decisions. Yet evidence from contemporary practice suggests these
issues may not be widely understood or are systematically ignored by authors and reviewers.
The implications of statistical control and CV use are important to organizational research for
several reasons. First, the use of statistical control is prevalent in management research—Atinc
et al. (2012) report that 812 of the 1,199 articles published from 2005 to 2009 in the Academy
of Management Journal, Journal of Applied Psychology, Journal of Management, and Strategic
Management Journal included CVs. But it is not clear from the limited information reported in
manuscripts (Becker, 2005; Breaugh, 2008) why researchers included statistical controls or whether
those efforts are successful. Second, despite an extensive literature documenting the problems asso-
ciated with partialling shared variance to achieve statistical control (Breaugh, 2005, 2008; Breaugh
& Arnold, 2007; Burks, 1926; Christianfeld et al., 2004; Meehl, 1970, 1971; Newcombe, 2003;
Spector et al., 2000; Spector & Brannick, 2011; Stone-Romero, 2007), questionable practice and
inappropriate interpretations persist. Breaugh (2008) documents several instances in recent man-
agement research where statistical control resulted in unexpected findings and substantive misin-
terpretations of results. Third, statistical control has several meanings and can be used to
accomplish different purposes in management research, yet multiple regression analysis (MRA)
applies a uniform treatment that does not accomplish some purposes. The resulting misalignment
of practice with purpose results in a series of half-truths and fundamental misunderstandings about
statistical control that are so common they often go unchallenged. Finally, given all of these prob-
lems, the limited concern about these issues among academics, journal editors, and reviewers is
noteworthy. Perhaps there is more to this story.
Conceptually we build on the work of earlier scholars (Becker, 2005; Breaugh, 2005, 2008;
Breaugh & Arnold, 2007; Christianfeld et al., 2004, Dohrenwend et al., 1984, Hull et al., 1992; Meehl,
1970, 1971; Newcombe, 2003; Spector & Brannick, 1995, 2011; Spector et al., 2000) by recognizing
different purposes for using CVs to achieve statistical control and integrating prior work around these
purposes. By mapping practices to these purposes, we demonstrate why current CV practice struggles
to accomplish any of them effectively while potentially reducing the interpretability of results. Empiri-
cally, we examine correlations between CVs and independent variables (IVs)—relationships that
receive little consideration in standard CV practice—demonstrating these relationships can and do
influence the magnitude and sign of regression coefficients, sometimes dramatically—just not very
often. In fact, the CVs in the studies we reviewed most often have little if any impact on research find-
ings or interpretations, creating the illusion of statistical control when little control actually occurs.
Finally, on the basis of our review of studies, we offer recommendations for further improving research
practice by becoming more purposeful in the use of CVs and aligning intent with research practice. In
general, a more conservative approach to the inclusion of CVs is suggested, driven by the guiding prin-
ciple ‘‘When in doubt, leave them out.’’

Control Variables and Their Roles in Research Designs

A variable can play one of three roles in a research design. The two most common roles are IV and
dependent variable (DV). IVs represent theorized explanations for changes in outcomes. DVs rep-
resent outcomes of theoretical (what the theory hopes to explain) or practical interest whose values
are presumed to ‘‘depend on’’ or are caused by factors represented by IVs. Even though nonexperi-
mental designs do not permit claims of causality or rigorous causal tests, the labels IV and DV are
commonly used in these designs as well.
The third role encompasses variables that influence our understanding of the relationships
between IVs and DVs. This category includes mediators, moderators (e.g., Baron & Kenny,
1986), and CVs. CVs are commonly used to capture factors that are broadly defined as extraneous
to the desired effect—sometimes referred to as ‘‘nuisance’’ variance (cf. Breaugh & Arnold, 2007;
Carlson and Wu 415

Meehl, 1970, 1971). Depending on the researcher’s purpose and the context of the research design,
any variable could potentially function as a CV, although some CVs appear more frequently in
management research (e.g., industry, organization size).

Achieving Control in Research Designs

Researchers often note their intent to ‘‘control for’’ extraneous influences that could independently
affect results. Control in research designs can be pursued through experimental or statistical means.
With experimental control, the researcher attempts to prevent the introduction of extraneous influ-
ences by constraining experimental conditions to hold constant those factors that might indepen-
dently influence outcomes. External influences on participants, for instance, can be controlled by
conducting research in laboratory-like contexts that limit the stimuli available to participants.
Laboratory research has the advantage of permitting tight control over sources of extraneous influ-
ence. However, strong experimental controls may also create contexts that differ substantively from
those normally observed, potentially limiting the generalizability of findings.
Statistical control, on the other hand, attempts to control extraneous influences after the fact. It
involves identifying potential sources of influence during study design and including CVs represent-
ing these sources of influence in the data collection. During data analysis, researchers then control
for these extraneous effects by mathematically partialling out variance associated with CVs in
calculating relationships between other variables. By foregoing experimental control, data can be
collected in more natural contexts, potentially increasing generalizability. Statistical controls are
often used when experimental control is not reasonable, ethical, or possible.

Purposes for Using Statistical Control

Achieving appropriate statistical control depends on the researcher’s intent. Statistical control has
been used to (a) estimate a ‘‘purified’’ relationship between variables (e.g., Spector & Brannick,
2011), (b) estimate a ‘‘controlled’’ relationship between two variables that accounts for the effects
of other meaningful variables (e.g., Breaugh, 2005, 2008; Meehl, 1970, 1971; Spector et al., 2000),
and (c) determine the incremental contribution that a variable makes to the prediction of a DV after
the effects of other variables have been considered. As Conger and Jackson (1972) note in discussing
suppressor variables, these are distinct objectives that require different types of information that
typically cannot be derived from a single analytic approach.

Purification. In purification (see Spector & Brannick, 2011), it is assumed that measures of one or
more IVs and/or the DV are contaminated and the contaminant can be removed mathematically
through partialling. The problem here is one of measurement; the observed variable does not rep-
resent the construct of interest, and the desired true conceptual relationship between such variables
can be attained by removing the contaminant. Control in this instance requires the existence of a
CV that covaries perfectly with the contaminant but is unrelated (orthogonal) to the variable of
interest. When such a variable can be identified, partialling out the effect of the CV results in a
residual variable that contains all of the relevant components of the desired variable and none
of the contaminant. The resulting regression coefficient captures the desired ‘‘purified’’ relation-
ship between the two constructs.
Relationships involving residual variables produced through partialling can take several forms
and are represented mathematically in partial and semipartial correlations. A partial correlation
represents appropriate statistical control of a targeted relationship when the same contaminant or
nuisance variance is present in both an IV and DV. The semipartial correlation accomplishes appro-
priate statistical control when a CV is associated with nuisance variance in only the IV or DV,
416 Organizational Research Methods 15(3)

resulting in two different semipartial correlations depending on which variable contains the nuisance
variance to be controlled. The difference between the partial and semipartial correlations and its con-
trol implications is important. Accomplishing appropriate statistical control involves capturing the
desired relationship. This requires that we be purposeful in determining what influences should be
controlled and where.
The primary challenge for purification is finding an appropriate CV. Often, candidate CVs cap-
ture only part of the contaminant, failing to achieve full control. More frequently, potential CVs
share variance with (are not orthogonal to) the desired ‘‘pure’’ variable. Consequently, partialling
removes a portion of the variance associated with the desired variable, resulting in what Spector
et al. (2000) referred to as throwing part of the baby out with the bathwater. Identifying appropri-
ate CVs is further complicated because an IV and CV may share combinations of meaningful and
contaminant variance, a problem that also occurs in the study of common method variance (e.g.,
Doty & Glick, 1998; Podsakoff, MacKenzie, Lee, & Podsakoff, 2003; Spector, 2006; Williams &
Anderson, 1994).
Confusion sometimes exists regarding the form of statistical control that occurs in MRA. As is
shown below, the standardized regression coefficient b1 in Equation 1 can be calculated using
Equation 2 (based on Kerlinger & Pedhauzer, 1973, Equation 6.7):
Y ¼ b1 X þ b2 Z þ e ð1Þ
rXY  rYZ rXZ
b1 ¼ ; ð2Þ
ð1  r2 xzÞ

where values are defined as above. The regression coefficients in MRA are closely related to the
semipartial correlation between the IV and DV controlling the IV (not the DV) for the effects of the
CV (cf. Breaugh, 2005, 2008; Spector & Brannick, 2011). In instances where nuisance variance
exists only in the DV or when both the IV and DV are affected, a regression coefficient does not
capture the intended relationship.

Accounting for other meaningful variables. Often, researchers seek to estimate the relationship
between an IV and a DV in the presence of, or otherwise recognizing the effects of, other variables
(see Breaugh, 2008; Meehl, 1970, 1971; Spector et al., 2000). As contrasted with purification, the
problem here is not one of measurement. The variables are argued to capture the constructs of inter-
est. Instead, there exists some reason to believe that the zero-order relationship between these vari-
ables is in some way limited, is inferior, or fails to capture the relationship of interest, and including
one or more CVs results in a regression coefficient (b or b) that captures the desired relationship. As
described by Cohen, Cohen, West, and Aiken (2003),

this is the familiar phenomenon of redundancy among correlated independent variables with
regard to what they explain. The Y relationship with each of the independent variables over-
laps to some degree with their relationships with other variables in the statistical model. This,
in turn, requires a concept of the unique (partialed) relationship of each variable with Y, in the
context of the other variables in the model. This picture is often sharply different from that
provided by looking at each factor singly. (p. 7)

This type of statistical control is described using a variety of stated intents. For instance, researchers
might state their desire to isolate the unique contribution of an IV in the presence of other variables
(e.g., Dierdorff & Morgeson, 2007) or rule out alternative explanations (e.g., Tiwana & Kiel, 2007).
Furthermore, researchers might be concerned that unmeasured variables generate spurious
Carlson and Wu 417

relationships (e.g., Bercovitz & Mitchell, 2007) and believe including CVs will produce more
accurate estimates of desired relationships (e.g., Nerkar & Shane, 2007).
A primary concern with this form of statistical control is its potential ambiguity. Determining the
effectiveness of statistical control efforts requires an understanding of what influence the undesired
effect exerts on outcomes. However, some purposes (e.g., to control for potential confounds of our
relationships; e.g., Karaevli, 2007; to avoid spurious relationships; Lazzarini, 2007) are so ambig-
uous as to provide no basis for determining whether CVs accomplished the desired controls. In what
way is a focal relationship confounded, spurious, or inaccurate? More importantly, how would we
judge whether the ‘‘controlled’’ relationship is in fact no longer confounded or spurious and is more
accurate? Other purposes (e.g., isolating the effects of our variables, accounting for alternative
explanations) create conceptual ambiguities for interpretation. What does it mean to isolate an
effect? Simply put, the absence of specifically defined intentions makes it very difficult to determine
how one would know whether statistical control using CVs accomplishes the intended purpose.
Even when the intent of statistical control can be unambiguously established, whether MRA with
CVs achieves the intended control can still be questioned. Consider, for example, what is perhaps the
least ambiguous intent in this category, holding other influences constant (e.g., Nerkar & Shane,
2007). Statistical control is often interpreted, as Hays (1988) notes, as ‘‘equivalent to holding [the
CV] at a constant value for each individual i making up the data, and then assessing the resulting
correlation between [X] and [Y]’’ (p. 610). But the experimental design effect of manipulating an
IV while holding other aspects constant cannot be replicated in correlational research using partial-
ling, particularly when IVs and CVs are correlated. Whether the estimated effect of changes in a
residual IV on a DV produced through partialling corresponds to the results for the comparable
manipulation of the IV in an experimental design is equally unclear. As Newcombe (2003) notes,
‘‘doing so may lead to the observation of ‘effects’ . . . that are profoundly misleading because they
statistically remove the effects of variables that cannot be removed in real life’’ (p. 1050).
Meehl (1970, 1971), Breaugh (2008), and Spector et al. (2000) all address the potential conse-
quences for reporting and interpreting results when partialling controls too much. Partialling shared
variance becomes more challenging as the magnitude of correlations among predictors (IVs and
CVs) increases. The more strongly predictors are correlated, the less likely it is that mathematically
derived residuals on which regression coefficients are based reflect the properties of the intended
‘‘controlled’’ relationship (Breaugh, 2008). Unlike purification, the assumption here is that the
IV and DV variables are already construct-valid representations of the intended constructs. Thus,
substantive partialling changes the regression coefficients’ meaning. The appropriate conceptual
interpretation of the resulting residual—which is mathematically, not theoretically, determined—
is difficult to judge. What is clear is that including CVs that are correlated with IVs can result in
regression coefficients that are not equivalent with the original unpartialled variable. Furthermore,
the specific definition of the resulting residual depends on the specific combination of other vari-
ables included in the regression equation; changing one or more IVs or CVs in the regression
model changes the definitions of the regression coefficients.

Incremental prediction. Unlike the two previous purposes, the focus of incremental prediction is not
the relationship between the IV and the DV but rather the incremental improvement in the overall
‘‘prediction’’ of a regression model (i.e., DR2 or DR; Beatty, 2003; D’Andrade & Dart, 1990). Of
primary interest is whether the addition of a new IV offers a meaningful improvement in variance
accounted for by the model. When baseline models include CVs representing the strongest previ-
ously identified predictors, the magnitude of incremental increases in R2 represents that variable’s
unique contribution. The challenge here centers first on identifying all meaningful predictors of the
DV of interest. Leaving out an important predictor limits the potential contribution of any evidence
418 Organizational Research Methods 15(3)

of incremental prediction. Conversely, adding large numbers of unimportant CVs (Becker, 2005)
reduces degrees of freedom and the power of designs.
In addition, not all CVs are created equally. Many CVs are variables drawn from theory that con-
tribute to attempts to explain why variations in DVs occur. Other CVs—identified here as artifact
CVs—reflect characteristics of samples or artifacts of research design decisions. These variables
(i.e., organization size, industry membership, research setting, or measurement approach) often cov-
ary with the DV, but because they are often empirically derived, they do not offer an explanation for
why the association exists. Artifact CVs are effectively placeholders—covariates of unknown the-
oretically meaningful variables that, when identified, would offer explanations for the nature of their
association with the DV that artifact CVs do not. In incremental prediction, new theoretically mean-
ingful IVs compete with other variables to account for variance in the DV. Evidence of incremen-
tal prediction beyond all recognized controls is strong evidence that an IV has utility. However,
entering artifact CVs first in hierarchical analyses places theoretically meaningful variables at
an inappropriate disadvantage.
Finally, it is important to recognize the distinction between the unique contribution a variable in
MRA makes to accounting for variation in the DV as estimated by b versus comparing changes in R2
(or R) between regression models with and without the variable. When the predictors in a regression
model are orthogonal (i.e., all correlations among predictors are zero), squaring b provides an esti-
mate of the unique variance in the DV accounted for by that IV. That value will equal the magnitude
of DR that would be produced by a hierarchical analysis in which the focal variable is added last into
a model that already contains all other predictor variables. That is not true when predictors are cor-
related. In these instances, MRA partials variance among correlated IVs (and CVs) in the manner
that minimizes overall errors of prediction (i.e., 1 – R2), parsing variance two predictors share with
the DV is parsed between the bs. Thus when IVs and CVs are correlated, a b represents a variable’s
relationship with the DV that is unique from the relationships that other bs within the model have
with the DV, but because of the way shared variance may be allocated among bs, that value may no
longer be equivalent to the incremental contribution to R produced when the same IV was added last
in a hierarchical analysis.
As a result, when using CVs in tests of incremental contribution, failing to include relevant the-
oretically meaningful CVs could result in an overestimate of the unique contribution of IVs.
Furthermore, including artifact CVs before IVs could inappropriately underestimate the contribu-
tion of IVs when artifact CVs that do not offer a theoretic explanation for variation in the DV are
entered first. Assuring the appropriate CVs are included is essential to the interpretation of the
analysis. But even then, when CVs and IVs are correlated, regression coefficients may not provide
evidence of unique contribution.

The Universal CV Playbook

Despite distinct purposes for using statistical control, a review of research practice suggests
researchers frequently adopt what we described as the ‘‘universal CV playbook.’’ Atinc et al.
(2012) noted, ‘‘The ubiquity of control variables in management research, especially the frequency
with which some are used (such as firm size in macro papers) struck us as being something akin to an
isomorphism’’ (p. 4). We suggest this consistency of practice extends not only to the frequent use of
specific CVs but to aspects of CV practice, including (a) selecting CVs on the basis of their demon-
strated or hypothesized associations with a study’s DVs (but not IVs); (b) erring on the side of
including too many, rather than not enough, CVs; (c) entering all CVs as a block of variables in
MRA; (d) entering CVs in the first step of hierarchical analyses; and (e) reporting and interpreting
the sign and significance of regression coefficients from a model containing all CVs, IVs, and, per-
haps, any interactions or higher-order variables. These universally applied practices create two types
Carlson and Wu 419

of problems. First, because they combine techniques appropriate to different statistical

control purposes, intended outcomes can be confounded. Second, these practices are only par-
tially aligned with the requirements of any of the intended purposes for using statistical control
defined above.
Table 1 provides an overview of potential misalignments between ‘‘universal CV playbook’’
actions and statistical control purposes. The practice of focusing exclusively on variables that have
strong relationships with DVs, for example, without respect to their relationship with other predic-
tors, is relevant for incremental prediction, is irrelevant for purification, and because of the
mechanics of MRA, has only a minor role when ‘‘accounting for other variables.’’ Order of entry
is important in incremental validity analyses when rxz is substantial, but the appropriate order in
which CVs should be included depends on purpose. Finally, interpreting the magnitudes of regres-
sion coefficients is consistent with purification and accounting for other variables, but whether the
assumptions required for these types of interpretations are ever met is in doubt.
As noted by Spector and Brannick (2011), including more CVs is not necessarily a safer design
strategy. Including CVs with unknown, nonzero associations with IVs has the potential to confound
statistical control efforts, potentially disrupting efforts to purify relationships and further restricting
residuals. Including CVs related to different control purposes can confound attempts to purify a rela-
tionship or hold another constant. As Becker (2005) notes, the more CVs are included, the more
likely that problems can occur. These combined effects decrease rather than increase the interpret-
ability of regression coefficients. As a result, following these practices at best does no harm beyond
increasing the complexity of the analysis and reporting. But at its worst, ritualistic application of
these practices results in the illusion of statistical control while producing conceptually ambiguous
effect size data that may have no meaningful interpretation.
Given the documented problems noted here and elsewhere, one might expect concerns about
CV practice would receive greater attention in contemporary research practice. This does not
appear to us to be true. To the extent our observations are representative, we suspect a number
of contributing factors. First, whereas achieving experimental control is an active component of
research design, statistical control in quasiexperimental designs often occurs after the fact and
simply may be perceived as tangential to both research design and analysis. Thus, statistical con-
trol may be more of an afterthought that simply does not receive much consideration. Second,
given Becker’s (2005) and Breaugh’s (2008) finding that many authors do not indicate the purpose
for including CVs, it is possible that perceptions of ambiguous intent and misalignment of purpose
and practice may not be sufficiently salient for authors or reviewers. Third, the consequences of
CVs on outcomes may not be easy to identify. The use of a wider range of statistical methods that
involve transformed variables (e.g., logic regression, negative binomial regression) or less famil-
iar effect size statistics (e.g., structural equation modeling [SEM], hierarchical linear modeling
[HLM]) may make it harder to recognize the influence of CV effects. Fourth, despite previous dis-
cussion of these issues, it is possible that researchers, focused on the primary hypotheses in their
studies, may not be aware of the concerns inherent in the use of CVs to achieve statistical control.
Although these factors may play a role, we do not believe they are sufficient to account for current
practice. An alternative hypothesis is that the misalignment of practice and purpose has little
impact on results. This could be the case if these effects occur infrequently or are of insufficient
magnitude to cause alarm.
To understand the extent of these potential problems, we extended previous analyses of CV
practice to more directly examine researchers’ stated statistical control intentions and the align-
ment of control intent with research practice. Control in MRA and potential problems in the inter-
preting regression coefficients in the presence of CVs both begin with meaningful CV-IV
correlations. Therefore, we also examined the extent to which meaningful rxz correlations exist
in contemporary research.
 Table 1. Alignment of Standard Control Variable Practice to Possible Purposes for Pursuing Statistical Control

420
Purposes
Practices Purification Accounting for Other Variables Incremental Prediction

Select CVs on basis of Not aligned Ambiguous Aligned

relationship to DV Appropriate only if the covariance of the CV Mathematically partialling the shared variance In incremental prediction, the researcher is
and DV reflects a contaminant in the DV and between CVs and IVs can cause regression interested in knowing whether an IV of
does not otherwise covary with the ‘‘pure’’ coefficients to have ambiguous interpreta- interest is capable of incrementing variance
DV. Never appropriate with MRA which tion unless the intended control effect is accounted for in the overall MRA model
only controls the IV for the CV. clearly specified and the selected CVs pro- beyond that offered by other already
vide the intended control. recognized IVs and CVs.
Incorporate all CVs into Not aligned Not aligned Aligned
the regression model Appropriate only if all CVs are correlated only If CVs representing different control objectives Appropriate. Placing all other predictors into
with contaminants affecting IVs. are included, any CV-IV effects for these the MRA model provides a severe test of the
variables may be confounded by the inclusion capacity of the focal IV to increment R2.
of CVs for different purposes. The more CVs
included, the more difficult interpretations of
regression coefficients become.
Enter all CVs first in Aligned Irrelevant Aligned (but problematic)
hierarchical analyses When CVs reflect only contaminants in IVs, The order of entry into the MRA has no effect Entering CVs first gives them the first priority
this assures contaminants are removed and on the magnitudes of bs within a given model. to account for variance in the DV. If CVs for
in estimation of IV-DV relationships. contexts are entered first, the variance that
should be attributed to conceptually impor-
tant IVs may be understated.
Report and interpret the Aligned (but problematic) Aligned (but problematic) Not aligned
regression coefficient Appropriate only if all CVs represent only Focuses on the relationships, but when the CV The focus of incremental prediction is R2 (or
contaminants and none covaries with the (or another IV) covaries with the IV, bs R). b in a full regression model is an estimate
substance of an IV. represent the effect of changing the (math- of the unique contribution the residual vari-
ematical derived) residual. Offers statistical able in the given model, but for correlated
control, but estimated effects may not be IVs and CVs, that value may not be the same
replicated in experimental settings. as the DR2 over a model that does not con-
tain the new IV.
It is safer to include a Not aligned Aligned (but problematic) Aligned (but problematic)
questionable variable Including CVs that do not accurately reflect the Including all possible contextual influences will Including large numbers of context CVs
than to leave it out contaminant only interfere with appropriate partial these effects out of the resulting increases the chances that unique variance
interpretations. regression coefficients, but the more these will be assigned to the CV instead of theo-
variables covary with IVs, the more difficult retically meaningful variables. When context
the resulting bs are to interpret. effects covary with IVs, differentiating among
their effects becomes difficult.
Note: CV ¼ control variable; DV ¼ dependent variable; IV ¼ independent variable; MRA ¼ multiple regression analysis. b is standardized regression coefficient.
Carlson and Wu 421

Method
Our sample was drawn from studies published during 2007 in three highly respected journals. We
reviewed every article published in the Academy of Management Journal, Journal of Applied Psy-
chology, and Strategic Management Journal during that year. To be included in the analysis, an arti-
cle had to report the results of an empirical analysis that included at least one CV in the research
design. If an article reported the results for two or more studies or multiple analyses within studies,
we included the first covariance analysis in the article that included a CV. A total of 162 out of
266 articles (60.9%) met this criterion. Our analysis drew from two of the journals Becker (2005)
used to identify the 60 randomly selected articles examined in his review, Academy of Management
Journal (49 articles) and Journal of Applied Psychology (67 articles). To these, we added one new
journal, Strategic Management Journal (46 articles). Our analysis represents a subset of the 813
articles with CVs incorporated in Atinc et al.’s (2012) review of CV practice from 2005 to 2009,
based on four journals (they also included Journal of Management). We extend Atinc et al.’s
analysis of all CVs in each design from a subset of 40 articles with strong justifications to all
162 studies and analyses in our sample.
We coded for the number and type of CVs included in each study. We distinguished between the-
oretical (e.g., affect, munificence, social desirability) and artifact CVs (e.g., organization size, orga-
nization tenure CVs, industry, gender, year). We also coded the type of statistical analysis used. We
began by simply recording the type of analysis used and then organized the forms of analysis into
five categories to simplify reporting of results. Categories included (a) ANCOVA and related anal-
yses (comparison of means); (b) ordinary least squares (OLS) regression; (c) logit, probit, Poisson,
negative binomial, and related maximum likelihood methods; (d) panel data methods, irrespective of
the form of analysis, which were grouped separately because of the unique control issues often dis-
cussed in these analyses; and (e) SEM and HLM. Critical factors in developing categories were
whether the analysis produces an interpretable measure of R2 (e.g., OLS) or allows the calculation
of one (e.g., ANOVA, SEM, HLM) and whether it is possible to determine different roles for CVs
within the analysis by assigning paths or level of analysis (e.g., SEM, HLM).
We coded for whether the authors provided a complete justification for the inclusion of each CV.
We distinguished between studies that offered a complete or partial justification for the inclusion of
each CV in the design, including how it was measured, as well as recorded whether no justification
was provided. We also examined whether the authors included a discussion of their intended reason
for including CVs, and if so, we recorded the specific intended purpose offered.
Given the importance of CV-IV correlations for interpreting the results of studies that include
CVs, a study was coded 1 if all CVs were included in the correlation matrix or otherwise reported
in the manuscript. It was coded 0 if one or more CVs included in the study design (even if they were
not included in the analyses) were not reported. We also coded whether the authors provided a sep-
arate interpretation of CV results on the basis of the zero-order correlations or using the regression
coefficients. Studies were coded 1 if at least one CV relationship was discussed and 0 otherwise.
Finally, studies were coded for whether, as a result of an examination of a CV’s relationships with
other variables, one or more CVs were excluded from the data analysis. Studies that included text
discussing why CVs were excluded from the analysis on the basis of the magnitude of relationships
or their impact on results were coded 1; otherwise, 0.
Finally, to examine perceived isomorphism in CV practice, we coded each study for the five
‘‘universal CV playbook’’ behaviors. We coded whether the authors’ justification for including CVs
was based solely on the CVs’ relationship to the DV. We used either an explicit statement in the
Method section or whether text elsewhere in the article indicated the inclusion was based on per-
ceived, demonstrated, or potential relationship with the DV. If so, studies were coded 1. Studies were
coded 0 if it was not possible to determine the reason for including the CV or if an explicit or implied
422 Organizational Research Methods 15(3)

justification for any CV was based on its perceived, demonstrated, or potential relationship to an IV
(even if it was also likely to be associated with the DV).
To assess interrater agreement, two raters evaluated a sample of 50 studies. Rater agreement was
evaluated as the percentage of matching ratings across raters. Initial rater agreement ranged from a
low of 72% (justification based solely on DV) to 98% (data for all CVs were reported in the study).
Disagreements were resolved by a reexamination of the manuscript by the first author. The primary
source of the disagreements of ratings was the placement of CV-relevant data in the manuscript.
Whereas some authors consolidated CV-relevant data at the end of the Method and beginning of the
Results sections, in other studies, CV-relevant data could be found throughout the article, requiring
careful reading of the entire manuscript.
To examine the substantive effects of CVs on results, we examined the magnitude of relation-
ships for all reported CVs in a two-thirds subsample of the studies (116), drawn proportionately from
the three journals. We identified unique combinations of DVs, IVs, and CVs in each study on the
basis of the author’s assigned use of each variable. Each combination represents a three-variable sys-
tem within which the effects of a CV can be isolated and understood by comparing the zero-order
correlation between the IV and DV to the magnitude of b for the IV. The magnitude of each correla-
tion between IV and DV (rxy), DV and CV (ryz), and IV and CV (rxz) was recorded using the zero-
order correlation matrix reported in each study. We were specifically interested in the magnitude of
rxz correlations, given their potential to influence the magnitudes of regression coefficients. How-
ever, rxy and ryz correlations are important, as they provide the context for understanding the impact
of meaningful rxz correlations. Then using Equation 2 and data on the component correlations, we
calculated a b for the IV in each three-variable system and compared it to the magnitude of the rxy,
recording both the absolute value and signed difference.

Results
Table 2 reports the primary results of our descriptive analysis of CV practice. The 162 studies in our
sample incorporated 2,864 CVs, or M ¼ 17.7 CVs per study (SD ¼ 65.42). This number is skewed by
the extremely large numbers of CVs (maximum of 682) involving year, firm, or industry effects in
analyses using panel data. Excluding these CVs results in M ¼ 6.6 (SD ¼ 4.48) CVs per study. The
Academy of Management Journal and Strategic Management Journal had similar mean numbers of
CVs per study, 7.7 (SD ¼ 5.06) and 8.7 (SD ¼ 4.56), respectively, whereas the mean for the Journal
of Applied Psychology was 4.3 (SD ¼ 2.61). Theoretically meaningful CVs, those based on a con-
struct drawn from theory, were much less likely to be used across studies (M ¼ 2.1, SD ¼ 3.00) than
were artifact CVs (M ¼ 15.6, SD ¼ 65.42). Overall ANCOVA, although occurring only four times,
involved the fewest CVs (M ¼ 1.8). Similar average numbers of CVs per study occurred in OLS
(M ¼ 7.2 across 76 studies), maximum likelihood analyses (M ¼ 9.0 across 22 studies), and SEM
and HLM (M ¼ 6.5 across 37 analyses). The outlier was analyses using panel data methods. As noted
above, these incorporated by far the largest numbers of CVs (M ¼ 80.6 across 23 analyses).
Becker (2005) noted that a larger number of studies in his analysis did not provide an adequate
justification for the inclusion of CVs and details on how they were measured. This was less true for
the studies we examined, which were published 2 years after Becker’s review. In our sample, the
vast majority (81%) provided a complete explanation for why CVs were included in their analyses
as well as descriptions of how they were measured. An additional 8% provided partial descriptions.
Only 12% of articles provided no explanation for why any of the CVs were included, simply noting
their inclusion.
Information on the authors’ intent for using CVs, however, was not well reported. Only 62 arti-
cles (38%) provided some statement of the intended reason for including CVs, and this varied by
journal (Academy of Management Journal, 39%; Journal of Applied Psychology, 18%; and Strategic
Carlson and Wu 423

Table 2. Analysis of Control Variable Practice

Category AMJ JAP SMJ Overall

Total articles published 59 136 71 266

Articles with CVs (%) 49 (83%) 67 (49%) 46 (65%) 162 (61%)
CVs per study
Total
M (SD) 28.3 (97.85) 11.6 (56.20) 15.2 (17.77) 17.7 (65.42)
Max 684 462 104 684
Without panel data CVs
M (SD) 7.7 (5.06) 4.3 (2.61) 8.7 (4.56) 6.6 (4.48)
Max 25 12 24 25
Type of CV
Theoretical CVs per study
M (SD) 2.3 (2.56) 1.1 (4.21) 3.3 (1.68) 2.1 (3.00)
Artifact CVs per study
M (SD) 26.0 (97.8) 10.6 (17.22) 11.8 (56.11) 15.6 (65.32)
Type of analysis
ANOVA (and related analyses) 2 2 — 4
M 1.5 2.0 — 1.8
Ordinary least squares regression 17 37 22 76
M 7.5 5.0 10.7 7.2
Maximum likelihood analysis 11 4 7 22
M 9.1 6.8 10.1 9.0
Panel data 10 1 12 23
M 106.7 462.0 27.0 80.6
Structural equation modeling, hierarchical 9 23 5 37
linear modeling
M 10.4 4.4 9.0 6.5
Justification for inclusion
Full 39 (80%) 56 (84%) 37 (80%) 132 (81%)
Partial 4 (8%) 2 (3%) 6 (13%) 12 (8%)
None 6 (12%) 9 (13%) 3 (7%) 18 (11%)
Intent for including CVs
Studies indicating intent 19 (39%) 14 (18%) 29 (63%) 62 (38%)
Total intent statements 26 15 44 85
Purification 1 0 4 5
Incremental validity 3 5 0 8
Accounting for other variables
Confounding 2 5 2 9
Avoiding or removing bias 2 6 0 8
Isolating an effect 4 4 3 10
Accounting for alternative explanations 9 6 3 18
Other 5 14 7 26
‘‘Universal CV playbook’’
Justifications based on CV-DV relationship 43 (88%) 54 (81%) 43 (93%) 140 (86%)
More CVs are preferred 33 (67%) 9 (13%) 37 (80%) 79 (49%)
Incorporate all CVs in the regression model 49 (100%) 61 (91%) 45 (98%) 155 (96%)
CVs first in hierarchical analyses 46 (94%) 62 (93%) 45 (98%) 153 (94%)
Interpret regression coefficients for the 43 (88%) 57 (85%) 42 (91%) 142 (88%)
model including all study variables
Reporting and reviewing CVs
Data for all CV correlations reported 35 (71%) 49 (73%) 25 (54%) 109 (67%)
Reviewed CV correlations or coefficients 21 (43%) 38 (57%) 19 (41%) 78 (42%)
Did not include after review 0 (0%) 13 (19%) 0 (0%) 13 (8%)

Note: AMJ ¼ Academy of Management Journal; JAP ¼ Journal of Applied Psychology; SMJ ¼ Strategic Management Journal; CV ¼
control variable; DV ¼ dependent variable.
424 Organizational Research Methods 15(3)

Management Journal, 63%). However, this included 85 separate statements of intent, with 16 studies
including at least two different statements of intent and a single study that incorporated five different
intent statements each associated with different CVs. Of the 85 statements of intent, purification and
incremental validity were infrequent, indicated in 5 and 8 studies, respectively. The majority (72)
involved attempts to account for other variables. However, these were scattered across 19 different
statements. The most common was accounting for alternative explanations (18), followed by isolat-
ing effects (11), avoiding confounding (9), and reducing bias (8). Additional explanations included
avoiding spurious findings (4), statistically controlling or partialling effects (4), providing better or
more accurate results (4), and removing noise in the DV (4).
With respect to the reporting of CV data, about two thirds of studies (67%) reported data on the
means, standard deviations, and zero-order correlations for all study variables. However, only 48%
of studies include any discussion of the magnitudes of regression coefficients associated with CVs,
and only 13 studies (8%) actually chose not to include CVs after a preliminary review of the cor-
relations of CVs with other study variables. These studies were all published in the Journal of
Applied Psychology and resulted from recognition of small correlations between CVs and other
study variables or a comparison of alternative analyses where results for models including CVs
produced ‘‘similar’’ results to those for models without CVs, and so the CV containing models
were not reported.
Some support exists for the perception that CV practice tends to be common across studies. The
vast majority of studies (86%) based their selection of CVs according to their relationship with the
studies’ DVs. Nearly all studies (96%) incorporated all CVs into their regression models, with
almost as many (94%) incorporating CVs in all models or first in hierarchical analyses. The vast
majority (88%) based their interpretations of results primarily, or exclusively, on regression coeffi-
cients from analysis including all CVs. However, a preference for more CVs was less common. Our
analysis focused on studies that incorporated large numbers (four or more) of artifact CVs. Only
49% of studies indicated a preference for more rather than fewer CVs, with this occurring least fre-
quently in the Journal of Applied Psychology (9%).

Presence of Meaningful Correlations Between IVs and CVs

We were able to identify 5,009 unique DV-IV-CV combinations. Table 3 reports data on the mag-
nitudes of the component zero-order correlations (i.e., rxy, ryz, rxz) for both the simple average (Mobs)
and the average of the absolute values (Mabs). The analysis included a total of N ¼ 972 unique rxy
correlations (Mabs ¼ .20, SD ¼ .160), N ¼ 1,320 unique ryz correlations (Mabs ¼ .13, SD ¼ .132), and
N ¼ 2,810 unique rxz correlations (Mabs ¼ .12, SD ¼ .125). Several details of this analysis are note-
worthy. First, the average rxz is small (rxz ¼ .12), suggesting, on the basis of Equation 2, that in many
instances, CVs may in fact have little impact on outcomes. However, although the mean is small, the
standard deviations suggest that some correlations are substantial. In fact, as Table 3 indicates, more
than 17% of ryz correlations have absolute values greater than .20. Second, the average value of ryz is
also small, r ¼ .13. This is interesting given that standard practice uses the value of ryz to justify the
inclusion of CVs in research designs. However, here again, substantial variation exists.
Because the average rxz and ryz correlations in Table 3 are both small, a large percentage of the
resulting regression coefficients would be expected to not differ substantially from the underlying
zero-order correlations. This is confirmed in Table 4, as 3,922 of the 5,009 IV-DV-CV combinations
(78.3%) produced standardized regression coefficients that are different by less than |.02| from the
underlying zero-order correlation. In another 653 cases (13.0%), the difference was only marginally
larger, ranging from |.02| to |.05|. Bedeian, Sturman, and Streiner (2009) argued that Dr ¼ .1 repre-
sents the difference in the magnitude of correlations at which individuals begin to perceive substan-
tive differences. Using this standard, in more than 96% of 5,009 instances where CVs could affect
Carlson and Wu 425

Table 3. Numbers and Magnitudes of Component Correlations Identified in the Sample of Studies Examined

Unique Mobs Mabs |.05| |.10| |.11-.20| |.21-.40| >|.40|

Correlation Instances (SD) (SD) (%) (%) (%) (%) (%)

rxy 972 .11 .20 195 324 250 294 104

(.232) (.160) (20.0) (33.3) (25.7) (30.2) (10.7)
ryz 1,320 .05 .13 438 718 305 239 58
(.181) (.132) (33.2) (54.4) (23.1) (18.1) (4.4)
rxz 2,810 .04 .12 1,134 1,698 624 384 104
(.166) (.125) (40.4) (60.4) (22.2) (13.7) (3.7)
Note: rxz, ryz, and rxz refer to zero-order correlations, where the subscripts refer to independent variables (x), dependent
variables (y), and control variables (z). Mobs is the mean of the observed zero-order correlations (combining both negative
and positive correlations). Mabs is the mean of the absolute values of the correlations.

Table 4. Effect of Control Variables on the Magnitude of the Standardized Regression Coefficient (b) as
Compared to the Zero-Order Correlation (rxy>) for Unique Three-Variable (DV-IV-CV) Combinations

Magnitude Magnitude
Difference Between Number Decreased Increased Sign of b
b and rxy of Instances b < rxy b > rxy Differs From rxy

<|.02| 3,922 (78.3%) 2,135 1,784 13

<|.05| 4,575 (91.3%) 2,530 2,044 36
<|.10| 4,854 (96.9%) 2,706 2,148 57
|.10-.20| 117 (2.3%) 88 29 13
|.20-.30| 24 (0.5%) 15 9 7
>|.30| 14 (0.3%) 7 7 4
Total 5,009 2,816 2,193 81
Note: Three-variable (DV-IV-CV) combinations refer to the unique pairings of a dependent variable, an independent variable,
and a control variable. Each study contributes multiple three-variable combinations to this analysis depending on the number
of each type of variable included in each study’s design (i.e., number of combinations ¼ DVs  IVs  CVs). b is the stan-
dardized regression coefficient for the relationship between the IV and DV when the CV is included in the model; rxy is the
zero-order correlation between the IV and DV.

results in the three-variable systems, the presence of the CV would have been judged to have no
substantive effect on results.
Although they were infrequent, there were instances where CVs produced substantial differences
between the regression coefficient and the underlying zero-order correlations. In 155 instances
(3.1%), the magnitude of b for the IV in each three-variable system differed by .10 or more from
rxy, and in 24 of these, b and rxy had different signs (þ and –). To determine whether these differ-
ences affected the actual reporting of results and interpretation of findings in the original studies, we
reviewed the 16 studies that involved the 48 most extreme effects—those involving differences of
more than |b – rxy|  .20 and any difference greater than |.10| that also resulted in a sign change (i.e.,
þ or –). In most cases, these effects did not affect the statistical significance or sign of the reported
findings or the author’s interpretations. In only 14 instances (representing 6 of the 116 studies in our
analysis) did we find evidence that CV effects had a substantive impact on the reporting or interpre-
tation of a study’s findings. These 14 are summarized in Table 5. For each, we report the results of
our analysis of the relevant three-variable system, the regression coefficient reported by the authors,
and descriptive comments from the reporting and interpretation of results drawn from the original
study. Our intent here is not to criticize the work of these authors but to highlight how even in the
426
Table 5. Examples of the Impact Control Variables (CVs) Can Have on the Magnitudes of Reported Research Findings as Estimated in Our Analysis of CV Effects (Est b) and
as Reported in the Study by the Authors (Rep b) as Compared to the Zero-Order Relationships Between Independent Variables (IVs) and Dependent Variables (DVs) (rxy)
and the Authors’ Interpretations

Citation Relationship (DV/IV/CV) rxy Est b Rep ba Authors’ Interpretations

Burt (2007) Salary/Direct network constraint/Job rank –.57 –.07 –.17 The presence of Job Rank as a control variable
Salary/Indirect network constraint/Job rank –.39 –.03 –.08 (ns) dramatically influences the magnitudes of
Annual evaluation/Indirect network con- –.111 .04 .09 (ns) results and changing the significance of
straint/Job rank indirect network constraint. ‘‘Returns to
direct brokerage [direct network constraint]
are strong. No returns are attributed to
secondhand brokerage [indirect network
constraint].... Job rank is the key control
variable. Managers in more senior positions
had more direct and secondhand brokerage
in the sense that their direct personal con-
tacts were people in separate groups who
managed work across groups of lower-rank
people (–.58 and –.41 correlations in Table
2 for job rank with direct network con-
straint and indirect network constraint).
Holding only job rank constant eliminates
the Figure 4 returns to secondhand
brokerage for compensation (–9.1 t-test in
Figure 4a drops to –1.2) and performance
evaluation (–2.5 t-test in Figure 4b drops to
0.6)’’ (pp. 130-131).

(continued)
 Table 5. (continued)

Citation Relationship (DV/IV/CV) rxy Est b Rep ba Authors’ Interpretations

Chen, Su, & Tsai (2007) Perceived competitive tensions, insiders/ .26 –.03 .09 Reports unstandardized regression coeffi-
Relative scale/No. routes not served cients for negative binomial regression. ‘‘A
Perceived competitive tensions, outsiders/ .32 –.04 .07 close look at the VIF [variance inflation fac-
Relative scale/No. routes not served tor] suggests that slightly high intercorrela-
Attack on rivals markets/Relative scale/No. –.18 –.67 –.07 tions did occur but only among some of our
routes not served control variables. However, the significance
levels of our results remained the same
whether or not we entered the correlated
control variables in our analysis. Indeed,
multicollinearity did not affect our model fit
and hypothesis testing’’ (p. 111).
Dierdorff & Morgeson (2007) Responsibility requirements /Autonomy/ .42 .05 –.07 ‘‘All three control variables were highly asso-
Job zone ciated with autonomy. . . . Autonomy was
Responsibility requirements /Autonomy/ .42 .14 predicted . . . to be unrelated to responsi-
Education bility requirements (Hypothesis 3b). . . . No
significant relationship was found for con-
sensus in responsibility requirements (b ¼
–.07, p ¼ .58)’’ (p. 1236). The regression
results supported the hypothesis of no
relationship, but the zero-order correlation
between autonomy and responsibility
requirements was positive and substantial
(r ¼ .42).
Lavie, Lechner, & Singh (2007) Market success/Timing of entry/Duration –.43 –.21 –.06 Negative binomial regression. ‘‘Table 3 shows
of membership that timing of entry produced a negative
Market exposure/Timing of entry/Duration –.35 .32 .001 (ns) linear effect on market success in model 4b
of membership (b ¼ –0.06, p < .01) and no effect on market
exposure in model 4c’’ (p. 593).

(continued)

427
428
Table 5. (continued)
Citation Relationship (DV/IV/CV) rxy Est b Rep ba Authors’ Interpretations

Marquis & Lounsbury (2007) No. foundings per community per year/ –.14 .20 .98 Poisson regression. ‘‘The concentration vari-
Four-firm concentration ratio/Banking able hypothesized to have a positive effect
institutions in Metropolitan Statistical on foundings in Hypothesis 2 is positive and
Areas (MSA) statistically significant at the 10 percent
No. foundings per community per year/No. .32 –.10 .0003 (ns) level (two-tailed test). . . . The effect of
bank professionals/Local savings local bank professionals on foundings is not
statistically significant; hence, Hypothesis 4
is not supported. As noted, however,
because this measure underestimates the
human capital in a community, it is likely a
conservative test of our claims’’ (p. 809).
Pastor, Mayo, & Shamir (2007) Charisma/Time 1 arousal/Time 2 charisma .33 .08 .21 ‘‘As expected, arousal at Time 1 showed a
Transactional leadership/ Time 1 arousal/ .77 .55 –.02 significant beta coefficient predicting ratings
Time 2 charisma of charisma at Time 2 (b ¼ .21, p < .01). We
also ran the regression without the statisti-
cal control of transactional leadership and
obtained similar results (Model 3; b ¼ .38,
p < .01)’’ (pp. 1590-1591). ‘‘Arousal at Time
1 had a statistically significant beta coeffi-
cient (b ¼ .21, p < .01) predicting ratings of
charisma at Time 2, whereas the corre-
sponding beta coefficient for ratings of
transactional leadership was not statistically
significant (b ¼ –.02). A z test showed that
the difference between these two coeffi-
cients was statistically significant (z ¼ 1.98, p
< .05)’’ (p. 1592).
Note: rxy is the zero-order correlation between the IV and the DV; Est b refers to the an estimate of the regression coefficient relating the IV to the DV controlling for the CV, based on Equation 2;
Rep ba refers to the regression coefficient for the same IV-DV relationship reported in the study but ordinary least squares regression coefficients unless otherwise stated. Rep b reflects the effects
of all CVs in the model, whereas Est b reflects the single CV likely to most directly affect results.
Carlson and Wu 429

best management research, CV effects can lead to unintended or unrecognized consequences. This
table not only highlights how infrequently the CVs in these studies had a substantive impact on
outcomes, but the challenge that exists is identifying CV effects. This is particularly difficult when
studies report coefficients for unstandardized coefficients, when they involve IV or DV transforma-
tions (e.g., logistic or negative binomial regression or latent path coefficients from structural equa-
tions modeling), and when the effects of different CVs (and other IVs) are combined.

Discussion
Newcombe (2003) argues that researchers ‘‘worship control’’ (p. 1050). What seems clear from the
evidence to this point is this: (a) Quasi- and nonexperimental designs are common in the manage-
ment literature; (b) including CVs in MRA is the dominant method used to achieve statistical con-
trol; (c) the nature of the expected or intended control effect either is not described or is ambiguous;
(d) when intent is articulated, much of current CV practice is poorly aligned with the major purposes
for which statistical control might be used (Becker, 2005; Breaugh, 2005, 2008; Breaugh & Arnold,
2007; Spector & Brannick, 2011); and (e) to date, despite its documented problems, the widespread
use of misaligned CV practices does not appear to create widespread problems for interpretations of
the findings of management research.
Our analysis demonstrates that researchers, editors, and reviewers of the journals we reviewed
have adopted Becker’s (2005) call for better reporting of CV data and the justifications for including
CVs. But these justifications focus primarily on the CVs relationship to the DV, an important rela-
tionship but one that has little bearing on statistical control in MRA. Statements providing justifica-
tion for the need for using statistical control occurred in fewer than half the studies, and in the
majority of these, the stated intent was ambiguous. This creates challenges for interpreting regres-
sion coefficients. Appropriate statistical control requires both an understanding of the intent of the
control and a determination of whether the CVs included actually provided that control. In the
absence of these checks on CV practice, it is not clear that statistically ‘‘controlled’’ studies pro-
vide results that are more meaningful in the intended way than those that do not incorporate CVs.
This leaves open the possibility for inadvertent misinterpretations when including CVs generates
unexpected findings. Furthermore, although we found no evidence to suggest this occurred in the
studies we reviewed, the ambiguity creates the potential for statistical legerdemain by researchers.
Thus vigilance by editors and reviewers is required.
Our review left us with the impression that authors, and perhaps editors and reviewers, believe
that including CVs achieves an omnibus form of control that always improves the rigor and interpret-
ability of results. This may be true in specific instances, but the absence of unambiguous statements of
intent provides little reason to believe this is always true. Furthermore, the notion that including more
CVs offers a more conservative or rigorous research design can and should be challenged. MRA
models that contain large numbers of CVs create coefficients with complex definitions. As the
number of CVs increases, the potential for mixing CVs associated with different statistical control
purposes creates increased potential for the effects of intended controls to be confounded. The
assumption that the inclusion of larger numbers of CVs automatically improves the interpretability
of results is hard to justify.
In contrast to legitimate concerns about ambiguous intent and misaligned practice, our analysis
found that few CVs included in designs appear to have any substantive impact on results. If statis-
tical control of any type is in fact important, the majority of CVs included in the sample of 116
studies examined here have little effect. In 77% of the DV-IV-CV three-variable systems, differ-
ence between the calculated standardized regression coefficient and the zero-order correlation was
less than |.02|. In these circumstances, the CV contributes little to the research design, offering the
pretense of control when in fact little actual control occurs.
430 Organizational Research Methods 15(3)

The accomplishment of statistical control in nonexperimental designs centers on the presence

of meaningful IV-CV correlations. Had these CVs been more strongly associated with IVs where
substantive control was desired, more substantive control would have been possible. However,
these are the very conditions that increase the potential for substantive misinterpretation of
research findings. To be clear, current CV practice can cause problems, particularly with respect
to appropriate interpretation of regression coefficients. Breaugh (2008) offers a number of
examples where the effects of CVs on the results of research designs led authors to inappropriate
interpretations of results. To this list, we add the results from the six studies identified in our
analysis. But these problems, as evidenced in these data, appear to occur infrequently. That is
the good news.
That so many CVs included in published research have little substantive effect supports the per-
ception that researchers apply very low evidentiary hurdles for the inclusion of CVs in research
designs. This is consistent with a belief that it is better to include a marginally relevant CV than
to leave out a potentially important one. On the surface, this strategy appears safe. But as former
Ohio State University coach Woody Hayes, who was noted for his conservative approach to the
game of football, is quoted as saying, ‘‘three things that can happen when you throw and two of them
are bad’’ (Rosenberg, 2008, p. 274). The same is true for including CVs in research designs. The one
good outcome occurs when one includes an appropriate CV and, by doing so, achieves the desired
control and increases the interpretability of the study’s findings. The second, not-so-bad outcome is
when a CV is included that has no impact on results. It increases costs, extends the effort required to
develop data and analyze and interpret results, and consumes more journal space but does not alter
the substantive findings. The third outcome is when a CV inappropriately (and unexpectedly) alters
results, generating ambiguous results without clear interpretations. Given that only the first outcome
offers a substantive benefit, adopting more stringent hurdles for identifying and including meaning-
ful CVs may be appropriate.

Barriers to Recognizing Inappropriate CV Effects

The limited impact of CVs on research outcomes likely reduces our vigilance for identifying
instances where CVs create problems for interpreting results. However, factors beyond low base rate
also contribute to difficulties in recognizing potential problems. First, although our analysis isolated
effects in each three-variable system, analysis-level results combine effects across IVs and CVs, as
well as including higher-order variables and interactions. The combinations of effects often results
in the tempering of more extreme effects. Of the 48 most extreme effects in our three-variable sys-
tems, only 6 resulted in substantive effects for IVs when the studies’ other IVs and CVs were
included in the analysis. However, combining several CVs with moderate correlations could poten-
tially generate large effects on regression coefficients, particularly when the magnitude of CV-DV
correlations approaches or exceeds that of IV-DV correlations.
Second, the scaling of variables can make problems difficult to recognize. The impact of CVs is
easily identified for standardized regression coefficients—these are in the same metric as the zero-
order correlation. Unstandardized regression coefficients, which are in the metric of the original
IV, are not directly comparable with zero-order correlations, making the impact of CVs harder to
see. A similar problem occurs with alternative regression methodologies, such as logistic or neg-
ative binomial regression, which involve a nonlinear transformation of the DV. As can be seen in
Table 5, in these instances, comparing zero-order correlations to regression coefficients offers lit-
tle insight into the existence of CV effects. In addition, basing interpretations on the statistical sig-
nificance of results further masks these effects. In many instances where CVs produce a
substantive impact on the magnitude of regression coefficients, these effects may not change the
statistical significance of findings.
Carlson and Wu 431

Table 6. Consolidated Recommendations for Improved Control Variable (CV) Practice

Recommendation

1. Adopt a Conservative Stance to the Use of CVs

Adding more CVs does not make a study more rigorous. Unless there is a very sound reason that including a
specific CV accomplishes an unambiguous and meaningfully statistical control objective, studies adding CVs
may confound, rather than enhance, the interpretations of findings. When in doubt, leave them out.
2. Offer a Complete Reporting of CV Methods (Becker, 2005)
Describe the purpose(s) for using CVs. Identify the specific CVs to be included and justification as well as the
intended statistical control objective. Ambiguous justifications should be avoided.
3. Align Methods/Analysis to Purposes
One size does not fit all. Match the selection of CVs, accompanying analyses, and reporting and interpretation
of results to the stated purpose. When the effects of CVs meaningfully affect results, report results with and
without CVs (Becker, 2005).
4. Report Data for All CVs Included in the Research Design (Becker, 2005)
Report the mean, standard deviation (or other measure of dispersion), reliability (if appropriate), and zero-
order correlations with all other variables for all CVs in the research design, even if they are not included in
analyses.
5. Review CV Zero-Order Correlations Before Proceeding to Data Analyses
Review all zero-order correlations before conducting analyses to highlight CV correlations that may influ-
ence findings as an advance warning of potential anomalies in findings and to remove CVs that will have no
impact on results.
6. Match the Analysis to the Hypothesis (Spector & Brannick, 2011)
If the hypothesis statement indicates a relationship exists between Variable X and Variable Y, but does not
mention CVs, the appropriate analysis to test the hypothesis is one that does not include CVs.
7. Distinguish Between Theoretical and Artifact CVs in Hierarchical Analyses
Distinguish between theoretically meaningful CVs (those that provide theory-based explanations) and arti-
fact CVs (e.g., size, gender, industry, company, etc.) that may be associated with dependent variables but
provide no explanation for why the association exists. Theoretically meaningful variables (CVs and inde-
pendent variables) should be given the first opportunity to account for variance in outcomes because they
offer explanations. Artifact CVs should be entered last.

Recommendations for Practice

We offer several new recommendations for improving research practice. These recommendations,
recapped in Table 6, are based on this review and build from the work of previous scholars in this
area (Becker, 2005; Breaugh, 2005; 2008; Spector & Brannick, 2011). Many of the problems asso-
ciated with current CV practice appear to result from confusion surrounding how statistical control is
accomplished in MRA and related analyses and ambiguous statistical control objectives. Purpose-
fully matching CV practices with these objectives is critical, and we believe this should be a respon-
sibility of authors as well as reviewers and editors.
First, we recommend that researchers, editors, and reviewers establish higher standards for the deci-
sion to include CVs in nonexperimental and quasiexperimental designs. A total of 48 nonexperimental
designs in published studies we reviewed included no CVs. These data suggest, despite the relatively
modest number of CVs per study, that researchers set a low hurdle for the inclusion of CVs in research
designs. This is consistent with a belief that including marginally relevant CVs is preferable to leaving
an important CV out of the design. However, given the potential for substantive CV-IV correlations to
affect findings, and concerns about the capacity to effectively accomplish two of the three purposes for
using statistical control, a more appropriate and conservative stance may be to adopt stronger standards
for attempting statistical control using CVs in research designs, and when ambiguity about the intent or
efficacy of controls raises doubts, researchers should be encouraged to leave them out.
432 Organizational Research Methods 15(3)

Second, extending Becker’s (2005) recommendation for greater reporting of decision making
related to CVs, we recommend that researchers be explicit in identifying the specific statistical con-
trol objectives adopted for their study. Researchers should be required to offer explicit statistical
control objectives and evidence that the CVs used accomplish the intended control. Statements that
indicate removing bias, avoiding confounding, or making estimates more accurate do not provide
sufficient guidance to determine whether the intended statistical control objectives are met.
Third, match the purpose of statistical control to the intended reason to desire statistical control.
One size does not fit all. Purification and incremental prediction have different objectives and
require different analyses. Match the analysis and inclusion of CVs to assure the specified statistical
control objective is accomplished. Combining CVs associated with different statistical control pur-
poses can confound results.
Fourth, report zero-order correlations before conducting analyses even if they are not incorpo-
rated in the data analysis. Fifth, review descriptive data for all CVs before conducting analyses. This
highlights CV correlations that may influence findings and offers an advance warning of potential
anomalies in findings. In addition, any CV that has no correlation with any other study variable of
r ¼ .10 or greater can effectively be left out of the analysis without substantively influencing results.
There may be theoretical reasons to still include these variables, and in those circumstances, authors
may choose to do so. However, as previously noted by Becker (2005), including CVs that do not
affect results simply reduces the degrees of freedom available in the analysis and expands the space
required to report results without adding value. We were surprised by our findings regarding how
little impact CVs had on research findings, particularly, the large number of very small relationships
that existed between CVs and other study variables.
Sixth, as argued effectively by Spector and Brannick (2011), analyses should match the hypoth-
eses. They address the practice of including CVs in hypothesis tests when the hypothesis statement
makes no mention of CVs. This highlights the importance of recognizing that regression coefficients
reflect specific partialled relationships. If CVs are included in regression models, the interpretations
of the resulting coefficients should recognize the presence of the CVs (and other variables) included.
Depicting these coefficients as the relationship of X with Y ignores the ‘‘controlled’’ variables. We
further concur with Becker’s (2005) recommendation to report results with and without CVs when
the inclusion of CVs meaningfully affects the magnitude of results.
Seventh, researchers should distinguish between theoretically meaningful CVs (those that pro-
vide an explanation for why relationships occur) and artifact CVs (e.g., size, industry, company, etc.)
that may be associated with DVs but provide no explanation for why the association exists. The addi-
tion of artifact CVs creates more difficult challenges for interpretations of regression coefficients.
Furthermore, in incremental prediction analyses, researchers are encouraged to add context CVs
in the last step in hierarchical analyses. As noted earlier, theoretically meaningful variables (IVs and
CVs) should be given the first opportunity to account for variance in outcomes because they are
associated with explanations. Entering artifact CVs last determines whether they still increment pre-
diction beyond all theoretically meaningful variables.

Conclusion
CVs are the Rodney Dangerfield of research design. Like the comedic actor, they get no respect.
Decisions about the inclusion of CVs may be designed not to enhance the meaning of findings but
to placate future reviewers. As a result, many CVs included in research designs have no impact on
findings, but if they did, the mismatch of current practice to statistical control purpose is more likely
to confound rather than clarify research findings. The potential risk in current practice is the illusion
of rigor offered by including CVs. At the center of statistical control reform is abandoning an iso-
morphic ‘‘one-size-fits-all’’ approaches to CV practice and adopting specific practices designed to
Carlson and Wu 433

meet each articulated statistical control objective. The challenges of statistical control center not on
the inherent capabilities of MRA or other tools but on our skillful use of these tools to accomplish
our intended outcomes, a responsibility of authors as well as reviewers. As Cohen et al. (2003) note,
skillful intentional use of these tools appears capable of offering useful insights. However, it also is
clear that less purposeful use is equally capable of obfuscation, with its concomitant effects on inter-
pretations. As a result, in the absence of specific intent and clear evidence that including CVs pro-
duces that control, researchers may be better served by leaving CVs out of research designs.

Acknowledgments
The authors thank Paul Spector, Michael Brannick, and Jim Breaugh for their thoughtful feedback on earlier
drafts of this manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or
publication of this article.

Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.

References
A list of the studies included the analysis is available from the first author.
Atinc, G., Simmering, M. J., & Kroll, M. J. (2012). Control variable use and reporting in macro and micro
management research. Organizational Research Methods, 15, 57-74.
Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological
research: Conceptual, strategic and statistical considerations. Journal of Personality and Social Psychology,
51, 1173-1182.
Beatty, M. J. (2003). Do we know a vector from a scalar? Why measures of association (not their squares) are
appropriate indices of effect. Human Communications Research, 28, 605-611.
Becker, T. E. (2005). Potential problems in the statistical control of variables in organizational research: A qua-
litative analysis with recommendations. Organizational Research Methods, 8, 274-289.
Bedeian, A. G., Sturman, M. C., & Streiner, D. L. (2009). Decimal dust, significant digits, and the search for
stars. Organizational Research Methods, 12, 687-694.
Bercovitz, J., & Mitchell, W. (2007). When is more better? The impact of business scale and scope on long-term
business survival, while controlling for profitability. Strategic Management Journal, 28(1), 61-79.
Breaugh, J. A. (2005). Rethinking the control of nuisance variables in theory testing. Journal of Business and
Psychology, 20, 429-443.
Breaugh, J. A. (2008). Important considerations in using statistical control for nuisance variables in
non-experimental studies. Human Resource Management Review, 18, 282-293.
Breaugh, J. A., & Arnold, J. (2007). Controlling nuisance variables using a matched-groups design. Organiza-
tional Research Methods, 10, 523-541.
Burks, B. S. (1926). On the inadequacy of the partial and multiple partial correlation technique. Journal of
Educational Psychology, 17, 625-630.
Burt, R. S. (2007). Secondhand brokerage: Evidence on the importance of local structure for managers, bankers,
and analysts. Academy of Management Journal, 50, 119-148.
Chen, M., Su, K., & Tsai, W. (2007). Competitive tension: The awareness-motivation-capability perspective.
Academy of Management Journal, 50, 101-118.
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for
the behavior sciences. Mahwah, NJ: Lawrence Erlbaum.
434 Organizational Research Methods 15(3)

Conger, A. J., & Jackson, D. N. (1972). Suppressor variables, prediction, and the interpretation of psychological
relationships. Educational and Psychological Measurement, 32, 579-599.
Christianfeld, N. J. S., Sloan, R. P., Carroll, D., & Greenland, S. (2004). Risk factors, confounding, and the
illusion of statistical control. Psychosomatic Medicine, 66, 868-875.
D’Andrade, R., & Dart, J. (1990). The interpretation of r versus r2 or why percent of variance accounted for is a
poor measure of effect size. Journal of Quantitative Anthropology, 2, 47-59.
Dierdorff, E. C., & Morgeson, F. P. (2007). Consensus in work role requirements: The influence of discrete
occupational context on role expectations. Journal of Applied Psychology, 92, 1228-1241.
Dohrenwend, B. S., Dohrenwend, B. P., Dodson, M., & Shrout, P. E. (1984). Symptoms, hassles, social sup-
ports and life events: Problem of confounded measures. Journal of Abnormal Psychology, 93, 220-230.
Doty, D. H., & Glick, W. H. (1998). Common methods bias: Does common methods variance really bias
results: An investigation of prevalence and effect. Organizational Research Methods, 1, 374-406.
Hays, W. L. (1988). Statistics (4th ed.). Fort Worth, TX: Holt Rinehart and Winston.
Hull, J. G., Tedlie, J. C., & Lehn, D. A. (1992). Moderator variables in personality research: The problem of
controlling for plausible alternatives. Personality and Social Psychology Bulletin, 18, 115-117.
Karaevli, A. (2007). Performance consequences of new CEO outsiderness: Moderating effects of pre- and post-
succession contexts. Strategic Management Journal, 28(7), 681-706.
Kerlinger, F. N., & Pedhazur, E. J. (1973). Multiple regression in behavioral research. New York: Holt,
Rinehart & Winston, Inc.
Lavie, D., Lechner, C., & Singh, H. (2007). The performance implications of timing of entry and involvement in
multipartner alliances. Academy of Management Journal, 50, 578-604.
Lazzarini, S. G. (2007). The impact of membership in competing alliance constellations: Evidence on the
operational performance of global airlines. Strategic Management Journal, 28(4), 345-367.
Marquis, C., & Lounsbury, M. (2007). Vive la resistance: Competing logics and the consolidation of US
community banking. Academy of Management Journal, 50, 799-820.
Meehl, P. E. (1970). Nuisance variables and the ex post facto design. In M. Radner & S. Winokur (Eds.),
Analyses of theories and methods of physics and psychology (pp. 373-402). Minneapolis: University of
Minnesota Press.
Meehl, P. E. (1971). High school yearbooks: A reply to Schwarz. Journal of Abnormal Psychology, 77,
143-148.
Nerkar, A., & Shane, S. (2007). Determinants of invention commercialization: An empirical examination of
academically sourced inventions. Strategic Management Journal, 28(11), 1155-1166.
Newcombe, N. S. (2003). Some controls control too much. Child Development, 74, 1050-1052.
Pastor, J. C., Mayo, M., & Shamir, B. (2007). Adding fuel to fire: The impact of followers’ arousal on ratings of
charisma. Journal of Applied Psychology, 92, 1584-1596.
Podsakoff, P. M., MacKenzie, S. G., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in
behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psy-
chology, 88, 879-903.
Rosenberg, M. (2008). War as they knew it. New York, NY: Grand Central.
Spector, P. E. (2006). Method variance in organization research: Truth or urban legend? Organizational
Research Methods, 9(2), 221-232.
Spector, P. E., & Brannick, M. T. (1995). The nature and effects of method variance in organizational research.
In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology
(Vol. 10, pp. 249-274). Wiley.
Spector, P. E., & Brannick, M. T. (2011). Methodological urban legends: The misuse of statistical control vari-
ables. Organizational Research Methods, 14, 287-305.
Spector, P. E., Zapf, D., Chen, P. Y., & Frese, M. (2000). Why negative affectivity should not be controlled in
job stress research: Don’t throw out the baby with the bath water. Journal of Organizational Behavior, 21,
79-95.
Carlson and Wu 435

Stone-Romero, E. F. (2007). Non-experimental designs. In S. Rogelsberg (Ed.), The encyclopedia of industrial

and organizational psychology (pp. 519-522). Beverly Hills, CA: Sage.
Tiwana, A., & Keil, M. (2007). Does peripheral knowledge complement control? An empirical test in technol-
ogy outsourcing alliances. Strategic Management Journal, 28(6), 623-634.
Williams, L. J., & Anderson, S. E. (1994). An alternative approach to method effects using latent-variable
models: Applications in organization behavior research. Journal of Applied Psychology, 79, 323-331.

Bios
Kevin D. Carlson is an Associate Professor of Management in the Pamplin College of Business at Virginia
Tech. He has published research on a wide variety of topics related to the measurement and evaluation of indi-
vidual, process and organizational effectiveness. His current research interests center on how research metho-
dology contributes to research progress.
Jinpei Wu is an Assistant Professor of Management in School of Business at Minnesota State University Moor-
head. He received his PhD in Management at Virginia Tech. His current research focuses on entrepreneurship,
emerging markets and business social responsibility.