Journal of

Board Publications Committee

Construction
Dennis L. Richards, P.E., D.WRE, F.ASCE,
Chair
Theodore Stathopoulos, Ph.D., P.Eng., F.ASCE
D. V. Griffiths, Ph.D., D.Sc., P.E., F.ASCE
Engineering and
Gordon Sterling, M.ASCE
Daniel L. Thomas, Ph.D., P.E., D.WRE,
F.ASCE
Kenneth L. Carper, M.ASCE,
Management
Corresponding Member VOLUME 136 / NUMBER 1 JANUARY 2010
Bruce Gossett, Aff.M.ASCE, ASCE Staff
Contact
SPECIAL ISSUE: Research Methodologies in Construction Engineering
Publications and Management
Bruce Gossett, Managing Director and
SPECIAL ISSUE EDITORS: John E. Taylor and Edward J. Jaselskis
Publisher
Introduction
Journals Department
1 Introduction to the Special Issue on Research Methodologies in Construction
Melissa Junior, Director, Journals
Engineering and Management
Kelly Anderson, Publishing Manager,
John E. Taylor and Edward Jaselskis
Journals
Holly Koppel, Managing Editor, Journals Technical Papers
Elizabeth Guertin, Publishing Manager,
Journals Quantitative Research Methodologies
Jennifer Parresol, Editorial Coordinator, 3 Methodology for Conducting Discrete-Event Simulation Studies in
Journals Construction Engineering and Management
Julio C. Martinez
Production Department
Angela Cochran, Director, Publications 17 Optimization Research: Enhancing the Robustness of Large-Scale
Production Multiobjective Optimization in Construction
Teresa Metcalfe, Senior Manager, Journals Amr Kandil, Khaled El-Rayes, and Omar El-Anwar
Production
Gene Sullivan, Senior Production Editor 26 Experimental Research in Construction
Nancy Green, Production Editor Leonhard E. Bernold and Tai Sik Lee
Rajashree Ranganathan, Production Editor
36 Empirical Modeling Methodologies for Construction
Xi Van Fleet, Manager, Information
Ian Flood and Raja R. A. Issa
Services
Donna Dickert, Reprints 49 Quantitative Research: Preparation of Incongruous Economic Data Sets for
Archival Data Analysis
Publishing Office Gunnar Lucko and Zane W. Mitchell Jr.
Journals Department
ASCE 58 Ethnographic Theory-Building Research in Construction
1801 Alexander Bell Drive Andreas F. Phelps and Michael J. Horman
Reston, VA 20191-4400
66 Charrettes as a Method for Engaging Industry in Best Practices Research
Telephone: 共703兲 295-6290
G. Edward Gibson Jr. and Donald A. Whittington
E-mail: [email protected]
76 Implementing Observational Research Methods to Study Team Performance
in Construction Management
Robert M. Leicht, Samuel T. Hunter, Chitwan Saluja, and John I. Messner

87 Action Research as a Proactive Research Method for Construction

Engineering and Management
Salman Azhar, Irtishad Ahmad, and Maung K. Sein

99 Qualitative Research: Application of the Delphi Method to CEM Research

Matthew R. Hallowell and John A. Gambatese

Downloaded 27 Dec 2009 to 158.42.244.25. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 VOLUME 136 / NUMBER 1
JANUARY 2010

Cross-Cutting Methodological Insights

108 Mixed Method Research: Fundamental Issues of Design, Validity, and

Reliability in Construction Research
Deborah A. Abowitz and T. Michael Toole

117 Contextualist Research: Iterating between Methods While Following an

Empirically Grounded Approach
Stuart D. Green, Chung-Chin Kao, and Graeme D. Larsen

127 Research Validation: Challenges and Opportunities in the Construction

Domain
Gunnar Lucko and Eddy M. Rojas

Downloaded 27 Dec 2009 to 158.42.244.25. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 INTRODUCTION

Introduction to the Special Issue on Research

Methodologies in Construction Engineering
and Management
John E. Taylor fied formal models. 共The number of articles increased during this
Guest Editor, Asst. Prof., Dept. of Civil Engineering and Engineering period due to an increase in the number of issues published per
Mechanics, Columbia Univ., 618 S.W. Mudd Bldg., 500 West 120th St., year.兲 Research that produces formal models—qualitative or
New York, NY 10027. E-mail: [email protected] quantitative—can provide the foundation for further formaliza-
tion, combination with other models, and development of new
Edward J. Jaselskis models with clearly argued contributions to knowledge. There-
Guest Editor, Prof., Dept. of Civil, Construction, and Environmental En- fore, the collective contributions to CEM knowledge are en-
gineering, Iowa State Univ., 456 Town Engineering Building, Ames, Iowa hanced by this trend toward more rigorous model development.
50011. E-mail: [email protected] With this special issue we aimed to capitalize on the trend
toward methodological rigor and comprehensiveness by soliciting
The purpose of this special issue is to initiate a dialog on the papers on methodologies we identified as being typically em-
rigorous application of research methodologies in construction ployed in CEM research. Contributing authors were asked to: 共1兲
engineering and management 共CEM兲 research. It is our hope that identify and discuss exemplar papers in the Journal that apply
engaging the CEM research community in such a dialog will have various methods; 共2兲 establish conventions for methodological
a positive impact on the methodological consistency of our col- rigor for each method as applied to CEM research; 共3兲 outline the
lective research endeavor. The application of research methodolo- unique challenges and opportunities researchers will face when
gies in the context of construction engineering and management applying each method in CEM; and 共4兲 present a path forward for
introduces unique challenges and creates new opportunities for future research applying each method. The response by the CEM
researchers. The interdisciplinary project-based nature of the research community to the special issue request for proposals far
work, industry fragmentation, one-of-a-kind end products, site- exceeded our expectations. Nearly 40 proposals were submitted to
based production, and other factors combine in ways that make develop research methodology articles for inclusion in the special
following established research methodologies difficult. issue. Due to space limitations in the Journal and methodological
Over the past fifteen years, the construction engineering and overlaps, we were limited to inviting 13 manuscripts.
management field has steadily improved the comprehensiveness This special issue includes five papers on rigorous application
and rigor of articles published. In preparation for this special of quantitative methods, five contributions on the rigorous appli-
issue, we examined 1,102 manuscripts published over the period cation of qualitative methods, and three cross-cutting papers that
from 1993 through 2007 in the Journal of Construction Engineer- provide methodological insights that apply to a range of research
ing and Management to examine trends in methodological utili- methodologies. Quantitative research methods covered include
zation and rigor. We found a broad range of methods in use by the papers on discrete event simulation by Julio Martinez; multiob-
CEM community. The relative frequency of each method is pre- jective optimization by Amr Kandil, Khaled El-Rayes, and Omar
sented in Table 1. We also observed an increasing trend in the El-Anwar; experimental research by Leonhard Bernold and Tai
specification of formal models over the period examined, which is Sik Lee; empirical modeling by Ian Flood and Raja R. A. Issa;
one indicator of the comprehensiveness and rigor of the methods and archival data analysis by Gunnar Lucko and Zane Mitchell. In
employed. In the period from 1993–1997, 70% of the 272 articles the area of qualitative research we have contributions on ethno-
specified formal models. Over the following five years, 72% of graphic theory building by Andreas Phelps and Michael Horman;
the 296 articles specified formal models. And over the remaining industry best practice charrettes by Edward Gibson and Donald
five-year period from 2003–2007, 76% of the 534 articles speci- Whittington; observational video research by Robert Leicht, Sam-
uel Hunter, Chitwan Saluja, and John Messner; action research by
Salman Azhar, Irtishad Ahmad, and Maung Sein; and the Delphi
Table 1. Research Methodologies Employed in the Journal of Construc- method by Matthew Hallowell and John Gambatese.
tion Engineering and Management 共1993–2007兲
Together the 10 aforementioned methodology papers provide a
Frequency of use comprehensive spectrum of the methods used by CEM research-
Research methodology 共n = 1 , 102兲 ers over the past 15 years. In addition to these, the special issue
Experimental research 27% includes three cross-cutting methodological contributions. Debo-
Survey research 22% rah Abowitz and Mike Toole provide a view on the fundamentals
Qualitative and quantitative case research 16% of social science research. Stuart Green, Chung-Chin Kao, and
Non-empirical theory building research 13%
Graeme Larsen discuss an empirically grounded approach to un-
derstanding enactment in CEM research. And finally, Gunnar
Simulation research 12%
Lucko and Eddy Rojas examine the challenges and opportunities
Optimization research 9%
of research validation in CEM. We would like to thank all of
Archival data set research ⬍1%
these authors who have developed what we believe is a collection

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 1

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
of papers that can have a meaningful impact on the collective James T. O’Connor, The University of Texas at Austin
research and contributions to knowledge in the CEM domain. Lucio Soibelman, Carnegie Mellon University
We were supported in the strategic development of this special Michael C. Vorster, Virginia Tech
issue and the review of manuscripts by a special issue Board of
Advisors. We would like to express our gratitude to the following We would like to further thank the Senior Editorial Board
thought leaders from the CEM academic community who gave a members of the Journal—Charles T. Jahren, Jesus M. de la
significant amount of time participating on the special issue Board Garza, and Carl Haas—who strongly supported the idea of this
of Advisors. special issue from the outset. We would also like to thank Adrian
Brügger, the graduate research assistant at Columbia University
who spent countless hours in the library dissecting journal articles
Special Issue Board of Advisors to identify the methodological approaches used.
We close this introduction with a disclaimer. This special issue
Simaan AbouRizk, University of Alberta is just the beginning. We hope the articles contained in this spe-
Dulcy M. Abraham, Purdue University cial issue can generate a dialog in the Journal about the rigorous
Paul Chinowsky, University of Colorado at Boulder application of the numerous methods we use to argue our contri-
James E. Diekmann, University of Colorado at Boulder butions to knowledge. Yet, the special issue is not complete.
Jesus M. de la Garza, Virginia Tech There are other methods in use by CEM researchers that still need
Carl T. Haas, University of Waterloo and merit examination and discourse. Nevertheless, we hope the
Chris Hendrickson, Carnegie Mellon University CEM community can benefit from this first step in what we hope
Charles T. Jahren, Iowa State University to be a continuing dialog about how to improve the rigor and
Raymond E. Levitt, Stanford University contributions to knowledge of our collective research endeavors.

2 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Methodology for Conducting Discrete-Event Simulation
Studies in Construction Engineering and Management
Julio C. Martinez, M.ASCE1

Abstract: This paper suggests the methodology to follow when conducting discrete-event simulation 共DES兲 studies in construction
engineering and management research. Emphasis is made on the steps that, due to the uniqueness of the construction environment, are
particularly important yet are not discussed extensively in the general DES literature. Guidelines are provided to determine what aspects
of a DES study demand a rigorous application of the theory depending on the purpose of the study. The paper concludes with the
importance of properly understanding the probabilistic concepts upon which DES relies and on coupling this understanding with engi-
neering judgment as a key for successful use of DES in construction research.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000087
CE Database subject headings: Analytical techniques; Quantitative analysis; Construction management; Methodology; Simulation.
Author keywords: Analytical techniques; Discrete-event simulation; Design of operations; Quantitative analysis.

Introduction Conferences are available online at http://wintersim.org/

pastprog.htm.
Discrete-event simulation 共DES兲 has been recognized as a very In general, the process for conducting a DES study of a con-
useful technique for the quantitative analysis of operations and struction operation is not different than the one that should be
processes that take place during the life cycle of a constructed followed in any other field. However, certain aspects of the con-
facility. According to Abudayyeh et al. 共2004兲, during 1997–2002 struction environment are very different, and these differences
共the last period included in the paper兲 simulation was the most sometimes dictate the use of 共or need for兲 different tools, tech-
frequent topic among articles published in the Journal of niques, and precautions. Some things are only very briefly dis-
Construction Engineering and Management 共JCEM兲. While cussed in the general theory of DES, yet are very important and
merit explicit attention in a construction setting.
simulation is a very broad term, an inspection of all articles 共348
This paper outlines the methodology for conducting DES stud-
as of this writing—see http://scholar.google.com/scholar?q
ies of construction operations. The parts of this methodology that
⫽simulation&as_publication⫽”journal of construction engineer-
are no different between construction operations and general use
ing and management”兲 containing the word “simulation” that
are covered very briefly since extensive sources for this are
have been published in the JCEM reveals that the overwhelming
readily available. Steps that are particularly challenging or impor-
majority of them are about the use of DES for modeling and
tant to construction and have not been fully discussed in the lit-
improvement of construction operations.
erature are discussed in more depth. The purpose for which a
The proper methodology to follow when performing a DES
simulation model is built has a strong impact on which aspects of
study is a subject that cannot be covered in a single article of
the methodology 共or parts of the model in question兲 require the
limited length. Law and Kelton 共2000兲 provided an extensive de- fullest rigor and which can be treated more flexibly—a discussion
scription of the process 共in 768 pages兲 that is both formally rig- of this follows the presentation of the methodology.
orous and practical. Even that thoroughly comprehensive source
refers readers to nearly 1,000 other sources for more in-depth
information. Shannon 共1998兲 provides a bird’s eye view of the
process for those that want a quick description. The subject of Application of DES to Construction Operations
simulation methodology is so extensive, that the annual Winter
Simulation Conference has dedicated tracks for this purpose 共the There are many ways in which to break up the steps needed for a
proper DES study. The steps listed next should be appropriate for
last few conferences have included two tracks for modeling meth-
DES studies in construction engineering and management:
odology and two more tracks for analysis methodology兲. Articles
• Determine the extent to which a DES model can lead to better
from the proceedings of the 1997 and later Winter Simulation
understanding of the system in question or to obtaining quan-
1
titative measures of performance for the problem of interest.
Associate Professor, School of Civil Engineering, Purdue Univ., 550 The purpose of this first step is to answer the question: “Is
Stadium Mall Dr., Room 1245, West Lafayette, IN 47907. E-mail: DES the appropriate tool for the problem at hand?”
[email protected]
• Establish the scope of the model and the specific questions that
Note. This manuscript was submitted on September 1, 2008; approved
on May 4, 2009; published online on May 8, 2009. Discussion period the model should answer.
open until June 1, 2010; separate discussions must be submitted for indi- • Define the model for the operation. This includes establishing
vidual papers. This paper is part of the Journal of Construction Engi- the level of detail of the model, selecting the elements that will
neering and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, be used to represent the real system 共e.g., resources, activities兲,
ISSN 0733-9364/2010/1-3–16/$25.00. and capturing the logic appropriately.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 3

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
• Collect and synthesize data about the operation to suit the to not be the ideal tool, the process of attempting to build a DES
model. In addition to actual data collection, this includes de- model forces the engineer to think about a problem in ways that
termining whether basic probabilistic assumptions hold 关typi- lead to its solution 共without relying on actually running simula-
cally independence and identical distribution 共IID兲兴, fitting tions兲.
distributions, and testing for goodness of fit.
• Verification of the model and data to ensure that it matches the
modeler’s understanding of the system. Model Definition
• Validation of the model to ensure that it matches the real or
imaginary system. If a model has been verified, validation A model is a representation of a current or proposed operation or
seeks to determine whether the modeler truly understood the system. If a model is valid, it is possible to experiment with the
real or proposed system. This step requires the participation of model to better understand the operation it represents. A model is
individuals that are intimately familiar with the real system. considered valid only for the purpose for which it is built, and not
• Analysis of simulation output for a single run. in absolute terms. A model built to determine the number of
• Design and execution of simulation experiments. trucks needed for an earthwork operation may be considered valid
• Analyze the output of experiments to determine the perfor- for that purpose, but may not be necessarily valid, for example, to
mance of various system configurations or to select the best determine the optimal fueling strategy for a fleet of trucks or to
from among several alternatives. obtain a cost estimate for bidding purposes.
• Document and present results. In order to build a model appropriately, significant effort has to
• Use results for decision making. be dedicated to understand the real or imaginary operation that it
will represent and the specific purpose of the model. The engineer
can then determine the level of detail with which to model differ-
Is Discrete-Event Simulation the Appropriate Tool ent parts of the operation.
for the Problem at Hand? The model is then created in a process that is both art and
science. Shannon 共1998兲 used an analogy to oil portraiture to
Quite often a simulation model is used for purposes that are better illustrate the artistic aspect of model building. Knowledge of per-
served by other methods of analysis that require less effort to spective, color theory, media, inks, and the study of portraits from
implement properly 共e.g., a spreadsheet兲. This often happens the masters does not mean that an individual can produce an oil
when DES has been newly introduced to an organization and it is portrait of quality. Extensive practice and talent are also needed.
incorrectly thought to be the solution to all problems. Using DES A variety of tools and techniques exist for model development.
only for situations that require it is critical. Martínez and Ioannou 共1999兲 discussed some tools that can be
In May 2001 the writer visited Dragados, a Spanish construc- used to model construction operations and describes in detail
tion firm that has successfully embraced DES as a tool, to learn some that have been created specifically for this purpose. The tool
about their experience implementing DES and to teach a short that is used is not that important, so long as the person developing
course on DES modeling using EZStrobe 共Martínez 2001兲. the model understands it and is capable of performing the art of
Dragados trains all its engineers to recognize situations where modeling with it. A key issue to consider while developing a
DES can be of value. When these engineers recognize these situ- model is that the aspects of reality that are relevant to the purpose
ations, they communicate it to a team of DES specialists who of the model must be faithfully represented without simplifica-
look at the problem in question and determine whether indeed tions and assumptions that would otherwise make the model
DES is the tool to use. Only in those cases do they perform a DES trivial or misleading 共Shannon 1999兲.
study, and they report returns on investments in their studies of
2,000%. The key to this success is that DES is applied only when
appropriate. Balbontin-Bravo 共1998兲 and Halpin and Martinez Data Collection and Synthesis
共1999兲 documented some of the operations where Dragados has
effectively used DES. Collecting data for construction operations can be very challeng-
What are the characteristics of problems where DES can be ing. Typically, construction activities are dependent on the condi-
particularly valuable? To answer this question it is essential to tions that exist at the time they are performed. The distribution of
thoroughly understand the problem at hand, what aspects of it can the time it takes for a truck to haul material along a path on a
and cannot be represented using DES, the effort required for mod- rainy winter day may be different than on a sunny day in the
eling and collecting the appropriate data, and the effectiveness spring; it may be different even on the same day for different
and applicability of other methods of analysis. Typically, the operators; or at different times of the day with the same operator.
problems that are well suited to DES: Most distribution fitting techniques are based on the assump-
1. Involve significant uncertainties in the time required to ac- tion that the observed data are IID, i.e., independent 共data points
complish tasks and/or in the amount and quality of materials are not related兲 and identically distributed 共they all have the same
consumed and produced. distribution兲. However, in many instances the data as observed is
2. Are logistically complex with a number of context sensitive not IID. In order to synthesize the data 共reduce it to a probability
dynamic rules and decisions. distribution兲 and later generate it appropriately during simulation,
3. Have interdependent components subject to complex activity the uncertainty needs to be expressed as a function of other vari-
start-up conditions where many resources with distinct prop- ables that are IID. Sometimes this is not possible and in those
erties must collaborate according to highly dynamic rules cases it is necessary to know the conditions that existed when
共Hooper 1986兲. each data point was collected, so that when grouped by common
Problems that do not involve significant uncertainties and/or logi- conditions, the data are IID or can be expressed as a function of
cal complexity can usually be analyzed more effectively using an IID variable.
other techniques. Often, however, even though DES is determined In an ideal world data are collected automatically along with

4 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 3

2.5

2
X(i)

1.5

0.5

0
0 20 40 60 80 100 120 140 160 180 200
i

Fig. 1. Sequentially ordered plot of truck loading times

the conditions. An interesting discussion on this topic can be ferences in the conditions but may not know whether the IID
found in Kannan and Vorster 共2000兲, who used the instrumenta- nature of the data are affected by this difference in conditions.
tion available in off-highway trucks to collect data autonomously To illustrate, assume that the times for a truck loading activity
along with the conditions associated with each data point. during day 1 共sunny, 60° F兲 follow a Uniform关1,2兴 min 共i.e., uni-
Another complication associated with collecting data for con- formly distributed between 1 and 2 min兲. Times for the same
struction operations involves lack of consistency in human judg- activity during day 2 共rainy, 38°兲 follow a Uniform关2,3兴 min. One
ment during collection. In a discrete system, activities are hundred samples are collected in the field on Day 1 and another
assumed to have a specific start and end time. However, in the hundred on Day 2. If that data are combined to form a set with
continuous real world, the point at which one activity starts and two hundred observations and then fed blindly to distribution fit-
the other ends may not be very clear. Hildreth et al. 共2005兲 de- ting software, it is likely that the software will strongly indicate
scribe a computerized procedure for collecting activity times in- that the data comes from a Uniform关1,3兴 min. A histogram of the
volving outdoor heavy equipment from GPS traces. Using their data will reinforce this since it would look identical to the histo-
techniques, it is possible to bring consistency to the definition of gram of a Uniform关1,3兴 min.
activity start and end times and to collect data autonomously. However, in this example the data are not IID, the distribution
Engineering judgment and a thorough understanding of prob- of the first one hundred numbers and of the second hundred num-
ability and statistics are essential for proper DES modeling. The bers are not identical. Using Uniform关1,3兴 in a simulation of the
availability of data as ideally desired is rare, but the engineer process will very likely produce invalid results, unless the sensi-
needs to know that this is a difference between the model that tivity of the model’s output to the time of the activity is extremely
uses the data and the real world. The engineer needs to couple small.
that knowledge with probability and statistics to understand how A graph of the observed data points with the observation num-
the model differs from reality, and to judge its impact on the ber 共i兲 on the horizontal axis and the observed value X共i兲 on the
validity of the model. In construction, most of the problems with vertical axis 共e.g., ordered plot兲, as shown in Fig. 1, makes this
the analysis and subsequent use of the data during simulation evident. Inlow 共2006兲 in his excellent article about teaching engi-
stem from using statistical techniques that assume IID, when in- neering students to understand the concept of independence, sug-
deed the data are not. Some specific ways in which data are not gested that the data points be sorted randomly and used to create
IID, and suggestions on how to analyze and restructure models so a randomly ordered plot, as shown in Fig. 2. A difference in the
that they can properly rely on the IID assumption, follow. visual character of the plots would provide indication that the data
are not IID. According to Inlow, comparing a sequentially ordered
plot to a randomly ordered plot “makes it easier to determine if 1.
Homogeneity apparent trends are systematic or simply random oscillations, and
2. if adjacent observations are related or autocorrelated since the
Sometimes it is not obvious to the engineer whether there is a random order plots will appear to be rougher or smoother than
difference in conditions between data collected at different times. the original data if negative or positive autocorrelation is
Other times the engineer may recognize that there are some dif- present.”

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 5

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 3

2.5

2
X(i)

1.5

0.5

0
0 20 40 60 80 100 120 140 160 180 200
i

Fig. 2. DES randomly ordered plot of truck loading times

Furthermore, the graph of X共i + 1兲 versus X共i兲 shown in Fig. 3 between 2 and 3 the next observation is also between 2 and 3 with
共e.g., a scatter diagram兲, clearly indicates that when an observa- certainty. Quantitatively, the coefficient of correlation between
tion is between 1 and 2 the chance that the next observation is X共i兲 and X共i + 1兲 can be used to assess the degree of independence
between 1 and 2 is near 100%. Similarly if the observation is between successive observations. The lower the absolute value of

2.8

2.6

Sample Correlation = 0.74

2.4

2.2
X(i+1)

1.8

1.6

1.4

1.2

1
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3
X(i)

Fig. 3. Scatter diagram for truck loading times

6 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 1

0.8

0.6

0.4
Corr(X,X+i)

0.2

-0.2

-0.4
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 127 133 139 145 151 157 163 169

Fig. 4. Correlation plot for truck loading times

the coefficient of correlation, the greater the independence. If the data sets was collected differs from the others. If this happens,
data are totally independent, the true coefficient of correlation is different conditional distributions should be determined for each
zero. Fig. 3 shows the sample coefficient of correlation as an set. Furthermore, the different conditions need to be understood
estimate of the true coefficient to be 0.74. This makes it pretty and modeled. The artificial example used earlier, for example, can
obvious that successive data points are not independent. be modeled by some process that at the start of each day samples
Even if successive data points are nearly independent, it is from a distribution to indicate good weather or bad weather 共and
possible for data points N observations apart to exhibit some cor- for simplicity we are assuming here that weather is discrete and
relation, which would be an indication that the data are not inde- can only be either good or bad兲. The time for the loading activity
pendent. The correlation plot of Fig. 4 starts at 0.74 for data can then be specified as conditional on the weather 共e.g., Uni-
points one observation apart, and shows the sample coefficient of form关1,2兴 given that weather is good, Uniform关2,3兴 given that
correlation for data points n observations apart. The plot shows weather is bad兲.
that the sample correlation coefficient starts to oscillate between
positive and negative with relatively small absolute values after n
reaches 100, clearly indicating that from then on there is no cor-
Autocorrelated Data and Nonstationary Input
relation 共i.e., the second set of 100 numbers is fully IID and
Processes
exhibits no autocorrelation兲.
This example was artificially created to make the point. There
is no real construction activity that is truly uniformly distributed, Sometimes data are correlated to itself. When autocorrelation is
let alone showing two perfectly adjacent and equally sized uni- positive a high value is likely to follow another high value, and a
form distributions on successive days. With real data that is not low value is likely to follow another low value. When autocorre-
homogeneous, and that may have been collected in several lation is negative, there is tendency for data samples to alternate
batches, it may be very difficult to make a judgment by looking at between high and low. In nonstationary input processes, the dis-
an ordered plot of the observations or a scatter diagram. Some- tribution of the data varies with time or sequence—the data ex-
times the engineer, due to an understanding of what is being hibit a trend. Nonstationary processes tend to have some degree
observed, will group the data ahead of time. Maio et al. 共2000兲, of autocorrelation. Examples of nonstationary inputs include the
for example, group truck travel times into several categories ac- time required to perform an activity that involves learning, and
cording to haul distance and weight before fitting distributions to hauling material at distances that are continuously increasing or
each. However, often, the engineer will not know if the data are decreasing.
homogeneous. In these cases, the Kruskal-Wallis hypothesis test The existence and degree of autocorrelation can be assessed
for homogeneity 共Law and Kelton 2000兲 can be used to determine graphically by comparing ordered and randomly ordered plots of
if two or more separately collected sets of data can be treated as the data, and by correlation plots and scatter diagrams. Formally,
homogenous and combined to determine a single distribution. it is possible to assess IID using runs tests 共Gibbons 1985兲. If the
If separately collected data sets are not homogenous, then this data are not IID, the underlying reason needs to be investigated,
is an indication that the conditions under which at least one of the understood, and modeled 共if possible兲. The ideal way to model

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 7

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 共Fig. 8兲 show that the data are not IID. In effect, the ordered plot
suggests a nonstationary distribution with central tendency that
follows an exponential learning curve. Fitting an exponential
equation to the data using least-squares yields Hours= 2.86
⫻ Unit−0.33, and is plotted in Fig. 6. We can then look at the
deviation of the data from the learning curve to see if the devia-
tions are IID, and if so, what distribution best explains the devia-
tions. Fig. 9 shows the ordered plot of the deviations, Fig. 10
shows the scatter diagram, and Fig. 11 shows the correlation plot.
All three indicate that the assumption of IID is likely. Fig. 12
shows the histogram and distribution fit of the deviations. Based
on this analysis, the installation of valves of this type could be
represented in a model using the formula 2.86⫻ Unit−0.33 + Beta
关−0.233, 0.194, 1.49, 1.26兴, where Unit is a dynamic variable that
represents the sequential number of the unit being installed 共the
data for this example was artificially generated using the formula
3 ⫻ Unit−0.34 + Uniform关−0.2, 0.2兴兲.
In many cases several activities depend on common underly-
Fig. 5. Histogram of hours required for valve installation ing conditions. As the conditions change, so do the distributions
of the activity times. When this happens, the data from the vari-
ous activities are correlated. In these cases, the underlying condi-
such processes is to define the source of the data as a distribution
tions need to be identified and modeled, and the distributions used
where one or more of its parameters are a function of time, se-
for the different processes expressed as functions of these condi-
quence, or other system state.
tions. It is also possible to determine n-variate distributions to
Consider an activity for the installation of valves, where the
explain the correlated data, but this is not easily done without
time required for each of the first 200 installations was collected.
many further assumptions regarding the underlying distributions
The histogram of the data along with the distribution fitted to it
and 1:1 correspondence among data sets.
共assuming that the data are IID兲 by Stat::Fit 共Benneyan 1998兲 are
shown in Fig. 5. Since the fit is very good 共the ChiSquare,
Kolmogorov-Smirnov, and Anderson-Darling tests do not reject
the hypothesis at the 0.15 level of significance兲, it may seem a Subjective Distributions
good idea to use the fitted distribution in a model.
However, further inspection by using a sequentially ordered Experts often have a very good understanding of the distribution
plot 共Fig. 6兲, a scatter diagram 共Fig. 7兲, and a correlation plot of some variable based on years of experience observing the vari-

3.5

2.5

2
Hours

1.5

0.5

0
0 20 40 60 80 100 120 140 160 180 200
Unit

Fig. 6. Ordered plot for valve installation hours

8 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 3

2.5

2
X(i+1)

1.5

Sample Correlation = 0.85

0.5

0
0 0.5 1 1.5 2 2.5 3
X(i)

Fig. 7. Scatter diagram for valve installation hours

0.8

0.6

0.4
Corr(X,X+n)

0.2

-0.2

-0.4
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 127 133 139 145 151 157 163 169
n

Fig. 8. Correlation plot for valve installation hours

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 9

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 0.25

0.2

0.15

0.1

0.05

0
R(i)

-0.05

-0.1

-0.15

-0.2

-0.25

-0.3
0 20 40 60 80 100 120 140 160 180 200
i

Fig. 9. Ordered plot of deviations

0.25

0.2

0.15

0.1

0.05

0
R(i+1)

-0.05

-0.1

-0.15

-0.2

Sample Correlation = -0.10

-0.25

-0.3
-0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25
R(i)

Fig. 10. Scatter diagram of deviations

10 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 0.25

0.2

0.15

0.1

0.05
Corr(X, X+i)

-0.05

-0.1

-0.15

-0.2

-0.25
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
i

Fig. 11. Correlation plot of deviations

able in a variety of conditions. The knowledge of these experts AbouRizk et al. 共1991兲 suggested a graphical computer based
may be significantly more extensive than what could be deter- procedure to determine the beta distribution that best explains
mined from a limited number of field observations. The problem subjective expert knowledge by eliciting the minimum possible,
is that this probabilistic knowledge is implicit—an expert cannot maximum possible, and two additional values from the following
typically express this knowledge as a probability distribution. pairs: mean and variance, mode and variance, mean and mode, or
Probability encoding is a subject that has been researched exten- two arbitrary percentiles. While this procedure is 100% accurate
sively and that, when done properly, can be used to quantify and if the four elicited quantities are 100% accurate, it is difficult for
transfer probabilistic knowledge about uncertain quantities from an expert to subjectively provide these quantities without biases
experts. and unconscious modes of judgment. The visual interactive dis-
At the very rudimentary level, that knowledge can be ex- play of the distribution can be of help, however, for experts who
pressed by a distribution with parameters determined by eliciting understand distribution probability concepts.
from experts the most favorable, unfavorable, and commonly oc- Spetzler and Von Holstein 共1975兲 provided an excellent intro-
curring 共mode兲 values. According to Perry and Greig 共1975兲, the duction to probability encoding that includes techniques to correct
most favorable time supplied by an expert can be assumed to be for biases and unconscious modes of judgment. More recently,
the 5th percentile and the most unfavorable time assumed to be Abbas et al. 共2008兲 reviewed and compared some of the tech-
the 95th percentile because most subjects have never actually niques that can be used to encode probabilities. These techniques
experienced the extremes, or known so if they have. Thus, when
asked for the most favorable time, the reply most experts provide
is probably closer to the 5th percentile, and when asked for the
most unfavorable time their reply is probably closer to the 95th
percentile. Perry and Greig then make assumptions about the
mean 共that it is the weighted average of the mode and the 5th and
95th percentiles, with 5% less weight given to the mode兲 and
standard deviation of the distribution 共that it is the spread between
the 5th and 95th percentiles divided by 3.25兲. These assumptions
have been proven to be relatively distribution free and quite ac-
curate. With these assumptions it is possible to determine the
parameters, location, and scale, of, for example, a beta distribu-
tion. Perry and Grieg’s procedure is very attractive because it
requires the elicitation of only three values 共as in Pert兲, is rela-
tively accurate 共much more than original Pert兲, and is imple-
mented as a native probabilistic sampling function in some
construction-oriented simulation tools 共e.g., UM-Cyclone and
Stroboscope兲. Fig. 12. Histogram and fit of deviation

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 11

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
should be used for important sources of uncertainty in a simula- Table 1. Number of Trucks Loads per Day in Earthmoving
tion model. Because construction operations are often condition Date Loads
sensitive, it makes sense to encode conditional probabilities from
experts and to use these in conjunction with a simulation of the May 17, 1997 223
conditions. May 19, 1997 264
May 20, 1997 240
May 22, 1997 276
Verification and Validation May 23, 1997 217
May 27, 1997 206
Verification is the process by which the individual共s兲 that develop May 28, 1997 272
a model ascertain that they have modeled what they intended to May 29, 1997 234
model 共i.e., model debugging兲. Some guidelines on how to do this May 30, 1997 280
are provided by Law and Kelton 共2000兲 共pp. 269–273兲. It is very May 31, 1997 248
easy to make modeling mistakes in most useful models of com-
plex systems, so verification needs to leave no doubt to the model
developer regarding the correctness of the model. ample, the objective may be to obtain a 90% confidence interval
In validation, the verified model and the actual or imaginary on the mean cost of construction that is smaller than a certain
system are compared. This validation is with respect to logic, dollar amount. To do this, a few runs of the model could be used
input, assumptions, and output. Since the model developers are to estimate the standard deviation of cost, and then this value
frequently not the experts and/or decision makers 共i.e., stakehold- could be used to calculate the total number of replications that are
ers and subject matter experts兲 in the actual operation, validation required. Alternatively, if the simulation software allows, it
requires that the model be clearly communicated to the stakehold- should be possible to replicate automatically until the objective is
ers, who can then assess many aspects of model validation. Ani- reached. Ioannou and Martínez 共1996兲 discussed in detail the de-
mation is particularly valuable in this aspect of validation since it sign of experiments for evaluation of alternative tunnel construc-
allows the stakeholders and experts to “see” a virtual version of tion strategies. The information in that source is useful and
the operation. Kamat and Martinez 共2003兲 discussed this and pre- readily adaptable to any type of simulation modeling whose ob-
sented a language and software that can be used to animate pre- jective is to compare among alternatives. For general cases of
viously simulated operations in 3D. broader applicability, the reader should refer to Law and Kelton
Validation at the input level is concerned with the probability 共2000兲.
distributions used in the model. This includes ascertaining that In contrast to most manufacturing operations, due to the un-
basic probabilistic assumptions hold 共e.g., the data are IID兲 and certainties and complexities that characterize construction, out-
goodness of fit testing 共e.g., Chi-Square, Kolmogorov-Smirnov, puts typically exhibit significant variance. Table 1, for example,
Anderson-Darling兲. shows the number of loads that a contractor was able to achieve
Validation at the output level 共i.e., results validation兲 is where each of 10 days when moving dirt for the construction of a road.
construction operations present interesting challenges when com- The data are from Cor 共1998兲.
pared to other disciplines. This an area that needs considerable Despite the nearly constant haul distance, weather, and the use
attention in future studies. of the same fleet, production varied significantly due to complexi-
At the single run level, it needs to be established that the ties and uncertainties related to right to use a temporary railroad
output of the model resembles the output that would be expected crossing and to equipment breakdowns. The fact that these com-
from the actual or imaginary system 共Law and Kelton 2000, p. plexities existed is what made the use of DES particularly valu-
279兲. In construction, the systems being modeled often do not able in that case compared to the standard tools used by the
exist 共but occasionally do兲. This makes comparison with the real contractor. A valid simulation of this operation should exhibit a
operation, prior to its execution, impossible. In these cases, the similar variability between independent runs. Operations where
simulation results need to be checked for reasonableness by ex- the output has little variability are typically relatively easy to
perts, or perhaps using Turing tests 共Carson 1986兲. If the system analyze using other methods 共i.e., DES is overkill for them兲.
exists 共e.g., when the modeling has the objective of improving it兲, For most decision making purposes, it is in fact more impor-
the model can potentially be compared statistically to the actual tant to understand the variability than the mean measure associ-
system output, but this is very difficult unless the operation will ated with an output. In validation, an expert should assess
be repeated a number of times. whether the distribution of the output is reasonable. This aspect of
A single simulation experiment yields one set of outputs 共i.e., validation is where many DES studies in construction fail—
cost, equipment utilization, production rates, inventory levels兲 frequently only a single run is reported and analyzed and other
that correspond to the exact sequence of events that took place in times the variance of the output is significantly smaller than could
a simulation. Because simulations are probabilistic, a second in- be expected.
dependent simulation of the same model can, and should, yield Variability of output makes it also very difficult to validate a
different results. For this reason, a simulation model should be simulation model by comparing the result in the field to what the
run several times in order to obtain 共ideally兲 the distribution of simulation had indicated. Typically the field observation is only
each output. Design of experiments refers to the process of deter- one data point that could have come from any number of output
mining how many replications 共and how to set up the replica- distributions. If the nature of construction were such that the op-
tions兲 of a model simulation to perform in order to achieve a eration could be carried out independently several times 共no
certain objective. Design of experiments also involves the cre- learning effect, etc.兲, then it would be possible to perform statis-
ation of proper simulation conditions, ensuring that the simulation tical tests 共i.e., a goodness of fit test兲 to determine the likelihood
components assumed to be IID are indeed simulated as IID, and that the observations come from the simulation output distribu-
ensuring independence between different simulation runs. For ex- tion. 共An ongoing earth moving operation with constant condi-

12 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 12.0%

10.0%

8.0%
Probability

6.0%

4.0%

2.0%

0.0%
16.2 16.8 17.4 18.0 18.6 19.2 19.8 20.4 21.0 21.6 22.2 22.8 23.4 24.0 24.5 25.1
Cost in Millions

Fig. 13. Distribution of tunnel construction cost using NATM

tions could be an example to the contrary, if production is looked Relationship between the Purpose of a Study, the
at on a daily basis. However, conditions rarely remain constant Rigor It Requires, and Its Validation
for long.兲 Fig. 13 shows the output distribution for the cost of a
tunnel construction operation using the New Austrian Tunneling Performing a DES study that follows the appropriate rigor in
Method 共NATM兲. This is from Martínez and Ioannou 共1996兲 and every one of its components may represent a very substantial
was built using 4,000 independent replications. Here, the simula- effort. Carrying out a DES study does not require that every
tion output is a distribution with very high variance that ranges single part be performed at full rigor. However, what is necessary,
from a low $16 million to a high $25.4 million. The output is very and cannot be circumvented, is a full understanding of the as-
useful for understanding the large risks associated with this op- sumptions that are made and their impact in the validity of the
eration. However, if the tunnel is built exactly as modeled 共the
model for the purpose for which it was built. A similar concern
model is 100% valid兲, the actual cost of construction will be
was expressed by Schexnayder 共1997兲.
anywhere within the range. If the contractor finds mostly good
When using a DES study in construction practice to design an
ground, the cost may be $17 million or lower. If the contractor
operation, sensitivity analysis can be used to determine which
finds mostly poor ground, the cost may rise to $23 million or
inputs or portions of the operation impact the performance mea-
higher. Whatever the actual cost, it cannot be used to prove or
disprove that the model was valid. If the model was invalid, and sures of interest in a significant way. The engineer can then con-
it were possible to build 共independently兲 several dozen identical centrate on modeling those parts at a reasonable level of detail
tunnels, then it would be statistically possible to prove at some and to follow all the probability theory that is pertinent rigorously.
level of significance that an invalid model is indeed invalid 共i.e., For parts that are inconsequential to the purpose of a model, flex-
there would be enough evidence to reject the hypothesis that the ibility may be used to model that input 共e.g., not too much detail,
observed costs come from the simulation model’s output distribu- treat non-IID data as IID兲. When the purpose of a model is to
tion兲. It would be totally incorrect to state something similar to compare among various alternatives, emphasis should be on the
“proof of the correctness of model is that the tunnel was built at a parts of the model where the alternatives differ.
cost of $19.4 million, which is close to the mean output of the When DES is used to demonstrate some principle or the effec-
simulation, $19.1 million.” tiveness of some technique, the emphasis should be on capturing
Validation of construction operation studies is based predomi- the principle faithfully. Input distributions need to make sense and
nantly on having experts convinced, to the point that they will rely be reasonable to individuals knowledgeable in the area of appli-
on the model to make decisions, that the model is an accurate cation, but they do not need to be based on collected data. If data
representation of the real or future operation for the purpose for are available from other sources, then it can be used. Tommelein
which the model was built. In other words, a model is considered 共1998兲, for example, used 10% as the probability of a pipe spool
to be valid for a particular purpose if it is credible to experts for needing rework in models to illustrate pull scheduling, because
that purpose. the range 1–10% had been indicated by Howell and Ballard

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 13

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
共1996兲 in a prior study and she wanted to demonstrate her model ment of SPSs in manufacturing using general purpose simulation
under the largest realistic uncertainty. This was consistent with languages. In the construction engineering domain several SPS
the purpose of the model used by Tommelein, which was to illus- tools have also been developed. Systems such as Simphony 共Haj-
trate the impact of coordination planning and the benefits of pull jar and AbouRizk 1999兲 have been designed specifically for the
over push when uncertainty is high. purpose of building such tools, and general purpose program-
Many DES models exist solely to demonstrate a research prod- mable and extendable construction simulation systems such as
uct that in some ways claims to advance the state-of-the-art in Stroboscope 共Martinez 1996兲 can also be used to create SPSs.
simulation modeling. This is another case where the specific in- In the case of construction operations, SPS systems allow
puts and details that are used are of no significant consequence practitioners to define DES models by working with modeling
共unless the research is related to data synthesis兲—they must sim- elements from the domain of the SPS tool. In an earthmoving SPS
ply be reasonable in order to give credibility to the work. Claims tool, for example, it may be possible to define a model by draw-
are typically that: ing haul routes and specifying their properties 共e.g., grade, rolling
1. Some aspect of the modeling process can be done more ac- resistance兲, by selecting trucks and excavators from databases;
curately without relying on simplifications that were other- and specifying the location of excavators, routes taken by trucks,
wise required; and quantities of cut and fill. The SPS tool will use this informa-
2. It can be done in a meaningful way following a thought tion to create a DES model that internally contains the equivalent
process that differs from those that already exist; and of activities such as load, haul, dump, and return; with parameters
3. It can be done with less training or effort without reducing its such as time distributions totally or partially determined from the
other characteristics. properties of the objects used in the SPS tool. In these cases, the
According to De la Garza 共2007兲, in order for research findings to relationship between the SPS modeling elements and the under-
withstand the scrutiny of peers, the work needs to be described at lying lower-level DES model is very direct. For example, deter-
a level of detail that will enable others to replicate it and confirm ministic travel times may be calculated from truck and haul route
the claimed findings. properties using established engineering calculations, and these
It is critical that complete data and details 共input and output兲 of may be made stochastic with input specified directly by the user
the examples used to illustrate a new tool or method be available 共e.g., users may provide a distribution by which the calculated
to others 共or at least to reviewers兲, so that they can perform alter- travel time will be multiplied to obtain the time for each haul兲. In
nate modeling and compare results. This has the added benefit of any case, the ability to configure the model very precisely to the
clarity with respect to the claims being made by the research. exact situation at hand is dependent on how well the processes
When the full data set, details of the model, and output are not and tools built into the SPS that are used to create the underlying
available; even trivial endeavors can appear to be grandiose. Re- model can match the current situation. Conceivably, the SPS tool
searchers should have developed enough confidence in the valid- can allow some degree of configuration to enable modelers to
ity and true value of their contributions to not consider it a risk to achieve this to some extent. A researcher creating an SPS answers
provide this information. The examples used to demonstrate a questions related to how to define a model using domain specific
procedure need not be real or confidential, or large but lacking elements, and how to use this information to create the corre-
complexity. In fact, a very significant part of the validation meth- sponding lower level DES. There is a responsibility on the part of
odology should include the development of small cases that can the researcher to explain very clearly how this is done, so that
be described and grasped in a reasonable amount of space, but users can understand what is being modeled and so they can judge
that fully exercise the claims set forth. whether this will be satisfactory for the case at hand 共i.e., users
should be able judge the extent to which all the issues that should
Discrete-Event Simulation as Part of Higher-Level be addressed in a DES study are considered in the SPS兲.
Modeling Systems It is possible to extend the SPS concept to domains that are not
related to construction operations modeling. One interesting ex-
The focus of this paper has been on the methodology to follow in ample is the virtual design team 共VDT兲, which is a computational
the types of DES research that have been traditionally published organization model that incorporates elements of DES, artificial
in the JCEM. Specifically, simulations of construction operations intelligence, and computational organization theory 共Levitt et al.
where the sequence of events, input distributions, statistical de- 1999兲. When creating VDT models, users work with project-level
sign of simulation experiments, and statistical output analysis are activities 关e.g., activities similar to those that would be used in a
to a large extent controlled and defined by the modeler. In this critical path method 共CPM兲 schedule兴 and specify for them, for
type of modeling, whether for a case study, to prove or disprove example, the required work volume, work skill, activity-level un-
some principle, or to demonstrate a new modeling concept; it is certainty, activity complexity, and interdependence strength to
possible to create models that contain logic errors, that are based other activities. These characteristics are specified in a descriptive
on incorrect statistical assumptions, or that simply do not capture rather than normative manner. The knowledge embedded in the
the real variability of outcome that characterizes construction. VDT then creates a DES that in addition to the actual work,
Due to the requirements for rigor, or necessity to relax rigor models communications, disruptions, and coordination based on
only when it follows a full understanding of the consequences of these parameters. The underlying DES model includes not only
doing so, DES research and application of this type requires sig- the direct work activities, but also the interruptions for communi-
nificant education and discipline. In recognition that many field cation and exchange of information among actors. The process by
engineers do not have this specialized knowledge and ability, which the project-level activities and their descriptive parameters
some research effort has attempted to answer the question of how translate into DES is based on a microcontingency theory devel-
to make this technology accessible to such field personnel. Spe- oped and validated by the VDT team for that purpose. According
cial purpose simulation 共SPS兲 has been proposed a possible an- to Levitt et al. 共1999兲 共p. 5兲, a microcontingency theory is “a
swer to this problem. theory to describe and predict behavior at the level of individual
Seppanen 共1990兲 provided early guidelines for the develop- actors and activities.” Because the relationship between the mod-

14 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
eling elements of a VDT model and the underlying DES is not of trends in construction research: 1985—2002.” J. Constr. Eng. Man-
direct, but instead based on its microcontingency theory, many of age., 130共3兲, 433–439.
the considerations needed to properly conduct DES studies of Balbontin-Bravo, E. 共1998兲. “Simulation of large precast operations.”
construction operations are simply not applicable to a researcher Proc., 1998 Winter Simulation Conf., IEEE, Piscataway, N.J., 1311–
using such a tool since there is a shift in responsibility for the 1377.
Benneyan, J.C. 共1998兲. Software review: Stat::Fit, OR/MS Today, 25共1兲,
modeler from the normative to the descriptive. On the other hand,
Informs—Institute for Operations Research and the Management Sci-
the tool itself and the knowledge embedded in it need to undergo
ences, Hanover, Md.
a very extensive and substantial validation process in order for Carson, J. S. 共1986兲. “Convincing users of model’s validity is challenging
users to trust it. The validation process followed for the VDT is aspect of modeler’s job.” Industrial Engineering, 18, 74–85.
documented in Thomsen et al. 共1999兲. Cor, H. 共1998兲. “Using simulation to quantify the impacts of changes in
construction work.” MS thesis, Virginia Tech, Blacksburg, Va.
Summary and Conclusions De la Garza, J. M. 共2007兲. “Sponsored research and its impact on univer-
sities, faculty, and journals.” J. Constr. Eng. Manage., 133共9兲, 708–
709.
The process for conducting a DES study in construction does not
Gibbons, J. D. 共1985兲. Nonparametric methods for quantitative analysis,
differ from the one used in other disciplines. Certain aspects,
2nd Ed., American Sciences, Columbus, Ohio.
however, warrant significant consideration in construction as
Hajjar, D., and AbouRizk, S. 共1999兲. “Simphony: An environment for
compared to studies in other disciplines.
building special purpose construction simulation tools.” Proc., 1999
Heavy dependency on conditions in the performance of activi-
Winter Simulation Conf., Vol. 2, IEEE, Piscataway, N.J., 998–1006.
ties, and the variability of these conditions during the simulated Halpin, D. W., and Martinez, L. 共1999兲. “Real world applications of con-
period, make many uncertain quantities 共e.g., activity times兲 to struction process simulation.” Proc., 1999 Winter Simulation Conf.,
not be IID. Since the assumption of IID is necessary for fitting Vol. 2, IEEE, Piscataway, N.J., 956–962.
distributions to data, in construction these uncertain quantities Hildreth, J., Vorster, M. C., and Martinez, J. C. 共2005兲. “Reduction of
must be expressed as functions of other quantities that are IID and short-interval GPS data for construction operations analysis.” J. Con-
that can in turn be appropriately described by probability distri- str. Eng. Manage., 131共8兲, 920–927.
butions. Hooper, J. W. 共1986兲. “Strategy related characteristics of discrete-event
Because the use of DES studies in construction practice is to languages and models.” Simulation, 46共4兲, 153–159.
make decisions prior to field execution of the modeled operations, Howell, G. A., and Ballard, H. G. 共1996兲. “Managing uncertainty in the
it is not possible to validate models by comparing model output to piping process.” Rep. No. RR 47-13, Construction Industry Institute,
real output before the models are used. In addition, since the Univ. of Texas, Austin, Tex.
Inlow, M. 共2006兲. “Teaching the IID assumption in engineering statistics
operations subject of the study are frequently carried out only
I.” Proc., 2006 American Society of Engineering Education Annual
once and exhibit significant variability among independent out-
Conf., American Society of Engineering Education, Washington, D.C.,
puts, it is not possible to validate models even after execution in
2006-1397, 具http://soa.asee.org/paper/conference/paper-view.cfm?id
the field. Validation must be carried out by judgment of experts, ⫽1599典.
stakeholders, and decision makers who need to be convinced that Ioannou, P. G., and Martínez, J. C. 共1996兲. “Comparison of construction
the model and the operation are one and the same for the particu- alternatives using matched simulation experiments.” J. Constr. Eng.
lar purpose for which the model was made. Evidence of validity Manage., 122共3兲, 231–241.
is reliance on the model for making decisions. Kamat, V. R., and Martinez, J. C. 共2003兲. “Validating complex construc-
It is not generally possible to perform a DES study of a con- tion simulation models using 3D visualization.” Syst. Anal. Model.
struction operation that follows full rigor in each and every as- Simul., 43共4兲, 455–467.
pect. Assumptions need to be made that are known to not be Kannan, G., and Vorster, M. C. 共2000兲. “Development of an experience
100% valid. However, this relaxation of rigor cannot take place database for truck loading operations.” J. Constr. Eng. Manage.,
126共3兲, 201–209.
blindly. An in-depth understanding of probability and statistics
Law, A. M., and Kelton, W. D. 共2000兲. Simulation modeling and analysis,
that are the basis for DES, and sound engineering judgment, are
3rd Ed., McGraw-Hill, New York.
critical elements that cannot be circumvented.
Levitt, R. E., Thomsen, J., Christiansen, T. R., Kunz, J. C., Jin, Y., and
In addition to use for the study of operations, DES studies can Nass, C. 共1999兲. “Simulating project work processes and organiza-
be used to prove or demonstrate theories and practices and also to tions: Toward a micro-contingency theory of organizational design.”
illustrate research that claims to advance the state-of-the-art in Manage. Sci., 45共11兲, 1479–1495.
construction DES. DES studies used to demonstrate new research Maio, C., Schexnayder, C., Knutson, K., and Weber, S. 共2000兲. “Prob-
advances should allow readers to verify, independently, the claims ability distribution functions for construction simulation.” J. Constr.
of the research. This often requires that significant effort be Eng. Manage., 126共4兲, 285–292.
placed on preparing the modeling examples so that they are small, Martinez, J. C. 共1996兲. “STROBOSCOPE: State and resource based
showcase the research claims, and are fully detailed 共including all simulation of construction processes.” Ph.D. dissertation, Civil Engi-
inputs and outputs兲. neering, Univ. of Michigan, Ann Arbor, Mich.
Martínez, J. C. 共2001兲. “EZStrobe—General-purpose simulation system
based on activity cycle diagrams.” Proc., 2001 Winter Simulation
References Conf., IEEE, Piscataway, N.J., 1556–1564.
Martínez, J. C., and Ioannou, P. G. 共1999兲. “General purpose systems for
Abbas, A. E., Budescu, D. V., Yu, H. T., and Haggerty, R. 共2008兲. “A effective construction simulation.” J. Constr. Eng. Manage., 125共4兲,
comparison of two probability encoding methods, fixed probability vs. 265–276.
fixed variable values.” Decis. Anal., 5共4兲, 190–202. Perry, C., and Greig, I. D. 共1975兲. “Estimating the mean and variance of
AbouRizk, S. A., Halpin, D. W., and Wilson, J. R. 共1991兲. “Visual inter- subjective distributions in pert and decision analysis.” Manage. Sci.,
active fitting of beta distributions.” J. Constr. Eng. Manage., 117共4兲, 21共12兲, 1477–1480.
589–605. Schexnayder, C. J. 共1997兲. “Discussion: Analysis of Earth-moving sys-
Abudayyeh, O., Dibert-DeYoung, A., and Jaselskis, E. 共2004兲. “Analysis tems using discrete-event simulation.” J. Constr. Eng. Manage.,

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 15

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 123共2兲, 199–199. Thomsen, J., Levitt, R. E., Kunz, J. C., Nass, C. I., and Fridsma, D. B.
Seppanen, M. 共1990兲. “Special purpose simulator development.” Proc., 共1999兲. “A trajectory for validating computational emulation models
1990 Winter Simulation Conf., IEEE, Piscataway, N.J., 67–71. of organizations.” Comput. Math. Org. Theory, 5共4兲, 385–401.
Shannon, R. E. 共1998兲. “Introduction to the art and science of simula- Tommelein, I. D. 共1998兲. “Pull-driven scheduling for pipe-spool installa-
tion.” Proc., 1999 Winter Simulation Conf., IEEE, Piscataway, N.J. tion: Simulation of lean construction technique.” J. Constr. Eng. Man-
Spetzler, C. S., and Von Holstein, S. 共1975兲. “Encoding in decision analy- age., 124共4兲, 279–288.
sis.” Manage. Sci., 22共3兲, 340–358.

16 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Optimization Research: Enhancing the Robustness
of Large-Scale Multiobjective Optimization in Construction
Amr Kandil1; Khaled El-Rayes2; and Omar El-Anwar3

Abstract: Many construction planning problems require optimizing multiple and conflicting project objectives such as minimizing
construction time and cost while maximizing safety, quality, and sustainability. To enable the optimization of these construction problems,
a number of research studies focused on developing multiobjective optimization algorithms 共MOAs兲. The robustness of these algorithms
needs further research to ensure an efficient and effective optimization of large-scale real-life construction problems. This paper presents
a review of current research efforts in the field of construction multiobjective optimization and two case studies that illustrate methods for
enhancing the robustness of MOAs. The first case study utilizes a multiobjective genetic algorithm 共MOGA兲 and an analytical optimi-
zation algorithm to optimize the planning of postdisaster temporary housing projects. The second case study utilizes a MOGA and parallel
computing to optimize the planning of construction resource utilization in large-scale infrastructure projects. The paper also presents
practical recommendations based on the main findings of the analyzed case studies to enhance the robustness of multiobjective optimi-
zation in construction engineering and management.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000140
CE Database subject headings: Optimization models; Parallel processing; Resource management; Housing; Multiple objective
analysis; Linear analysis; Algorithms; Construction industry.
Author keywords: Optimization models; Parallel processing; Resource management; Housing; Multiple objective analysis; Linear
analysis; Algorithms.

Introduction One of the main complexities that construction projects face is

the presence of multiple and conflicting objectives that need to be
The construction industry in the United States is one of the largest achieved for the successful completion of the project 共Burns et al.
sectors of the economy, contributing $1,175 billion to the nation’s 1996兲. These objectives and their relative importance vary from
economy 关U.S. Bureau of Economic Analysis 共BEA兲 2007兴. What one project to another, and they often include minimizing con-
makes this contribution even more significant is the impact that struction time and cost while maximizing safety, quality, and sus-
construction has on other sectors of the economy including the tainability. Construction engineers and managers often need to
manufacturing sector 共Carty 1995兲. Construction engineering and analyze and optimize the impact of their decisions on these im-
management research is of great importance to the viability and portant project objectives. To support construction engineers and
continued success of this important industry 共Levitt 2007兲. The managers in this critical and challenging task, a number of re-
importance of construction research is underscored by the mar- search studies focused on developing optimization models that
utilize multiobjective optimization algorithms 共MOAs兲 共El-Rayes
ginal gains in productivity that the construction industry has
and Kandil 2005; Hyari and El-Rayes 2006; Kandil and El-Rayes
achieved in comparison to manufacturing industries 共Carty 1995兲.
2006a; Lee and Kim 2007; Liu et al. 1997; Liu and Frangopol
These small productivity gains could in part be attributed to the
2005; Chan and Hu 2002兲. These algorithms could be classified
increased complexity of construction projects, which requires the
into two main types: 共1兲 naturally inspired algorithms that mimic
development of more effective methods for managing those
a natural process in their optimization approach and 共2兲 analytical
projects.
algorithms that utilize a mathematical approach to the optimiza-
tion problem. Regardless of the MOA used, the developed models
1
Assistant Professor, Division of Construction Engineering and need to be 共1兲 efficient to enable the optimization of large-scale
Management, School of Civil Engineering, Purdue Univ., 550 Stadium real-life construction problems in a practical and feasible time and
Mall Dr., West Lafayette, IN 47907 共corresponding author兲. E-mail:
共2兲 effective in identifying optimal or near optimal solutions to
[email protected]
2
Associate Professor. Dept. of Civil and Environmental Engineering,
the optimization problem. These two criteria are collectively re-
Univ. of Illinois at Urbana Champaign, 3127 NCEL, 205 N. Mathews ferred to as the robustness of the algorithm, and are often chal-
Ave., Urbana, IL 61801. E-mail: [email protected] lenging to evaluate and attain.
3
Assistant Professor, Dept. of Construction Management, Univ. of
Washington, Seattle, WA, 98195. E-mail: [email protected]
Note. This manuscript was submitted on August 4, 2008; approved on Objectives
August 19, 2009; published online on December 15, 2009. Discussion
period open until June 1, 2010; separate discussions must be submitted
for individual papers. This paper is part of the Journal of Construction The main objective of this paper is to outline a number of meth-
Engineering and Management, Vol. 136, No. 1, January 1, 2010. ods for evaluating and attaining robustness in construction multi-
©ASCE, ISSN 0733-9364/2010/1-17–25/$25.00. objective optimization models. To accomplish this objective, this

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 17

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
paper presents: 共1兲 a review of recent construction multiobjective Kuang 共2008兲 that used ACO in solving time-cost tradeoff prob-
optimization research and approaches to increase the robustness lems for construction project planning; and 共2兲 Afshar et al.
of optimization algorithms; 共2兲 two large-scale case studies to 共2007兲 that used ACO to solve time-cost-quality tradeoff prob-
analyze the robustness of naturally inspired and analytical MOAs; lems. A similar optimization technique called PSO was developed
and 共3兲 recommendations for increasing the robustness of con- by Kennedy and Eberhart 共1995兲. PSO is also a naturally inspired
struction multiobjective optimization models. algorithm that imitates the social behavior of a flock of migrating
birds trying to reach an unknown destination. Therefore, the op-
timization process in PSO involves: 共1兲 local search, where birds
Literature Review use intelligence to learn from their own experience and 共2兲 global
search, where birds use social interaction to learn from the expe-
Existing research studies that focused on construction multiobjec- rience of other birds in the flock 共Elbeltagi et al. 2005兲. An ex-
tive optimization have used a number of naturally inspired and ample of a construction multiobjective optimization model that
analytical optimization algorithms. This section reviews and dis- used PSO is that of Yang 共2007兲 that used PSO to solve the
cusses recent research efforts that used 共1兲 naturally-inspired time-cost tradeoff problems for construction project planning.
algorithms such as genetic algorithms 共GAs兲, ant-colony optimi-
zation 共ACO兲, and particle swarm optimization 共PSO兲 and 共2兲 Analytical Optimization Algorithms
analytical optimization algorithms such as linear, integer, and dy-
namic programming. The review also covers a number of ap- A number of construction multiobjective optimization models
proaches that were used to enhance the robustness of optimization have also used analytical algorithms. Many of these models used
algorithms such as parallel and distributed computing. linear, integer, and dynamic programming algorithms. For ex-
ample, Burns et al. 共1996兲 presented an optimization model that
used a hybrid method that integrated linear and integer program-
Naturally Inspired Optimization Algorithms ming for finding optimal time-cost trade-offs. The model was
Naturally inspired optimization algorithms have been developed applied in two main stages: 共1兲 linear programming stage to find
in an effort to computationally mimic natural processes that lead lower bounds of the tradeoffs and 共2兲 integer programming stage
to optimal results 共Goldberg 1989兲. For example, GAs were first to obtain the exact tradeoff. Another model that used linear and
introduced by Holland 共1975兲, and are used as search and optimi- integer programming was developed by Mattila and Abraham
zation algorithms that mimic genetic operations based on Dar- 共1998兲. This model was dedicated to the optimization of repetitive
win’s theory of natural selection 共Goldberg 1989; Marler and projects and attempted to level resource utilization for this type of
Arora 2004兲. They adopt the survival of the fittest and the struc- projects. Gomar et al. 共2002兲 introduced a model that attempts to
tured exchange of genetic materials among population members optimize the allocation of mulitskilled work crews using integer
over successive generations as a basic mechanism for the search linear programming. The developed model optimized the alloca-
process 共Goldberg 1989; El-Rayes and Kandil 2005兲. GAs have tion of multiskilled workforces in order to increase labor effi-
been widely used to solve multiobjective construction engineering ciency on the project and was validated using CII Model Plant
and management problems, including 共1兲 deterministic time-cost Data. Ipsilandis 共2007兲 used a parametric objective function to
tradeoff analysis for construction project planning 共Feng et al. develop a multiobjective linear programming model for schedul-
1997; Kandil and El-Rayes 2006b; Kasaeian et al. 2007; Schau- ing linear repetitive projects. In this model, different optimal so-
mann et al. 1998; Zheng et al. 2004兲; 共2兲 stochastic time-cost lutions can be generated by changing the values of the objective
tradeoff analysis 共Feng et al. 2000; Zheng and Ng 2005兲; 共3兲 function parameters.
time-cost-quality tradeoff analysis 共Kandil and El-Rayes 2006a兲; A number of dynamic programming models have also been
共4兲 linear repetitive projects scheduling 共Hyari and El-Rayes developed to optimize resource utilization in repetitive construc-
2006兲; 共5兲 site layout planning 共El-Rayes and Khalafallah 2005; tion projects. These models focused on 共1兲 minimizing the overall
Khalafallah and El-Rayes 2006兲; 共6兲 lighting design optimization project cost by striking an optimal balance between minimizing
for nighttime highway construction 共El-Rayes and Hyari 2005兲; the project direct cost and duration 共Moselhi and El-Rayes 1993;
共7兲 postdisaster temporary housing arrangements optimization El-Rayes and Moselhi 2001兲 and 共2兲 minimizing the total com-
共El-Anwar and El-Rayes 2007兲; 共8兲 repair and rehabilitation plan- bined bid in A + B highway contracting methods 共El-Rayes 2001兲.
ning in bridge management systems 共Lee and Kim 2007; Liu et
al. 1997; Liu and Frangopol 2005兲; 共9兲 production scheduling for
Parallel and Distributed Computing
precast plants 共Chan and Hu 2002兲; and 共10兲 double sampling
plan optimization for construction quality inspection 共Cheng and Naturally inspired and analytical optimization algorithms have
Chen 2007兲. Due to this wide utilization of GAs in construction relative strengths that make each of them better suited for certain
multiobjective optimization studies, the first and second case types of optimization problems. However, when both types of
studies in this paper will focus on analyzing the robustness of this algorithms are applied to large and more complex problems the
optimization technique. time required to solve the problem increases, decreasing their
Another important naturally inspired optimization technique is efficiency 共Balla and Lingireddy 2000; Cantú-Paz 1997a兲. This
ACO which was developed by Dorigo and Maniezzo 共1996兲. This led to a number of attempts to increase efficiency in solving large
algorithm models the natural process by which ant colonies are and complex problems using a number of approaches including:
able to find the shortest route between their nest and a source of 共1兲 global parallelization and 共2兲 coarse-grained parallelization
food. This process, therefore, simulates the use of pheromone 共Cantú-Paz 1997a兲.
trails, which ants deposit whenever they travel as a form of indi- Global parallel optimization algorithms have a single search
rect communication 共Elbeltagi et al. 2005兲. The effectiveness of space 共i.e., not segmented兲 and behave in a way similar to the
this optimization technique was tested in a number of multiobjec- serial ones 共Cantú-Paz 1997b兲. These algorithms utilize the
tive construction optimization models including: 共1兲 Xiong and manager/worker paradigm or parallel computing. This paradigm

18 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 ments, which needs a robust optimization methodology to analyze
in a reasonable and practical computational time.
The present case study includes two main optimization objec-
tives, namely: 共1兲 maximizing safety of displaced families and 共2兲
minimizing total public expenditures 共TPEs兲 on temporary hous-
ing. First, the safety of displaced families is maximized by mini-
mizing the vulnerability of temporary housing to potential
postdisaster hazards, which includes aftershocks at the earthquake
fault area and hazardous material releases at the locations of five
toxic chemical plants, as shown in Fig. 1. To this end, a safety
index 共SI j兲 is computed for each temporary housing alternative j,
which accounts for 共1兲 potential hazards; 共2兲 temporary housing
type, such as travel trailers, mobile homes, and tent camps; 共3兲
distance between each temporary housing alternative and the po-
tential hazards; 共4兲 expected level of damage in each temporary
housing in case the potential hazard occurs; and 共5兲 the probabil-
ity of each potential hazard to occur given that the natural disaster
Temporary Housing Alternative Earthquake Fault Area Toxic Chemical Plant
has occurred. In order to maximize the safety of the displaced
families, they should be assigned to temporary housing facilities
with the highest SI j, which can range from 0 共i.e., unsafe兲 to 1
Fig. 1. Impacted area by the earthquake 共i.e., completely safe兲. Second, TPEs on temporary housing is
minimized by computing the monthly ownership or lease cost
共C j兲 of each temporary housing alternative j and assigning fami-
is reported to provide a number of advantages including the in- lies to the least expensive alternatives.
creased flexibility in program structuring, and the ease of the data The following sections describe 共1兲 the model formulation; 共2兲
partitioning, mapping and load balancing procedures. Despite the model implementation using multiobjective GAs 共MOGA兲;
these advantages this environment was also reported to suffer 共3兲 the model implementation using weighted integer program-
from efficiency losses as the number of used processors increases ming; and 共4兲 the comparison between the optimization results of
共De Santiago and Law 2000; Balla and Lingireddy 2000; Alonso each optimization technique, as shown in Fig. 2.
et al. 2000兲. The second category of parallel optimization algo-
rithms is the coarse-grained parallel algorithms, which is also
known as island or distributed GAs. These algorithms implement Model Formulation
a form of data parallelism in which data are distributed over the The multiobjective optimization model is formulated in three
processors and each processor implements a separate copy of the main steps: 共1兲 designing decision variables; 共2兲 formulating op-
program independently and occasionally communicates with the timization objective; and 共3兲 modeling the optimization con-
other processors 共Thiagarajan and Aravamuthan 2002兲. straints, as shown in Fig. 2.
First, the decision variables are designed to represent the pos-
sible configurations of temporary housing arrangements. To this
First Case Study: Temporary Housing Planning end, each decision variable x j represents the number of displaced
families assigned to temporary housing alternative j. Accordingly,
In order to analyze and compare the robustness of naturally in- the number of decision variables is initially equal to the number
spired and analytical MOAs, a large-scale case study of a postdi- of available temporary housing alternatives 共J兲, and in the present
saster temporary housing planning is analyzed. The case study case study there are 222 decision variables.
requires decision-makers to identify optimal plans of temporary Second, the two main optimization objectives are formulated
housing arrangements for 20,000 displaced families after a major using Eqs. 共1兲 and 共2兲
earthquake. There are 222 available temporary housing alterna-
tives in and around the impacted area with a total capacity of J
1
39,001 families, as shown in Fig. 1. Accordingly, there is a very
large number of possible plans of temporary housing arrange-
SI =
n
⫻ 兺
j=1
共SI j ⫻ x j兲 共1兲

1. Model 2. NSGA2 Implementation 4. Optimization

Formulation Results Comparisons
Solutions Population
Initialization Ranking
Decision Evaluation Generation
Effectiveness
Variables
Optimal Tradeoffs
Optimization Quality
Objectives 3. Weighted Integer Programming Implementation
Efficiency
Optimization
Optimization Optimal Limits
Problems Optimization Computational
Identification
Constraints Generation Time

Fig. 2. Temporary housing planning case study

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 19

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 J objective functions and its performance in the other objective
TPE = 兺 共C j ⫻ x j兲 共2兲 function is at least as good as solution s2 共Deb et al. 2001兲.
j=1 Fourth, the population generation phase is designed to gener-
ate a new improved generation using selection, crossover, and
mutation. In this phase, the values of the off-springs genes are
where SI= average safety index per displaced family for a specific forced to be integers in order to fulfill the third optimization con-
configuration of temporary housing arrangements; n = number of straint. Phases 共2兲–共4兲 are then repeated until a predetermined
displaced families in need of temporary housing; J = number of number of generations are completed. In the present case study,
temporary housing alternatives; SI j = safety index for temporary the model generated 300,000 generations. Each solution in the
housing alternative j; x j = number of displaced families assigned first rank in last generation is a nondominated optimal solution
to temporary housing alternative j; TPE= TPE on temporary providing an optimal tradeoff between maximizing safety and
housing for a specific configuration of temporary housing ar- minimizing expenditures.
rangements; and C j = monthly cost of each temporary housing al-
ternative j per house family.
Third, three optimization constraints are formulated to ensure Model Implementation Using Weighted Integer
the practicality of generated optimal solutions. These constraints Programming
are designed to ensure that 共1兲 the capacity of any temporary The linear nature of optimization objectives and constraints in this
housing alternative should not be exceeded; 共2兲 all families case study enables the utilization of analytical optimization tech-
should be housed; and 共3兲 number of families assigned to any niques, such as integer programming. Weighted integer program-
temporary housing should be positive integers. ming is used in order to simultaneously optimize the two main
objectives, where each of the optimization objectives is first nor-
malized 共Marler and Arora 2004兲 and then their weighted sum is
Model Implementation Using Multiobjective Genetic maximized, as shown in Eq. 共3兲. Moreover, the three optimization
Algorithms constraints are fulfilled using Eqs. 共4兲–共6兲
Nondominated Sorting GA II 共NSGA2兲 is used to optimize this
SImax − SI TPE − TPEmin
large-scale optimization problem, because of its important char- maximize WSI ⫻ − WTPE ⫻
acteristics such as fast nondominated sorting, crowding, elitism, SImax − SImin TPEmax − TPEmin
and constrained-domination principle; in addition to its superior 共3兲
performance compared to other MOGAs 共D’Souza and Simpson
subject to Constraint 1: capacities of temporary housing alterna-
2002; Deb et al. 2001; Deb 2005; Weile et al. 1996兲. The model is
tives should not be exceeded
implemented in four main phases: 共1兲 initialization; 共2兲 solutions
evaluation; 共3兲 ranking; and 共4兲 population generation, as shown x j 艋 cap j, ∀ j = 1,2 . . . J 共4兲
in Fig. 2.
First, the initialization phase is designed to generate a popula- Constraint 2: all displaced families should be housed
tion of random solutions 共chromosomes兲, where each population J
in this case study is specified to consist of 2,000 solutions. Each
solution consists of a number of real-coded genes, where each 兺
j=1
x j = n, ∀ j = 1,2 . . . J 共5兲
gene represents one of decision variables 共x j兲. Therefore, in the
present case study each solution consists of 222 genes. In addi- Constraint 3: All assignments should have positive integer values
tion, in order to fulfill the first optimization constraint the value of
x j 艌 0, x j is an integer, ∀ j = 1,2 . . . J 共6兲
each gene 共x j兲 is set to range between 0 and the capacity 共cap j兲 of
temporary housing alternative j, which ensures that the number of where WSI = relative weight of maximizing safety of displaced
families assigned to any temporary housing will not exceed its families; SImax and SImin = maximum and minimum values of av-
capacity. erage safety index per displaced family among the optimal solu-
Second, the solution evaluation phase is designed to evaluate tions, respectively; SI= average safety index per displaced family
each generated solution based on its performance in 共1兲 the first for a specific configuration of temporary housing arrangements,
objective function, which represents maximizing safety for dis- which can be computed using Eq. 共1兲; WTPE = relative weight of
placed families and is computed using Eq. 共1兲; 共2兲 the second minimizing TPEs on temporary housing; TPEmax and TPEmin
objective function, which represents minimizing TPEs on tempo- = maximum and minimum values of TPEs among the optimal
rary housing and is computed using Eq. 共2兲; and 共3兲 a constraint solutions, respectively; TPE= TPE on temporary housing for
compliance function ensuring that the sum of the number of fami- a specific configuration of temporary housing arrangements;
lies assigned to all temporary housing alternatives is equal to the x j = number of displaced families assigned to temporary housing
number of displaced families in order to fulfill the second optimi- alternative j; cap j = capacity of temporary housing alternative j
zation constraint. It should be noted that any solution not fulfilling in number of families; J = number of temporary housing alterna-
that constraint is considered infeasible. tives; and n = number of displaced families in need of temporary
Third, the ranking phase is used to rank all the solutions based housing.
on the constrained-domination principle. In this approach, the In order to generate the optimal tradeoffs between maximizing
evaluated solutions are ranked based on their domination of other safety and minimizing expenditures, the weighted integer pro-
solutions, where solution s1 is said to dominate solution s2 if any gramming model is implemented in three main phases: 共1兲 opti-
of the following conditions is true: 共1兲 solution s1 is feasible and mal limits identification; 共2兲 optimization problems generation;
solution s2 is infeasible; 共2兲 both solutions are infeasible, but so- and 共3兲 optimization, as shown in Fig. 2.
lution s1 has a smaller constraint violation; or 共3兲 both solutions First, the optimal limits identification phase is designed to
are feasible, but solution s1 outperforms solution s2 in one of the identify SImax, SImin, TPEmax, and TPEmin, which is necessary for

20 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Optimal Tradeoffs identify better results with more generations, but the rate of im-
0.82 provement in the quality of solutions is decreasing; and 共3兲 all the
0.8
generated solutions are concentrated at the middle portion of the
Pareto front. On the other hand, the optimization results of using
0.78
weighted integer programming illustrate the following: 共1兲 it
0.76 could identify optimal solutions; and 共2兲 the solutions are distrib-
Safety Inde SI

NSGA2 (300,000 generations)

0.74 NSGA2 (100,000 generations) uted along the Pareto front. These results illustrate that weighted
NSGA2 (10,000 generations) integer programming outperforms NSGA2 in terms of the quality
0.72 Weighted Integer Programming
of generated optimal tradeoffs between maximizing safety and
0.7 minimizing expenditures for the temporary housing planning
0.68
problem.
Second, the efficiency of the two techniques can be compared
0.66
using the total computation time for optimizing this large-scale
0.64 temporary housing problem on a 2.6 GHz Intel Pentium 4 proces-
30 32 34 36 38 40 42 44 46 sor with 512 MB of RAM. The total computation time required
Total Public Ependitures millions of  by the NSGA2 to complete 300,000 generations was approxi-
mately 90 h; while that required by the weighted integer program-
Fig. 3. Optimization results ming was less than one minute. These results clearly illustrate that
the computational efficiency of the weighted integer program-
ming model outperform those of NSGA2 for this particular large-
normalizing the optimization objectives as shown in Eq. 共3兲. SImax scale temporary housing planning problem. It should be noted,
is identified by maximizing Eq. 共1兲 using integer programming however, that the efficiency of NSGA2 can be significantly im-
and the corresponding value of TPEs is stored as TPEmax. Simi- proved using parallel computing paradigms, especially for large-
larly, TPEmin is identified by minimizing Eq. 共2兲 using integer scale construction optimization problems that cannot be modeled
programming and the corresponding value of average safety using analytic optimization algorithms. The following case study
index is stored as SImin. analyzes one of these problems and investigates the potential of
Second, the optimization problems generation phase is de- parallel computing to enhance the computation efficiency.
signed to generate unique combinations of relative weights 共WSI
and WTPE兲 for the two optimization objectives, where the sum of
those weights is always equal to 1 共i.e., WSI and WTPE = 1兲. Each Second Case Study: Construction Resource
unique combination represents a unique integer programming op- Optimization
timization problem, which is sent to the optimization phase to
generate a unique optimal solution. The number of unique com- This case study is analyzed to investigate the robustness of natu-
binations, and accordingly the number of unique optimal solu- rally inspired MOAs and parallel computing paradigms in opti-
tions, is equal to k + 1, where k is equal to 100 divided by the mizing resource utilization in large-scale infrastructure projects.
value of the weight increment, which is set by the decision-maker. The optimization is performed using a MOGA called the Non-
In the present case study a weight increment of 5% is used, and dominated Sorting GA II 共NSGA2兲. Despite its advantages and
therefore the optimization problems generation phase generates flexibility in enabling multiobjective optimization for linear and
21 unique combinations of relative weights. nonlinear problems, NSGA2 often requires extensive computation
Third, the optimization phase is designed to generate a unique time to solve large-scale optimization problems. To investigate
optimal tradeoff between maximizing safety and minimizing ex- and enhance this critical robustness issue, this case study focuses
penditures for each unique combination of relative weights by on: 共1兲 model formulation and implementation using NSGA2; 共2兲
maximizing Eq. 共3兲 using weighted integer programming. In the the utilization of parallel computing paradigms to enhance the
present case study the optimization phase generates 21 unique robustness of the optimization model; and 共3兲 analyzing the effi-
optimal solutions each representing a unique tradeoff between the ciency and effectiveness of the implemented NSGA2 model and
two main optimization objectives. parallel computing paradigms.

Comparison of Analyzed Optimization Techniques Model Formulation and Implementation

This section compares the results obtained from the MOGAs An optimization model is formulated to support construction en-
using NSGA2 and the weighted integer programming. The results gineers and managers in identifying optimal resource utilization
are compared based on two criteria: 共1兲 effectiveness that evalu- plans that provide optimal tradeoffs between construction time
ates the quality of the generated optimal tradeoffs; and 共2兲 effi- and cost. Accordingly, the main decision variables considered in
ciency that measures the required computational time, as shown this model for each construction activity in the project include:
in Fig. 3. 共1兲 construction method, which represents the different types of
First, the effectiveness of the two techniques can be compared materials and/or methods that can be used in the construction of
using Fig. 3 that shows the optimal tradeoffs identified by the activity; 共2兲 crew formation, which models available sizes and
NSGA2 after 10,000, 100,000, and 300,000 generations using a configurations for construction crews; and 共3兲 crew overtime
populations size of 2,000; and those identified by weighted inte- policy, which represents available overtime hours and nighttime
ger programming using a 5% increment. As shown in Fig. 3, the shifts. These three main decision variables are then combined into
optimization results of using NSGA2 illustrate the following: 共1兲 one called resource utilization option 共n兲 in order to reduce the
NSGA2 could reach near optimal solutions after 300,000 genera- complexity of the optimization problem. The main goal of the
tions; 共2兲 NSGA2 did not converge yet and has the potential to optimization algorithm employed to this problem is then to select

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 21

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 140 320
Global Parallel
Coarse Grained Every Generation 300
120 Coarse Grained Every 2 Generations Global Parallel
280

Nondominated Optimal Points

Coarse Grained Every 4 Generations Coarse Grained Every Generation
100 260

Elapsed Time (Hrs)

Coarse Grained Every 2 Generations
240 Coarse Grained Every 4 Generations
80
220

60 200

180
40
160

20 140

120
0
100
1 5 10 15 20 25 30 35 40 45 50
1 5 10 15 20 25 30 35 40 45 50
Processors Processors

Fig. 4. Efficiency and effectiveness of parallel computing approaches

the optimal resource utilization option for each activity in the developed MOGA. Accordingly, two parallel programming para-
project. Since the employed naturally inspired MOA is a MOGA, digms are implemented and tested to analyze their impact on
these resource utilization are modeled in a GA string with each improving the efficiency of the developed MOGA. The two par-
location of that string representing a resource utilization option of allel computing paradigms that were analyzed in this case study
one of the project activities. are global parallelization and coarse-grained parallelization.
The fitness of the formulated GA strings needs to be evaluated The global parallel GA uses the global paradigm of paral-
in order for NSGA2 stages to be implemented. The two main lel computing for increasing the efficiency of the MOGA 共De
objectives considered in the present case study were project cost Santiago and Law 2000兲. In this approach, one processor is as-
and time, as shown in Eqs. 共7兲 and 共8兲 signed the role of a manager processor that manages the compu-
tations on the remaining worker processors. The manager
兺i=1 关共M ni + Dni ⫻ Rni 兲 + 共Bni 兲兴
t
Minimize Project Cost = 共7兲 processor starts by implementing the initialization task of the
MOGA, and then breaks down the generated population of solu-
where M ni = material cost of activity 共i兲 using resource utilization
tions into a number of subpopulations and equally distributes
共n兲; Dni = duration of activity 共i兲 using resource utilization 共n兲; them to all the worker processors in the used computing cluster.
Rni = daily cost rate in dollars per day of resource utilization 共n兲 in The worker processors are then used to evaluate the fitness 共i.e.,
activity 共i兲; Bni = subcontractor lump sum cost for resource utiliza- project duration and cost兲 of all solutions in their subpopulations
tion 共n兲 in activity 共i兲, if any and return their evaluated solutions to the manager processor. The
兺i=1 Tni
t manager processor then performs the generation evolution task of
Minimize Project Time = 共8兲
the multiobjective optimization module in order to create the next
where Tni = duration of activity 共i兲 on the critical path using re- generation of solutions. The generation evolution and fitness
source utilization 共n兲. In order to determine the critical path in evaluation tasks are repeated iteratively using the manager and
project activity networks, the critical path method 共CPM兲 is em- worker processors until the convergence criteria are met. The
ployed to calculate activity times and floats. The utilization of the implementation of this parallel GA for a construction project size
CPM requires the implementation of graph traversal algorithm of 720 activities was able to reduce the overall computational
which makes the representation of this objective in closed math- time from 136.50 h on a single processor to 19.72 h using a
ematical form infeasible which precludes the utilization of ana- cluster of 20 processors, as shown in Fig. 4. Despite this signifi-
lytic optimization algorithms. Accordingly, this formulated model cant time saving, the performance of the global parallel comput-
was implemented as an NSGA2 model using a similar procedure ing paradigm used in this module was reported in the literature to
to that described earlier in the first case study. suffer from efficiency losses especially as the number of proces-
sors increases 共De Santiago and Law 2000兲, which was the main
motivation to examine the second paradigm of coarse-grained
Parallel Computing to Enhance Robustness parallel computing.
To examine the efficiency of the developed MOGA, a preliminary The course-grained parallel GA on the other hand, employs
analysis was performed to measure the needed GA processing data parallelism by dividing the evaluated population of solutions
time. In this analysis, the MOGA was used to simultaneously into a number of subpopulations called demes that are evolved
minimize the duration and cost for a large-scale project that in- independently on a number of parallel processors. Accordingly,
cludes 720 activities. The resource utilization for this project was all processors involved in the computations perform all the tasks
optimized using a single computer/processor in order to identify, of the MOGA in a completely nonhierarchical fashion, which
for each activity in the project, an optimal selection of construc- eliminates the need for a single processor to manage and dis-
tion method, crew formation and/or overtime policy. The results tribute the computations 共Cantú-Paz 1997a; Cantú-Paz 2000;
of this preliminary analysis indicated that the GA computations Noda et al. 2002; Adeli 2000兲. The nonhierarchical structure of
on a single processor/computer required approximately 137 h for the module also leads to the elimination of synchronized commu-
optimizing this preliminary analysis. The findings of this analysis nications, which also allows for further time savings and speed-
confirm the need for methods to enhance the efficiency of the ups 共Cantú-Paz 2000兲. The asynchronous communications

22 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
implemented in this framework also prevent slower processors in Formulation of Optimization Problem
the computation from impeding the progress of the remaining
The formulation of construction multiobjective optimization
faster processors in the cluster. In order to compensate for the
problems is one of the important stages that has a significant
independent evolution of the subpopulation 共i.e., demes兲 in this
impact on the effectiveness and efficiency of the optimization
paradigm, the present algorithm enables each processor to share
process. Accordingly, the following recommendations can be
and exchange the best found solutions in its deme with the re- used to enhance robustness whenever they are feasible and can be
maining processors in the cluster using a procedure called the applied.
migration process 共Cantú-Paz 1997b兲. 1. Minimize the number of decision variables, if possible. De-
creasing the number of decision of variables by possibly
combining them 共similar to the decision variables combined
Comparison of Parallel GA Paradigms
in the second case study of this paper兲 decreases the com-
The performance of the aforementioned parallel GA paradigms plexity of the optimization and allows for it to be solved
was evaluated in this case study using 44 experiments that tested more efficiently.
various combinations of: 共1兲 eleven different clusters of parallel 2. Minimize or relax constraints if possible. The reduction of
processors that ranged from 1 to 50 with an increment of 5 pro- optimization constraints has the potential of creating a non-
cessors, as shown in Fig. 4; 共2兲 two migration rates in the coarse- constrained optimization problem, which is particularly use-
grained parallel module that specify the transfer of the top 25% ful in naturally inspired optimization algorithms because
many of those algorithms do not have fully mature mecha-
and 75% of the solutions in each deme; and 共3兲 three migration
nisms for constraint handling.
intervals that require the execution of the migration procedure
3. Favoring objective functions that could be modeled in closed
every one, two, and four generations, as shown in Fig. 4. The
mathematical form. This allows the employment of analyti-
main metric used to judge the efficiency of the two parallel com-
cal algorithms. The use of objective functions that are
puting approaches is the Elapsed time of the algorithm, while the modeled through a computer algorithm or simulation may
main metric for judging effectiveness of the approach was the preclude the use of analytical optimization algorithms.
number of nondominated points obtained. The used number of
parallel processors as well as the specified migration rate and
interval affected both the GA efficiency and effectiveness 共Noda Selection of the Optimization Algorithm
et al. 2002兲. First, increasing the number of parallel processors led
The selection of the optimization algorithm should be based on
to: 共1兲 an increase in the GA efficiency due to the decrease in the the type and nature of the problem. Based on the conducted
overall computational time as shown in Fig. 4; and 共2兲 a decrease analysis of the two case studies, the following recommendations
in the GA effectiveness due to the reduced size of evaluated can be applied to enhance the robustness of the optimization
subpopulations/demes. Second, increasing the migration rate and process.
interval produced: 共1兲 reduced GA efficiency due to the increase 1. The first case study illustrated that the optimization effective-
in migration and communication times among processors; and 共2兲 ness and efficiency of analytic optimization algorithms out-
improved GA effectiveness due the increase in the exchange rate perform those of the analyzed naturally inspired one.
and interval of best found solutions among analyzed demes lead- Therefore, the use of analytic optimization algorithms can be
ing to improvements in the explorative and exploitive power of preferred, whenever possible, over naturally inspired algo-
the algorithm 共Cantú-Paz 2000; Noda et al. 2002兲. rithms.
Additionally, the above results show that the coarse-grained 2. The second case study illustrated that many construction op-
parallel approach did not lose effectiveness as its efficiency in- timization problems may not be good candidates for analytic
creased, while the coarse-grained approach showed a tradeoff be- optimization algorithms due to the mathematical formulation
tween its efficiency and effectiveness. The coarse-grained of its objectives. Accordingly, these problems can best be
approach, however, was able achieve higher increases in effi- optimized using naturally inspired optimization algorithms in
ciency as the number of processors used increased. These results combination with parallel computing paradigms to enhance
show the feasibility of parallel computing paradigms and their the computation time and optimization efficiency, especially
potential to enhance the robustness of the developed MOGA. in large-scale construction optimization problems.
3. Optimization algorithms require extensive testing and valida-
tion to ensure that they function properly. Many optimization
algorithms are currently present in proprietary or open-
Recommendations to Enhance Robustness source libraries that facilitate the development of optimiza-
and Research Rigor tion models. Therefore, researchers should favor those
implementations, available from trusted sources, over devel-
The above explained case studies showed that the performance of oping their own code if possible. If not, researchers should
MOAs and parallel computing approaches varied depending on use well tested code as a benchmark for their developed
the type and size of the construction optimization problem ad- implementations.
dressed. Therefore, the main findings of the above two case stud-
ies are used to develop a number of recommendations to enhance
Testing and Validation
the robustness and research rigor in this critical area of construc-
tion research. These recommendations focus on enhancing: 共1兲 The testing and validation of optimization models is a challenging
the formulation of the optimization problem; 共2兲 the selection of task. The main challenge in many construction optimization prob-
optimization algorithm; and 共3兲 the testing and validation of op- lems is that the true optimal solutions are not really known, and if
timization algorithms. know from other research studies, that solution may not be a true

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 23

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
optimum. In order to overcome this challenge, the following rec- Alonso, J., et al. 共2000兲. “Parallel computing in water network analysis
ommendations can be applied whenever possible. and leakage minimization.” J. Water Resour. Plann. Manage., 126共4兲,
1. The validation could be performed by analyzing the optimi- 251–260.
zation problem using multiple optimization algorithms to Balla, M., and Lingireddy, S. 共2000兲. “Distributed genetic algorithm
evaluate the relative performance of the algorithms giving model on network of personal computers.” J. Comput. Civ. Eng.,
14共3兲, 199–205.
preference to the results of algorithms with better perfor-
Burns, S. A., Liu, L., and Feng, C. 共1996兲. “The LP/IP hybrid method for
mance on standard problems of equivalent complexity.
construction time-cost trade-off analysis.” Constr. Manage. Econom.,
2. If an analytical optimization algorithm requiring an initial 14, 265–276.
starting point, the algorithm should executed with multiple Cantú-Paz, E. 共1997a兲. “A survey of parallel genetic algorithms.”
starting points to ensure convergence to a true optimal solu- ILLGAL Rep. No. 97003, Illinois Genetic Algorithms Laboratory,
tion. Univ. of Illinois at Urbana Champaign, Urbana, Ill.
3. If a small-scale problem exists such that all solutions could Cantú-Paz, E. 共1997b兲. “Designing efficient master-slave parallel genetic
be enumerated to find its optimal solution, that problem algorithms.” ILLGAL Rep. No. 97004, Illinois Genetic Algorithms
should be used initially for testing the used optimization al- Laboratory, Univ. of Illinois at Urbana Champaign, Urbana, Ill.
gorithm before testing the algorithm on large-sized problems. Cantú-Paz, E. 共2000兲. Efficient and accurate parallel genetic algorithms,
Kluwer Academic, Boston.
Carty, G. 共1995兲. “Construction.” J. Constr. Engrg. Manage., 121共3兲,
Conclusions 319–328.
Chan, W., and Hu, H. 共2002兲. “Production scheduling for precast plants
This paper presented a review of current research efforts in the using a flow shop sequencing model.” J. Comput. Civ. Eng., 16共3兲,
field of construction MOAs and a detailed analysis of two large- 165–174.
scale case studies that were designed to evaluate the robustness of Cheng, T., and Chen, Y. 共2007兲. “A GA mechanism for optimizing the
design of attribute double sampling plan.” Autom. Construct., 16,
naturally inspired and analytical MOAs. The first case study
345–353.
aimed at optimizing the planning of postdisaster temporary hous-
D’Souza, B., and Simpson, T. 共2002兲. “A genetic algorithm based method
ing. This case study first used a MOGA, however the processing
for product family design optimization.” Proc., DETC 2002: 2002
time was found to be very large. The case study then used an
ASME Design Engineering Technical Conf., ASME, New York.
analytical optimization algorithm that was found to outperform
De Santiago, E., and Law, K. H. 共2000兲. “A distributed implementation of
the MOGA in both effectiveness and efficiency. The second case an adaptive finite-element method for fluid problems.” Comput.
study aimed at optimizing construction resource utilization in Struc., 74共1兲, 97–119.
large-scale infrastructure projects. In this case study, a MOGA Deb, K. 共2005兲. ”Real-coded genetic algorithms.” Kanpur Genetic Algo-
was used for performing the optimization process. In order to rithm Laboratory, Indian Institute of Technology, Kanpur, India,
increase the robustness of this algorithm, two parallel computing 具http://www.iitk.ac.in/kangal/resources.shtml典 共Nov. 12, 2005兲.
approaches were used. The results of these approaches illustrated Deb, K., Agrawal, S., Pratap, A., and Meyarivan, T. 共2001兲. “A fast elitist
that significant efficiency increases could be achieved, and that in non-dominated sorting genetic algorithm for multi-objective optimi-
some cases, tradeoffs exist between efficiency and effectiveness. zation.” KANGAL Rep. No. 200001, Genetic Algorithm Laboratory,
The paper also presented a number of recommendations enhance Indian Institute of Technology, Kanpur, India.
the robustness of MOAs and the level of rigor in the research Dorigo, M., and Maniezzo, V. 共1996兲. “ColorniA. Ant system: Optimiza-
performed in this field. The recommendations focused on: 共1兲 tion by a colony of cooperating agents.” IEEE Trans. Syst. Man Cy-
problem formulation; 共2兲 selection of the optimization algorithm; bern., 26共1兲, 29–41.
El-Anwar, O., and El-Rayes, K. 共2007兲. “Post-disaster optimization of
and 共3兲 testing and validation methods. These recommendations
are expected to enhance the development and utilization of mul- temporary housing efforts.” Proc., ASCE Construction Research Con-
tiobjective construction optimization models, however further re- gress, ASCE, Reston, Va.
El-Rayes, K. 共2001兲. “Optimum planning of highway construction
search is needed to analyze and address other causes that still
under the A + B bidding method.” J. Constr. Eng. Manage., 127共4兲,
limit the widespread utilization of optimization techniques in the
261–269.
construction industry. El-Rayes, K. and Hyari, K. 共2005兲. “Optimal lighting arrangements for
nighttime highway construction projects.” J. Constr. Eng. Manage.,
131共12兲, 1292–1300.
Acknowledgments El-Rayes, K. and Kandil, A. 共2005兲. “Time-cost-quality trade off analysis
for highway construction.” J. Constr. Eng. and Manage., 131共4兲,
This material is based upon work supported by the Qatar National 477–486.
Research Fund under Award No. QNRF-NPRP26-6-7-2. Any El-Rayes, K. and Khalafallah, A. 共2005兲. “Tradeoff between safety and
opinions, findings, and conclusions or recommendations ex- cost in planning construction site layouts.” J. Constr. Eng. Manage.,
131共11兲, 1186–1195.
pressed in this publication are those of the writers and do not
El-Rayes, K., and Moselhi, O. 共2001兲. “Optimizing resource utilization
necessarily reflect the views of the Qatar National Research Fund.
for repetitive construction projects.” J. Constr. Eng. Manage., 127共1兲,
18–27.
Elbeltagi, E., Hegazy, T., and Grierson, D. 共2005兲. “Comparison among
References five evolutionary-based optimization algorithms.” Adv. Eng. Inf.,
19共1兲, 43–53.
Adeli, H. 共2000兲. “High performance computing for large-scale analysis, Feng, C., Liu, L., and Burns, S. 共1997兲. “Using genetic algorithms to
optimization, and control.” J. Aerosp. Eng., 13共1兲, 1–10. solve construction time-cost trade-off problems.” J. Comput. Civ.
Afshar, A., Kaveh, A., and Shoghli, O. 共2007兲. “Multi-objective op- Eng., 11共3兲, 184–189.
timization of time-cost-quality using mutli-colony ant algorithm.” Feng, C., Liu, L., and Burns, S. 共2000兲. “Stochastic construction time-
Asian J. Civil Engineering (Building and Housing), 8共2兲, 113–124. cost trade-off analysis.” J. Comput. Civ. Eng., 14共2兲, 117–126.

24 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Goldberg, D. E. 共1989兲. Genetic algorithms in search, optimization and tion.” Comput. Aided Civ. Infrastruct. Eng., 20, 343–353.
machine learning, Addison-Wesley, Reading, Mass. Marler, R. T., and Arora, J. S. 共2004兲. “Survey of multi-objective optimi-
Gomar, J., Haas, C. and Morton, D. 共2002兲. “Assignment and allocation zation methods for engineering.” Struct. Multidiscip. Optim., 26共6兲,
optimization of partially multiskilled workforce.” J. Constr. Eng. 369–395.
Manage., 128共2兲, 103–109. Mattila, K., and Abraham, D. 共1998兲 “Resource leveling of linear sched-
Holland, J. H. 共1975兲. Adaptation in natural and artificial systems, Uni- ules using integer linear programming.” J. Constr. Eng. Manage.,
versity of Michigan Press, Ann Arbor, Mich. 124共3兲, 232–244.
Hyari, K. and El-Rayes, K. 共2006兲. “Optimal planning and scheduling for Moselhi, O., and El-Rayes, K. 共1993兲. “Scheduling of repetitive projects
repetitive construction projects.” J. Manage. Eng., 22共1兲, 11–19. with cost optimization.” J. Constr. Eng. Manage., 119共4兲, 681–697.
Ipsilandis, P. 共2007兲. “Multiobjective linear programming model for Noda, E., Coelho, A., Ricarte, I., Yamakami, A., and Freitas, A. 共2002兲.
scheduling linear repetitive projects.” J. Constr. Eng. Manage., “Devising adaptive migration policies for cooperative distributed ge-
133共6兲, 417–424. netic algorithms.” Proc., 2002 IEEE Int. Conf. on Systems, Man and
Kandil, A. and El-Rayes, K. 共2006a兲. “MACROS: Multi-objective auto- Cybernetics, IEEE, Piscataway, N.J.
mated construction resource optimization system.” J. Manage. Eng., Schaumann, E. J., Balling, R., and Day, K. 共1998兲. “Genetic algorithms
22共3兲, 126–134. with multiple objectives.” American Institute of Aeronautics and As-
Kandil, A. and El-Rayes, K. 共2006b兲. “Parallel genetic algorithms for
tronautics, Inc., Rep. No. A98-39919, AIAA, Reston, Va.
optimizing resource utilization in large-scale construction projects.”
Thiagarajan, G., and Aravamuthan, V. 共2002兲. “Parallel strategies for
J. Constr. Eng. Manage., 132共5兲, 491–498.
element-by-element preconditioned conjugate gradient solver using
Kasaeian, A., Shoghli, O., and Afshar, A. 共2007兲. “Nondominated ar-
high performance Fortran for unstructured finite element applications
chiving genetic algorithm for multi-objective optimization of time-
on Linux clusters.” J. Comp. Civ. Eng., 16共1兲, 1–10.
cost trade-off.” Proc., 8th WSEAS Int. Conf. on Evolutionary
U.S. Bureau of Economic Analysis 共BEA兲. 共2007兲. Annual industry re-
Computing, WSEAS.
ports, 具http://www.bea.gov/industry/gpotables/gpo_action.cfm?anon
Kennedy, J., and Eberhart, R. 共1995兲. “Particle swarm optimization.”
⫽54811&table_id⫽19025&format_type⫽0典 共June 1, 2007兲.
Proc., IEEE Int. Conf. on Neural Networks, IEEE, Piscataway, N.J.,
Weile, D. S., Michielssen, E., and Goldberg, D. E. 共1996兲. “Genetic al-
1942–1948.
gorithm design of pareto-optimal broadband microwave absorbers.”
Khalafallah, A. and El-Rayes, K. 共2006兲. “Optimizing airport construc-
IEEE Trans. Electromagn. Compat., 38共3兲, 518–525.
tion site layouts to minimize wildlife hazards.” J. Manage. Eng.,
Xiong, Y., and Kuang, Y. 共2008兲. “Applying an ant colony optimization
22共4兲, 176–185.
Lee, C., and Kim, S. 共2007兲. “GA-based algorithm for selecting optimal algorithm-based multiobjective approach for time-cost trade-off.”
repair and rehabilitation methods for reinforced concrete 共RC兲 bridge J. Constr. Eng. Manage., 134共2兲, 153–156.
decks.” Autom. Constr., 16, 153–164. Yang, I. 共2007兲. “Using elitist particle swarm optimization to facilitate
Levitt, R. 共2007兲. “CEM research for the next 50 years: Maximizing bicriterion time-cost trade-off analysis.” J. Constr. Eng. Manage.,
economic, environmental, and societal value of the built environ- 133共7兲, 498–505.
ment.” J. Constr. Eng. Manage., 133共9兲, 619–628. Zheng, D., and Ng, S. 共2005兲. “Stochastic time-cost optimization model
Liu, C., Hammad, A., and Itoh, Y. 共1997兲. “Multiobjective optimization incorporating fuzzy sets theory and nonreplaceable front.” J. Constr.
of bridge deck rehabilitation using a genetic algorithm.” Microcom- Eng. Manage., 131共2兲, 176–186.
put. Civ. Eng., 12, 431–443. Zheng, D., Ng, T., and Kumaraswamy, M. 共2004兲. “Applying a genetic
Liu, M., and Frangopol, D. 共2005兲. “Bridge annual maintenance prioriti- algorithm-based multiobjective approach for time-cost optimization.”
zation under uncertainty by multiobjective combinatorial optimiza- J. Constr. Eng. Manage., 130共2兲, 168–176.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 25

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Experimental Research in Construction
Leonhard E. Bernold, M.ASCE1; and Tai Sik Lee, M.ASCE2

Abstract: Humans’ unique use of tools originated with their capability to observe, to link causes with effects and to conduct trial and
error experiments. Experimental research today is still using these same basic elements albeit augmented by sophisticated tools and
methods. Researchers conducting scientific experiments in the construction arena, however, face a “harsh” environment to work in. This
paper draws from 20 years of experimental work in construction and engineering education. It provides a short historic background before
discussing a framework useful to categorize the various ways researchers conducted experiments that provided meaningful results. Many
examples are used to underscore the most important points.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000085
CE Database subject headings: Experimentation; Research; Comparative studies; Construction management; Engineering education.
Author keywords: Experimental research; Controlled experiments; Comparative evaluations; Pilot study; Consent forms.

Introduction and Background Galilei namely the Frenchman Rene Descartes born in 1596. Des-
cartes not only received a solid education but also served in the
“The more I know, the more I know that I don’t know,” is a French military as an Ingenieur Civil. His most important contri-
frustration uttered by many researchers especially those involved bution was the Discourse on the Method of Rightly Conducting
in experimental research. Nevertheless, one can safely assume the Reason, and Searching for Truth in the Sciences (Discours de
that this kind of intellectual pursuit is as old as humans’ capability la méthode pour bien conduire sa raison, et chercher la verité
to observe and to establish causal relationships. Observing does dans les sciences) which he published in 1637. It included the
not simply mean watching a bird fly but to understand the phe- memorable assertion, “Je pense, donc je suis” which translates
nomenon of flight that allows them to traveling long distances in into “I think, therefore I am.” According to his biographer Des-
the air. Discerning natural occurrences helped them to develop mond Clarke 共Clarke 2006兲, Descartes emphasized but cautioned
not one but various methods to start a fire or to develop a series of about the value and importance of experiments to advance knowl-
medicines to treat an illness. A major step to establishing “tighter” edge. Lamenting the complexity of nature as overwhelming he
and much more reliable cause-effect associations was the emer- stressed the need to break complex investigations into their sim-
gence of mathematics. Ibn al-Haytham, an early Iraqi physicist plest components and to commence experimental discoveries with
who lived from 965–1039, established the first regimented ap- the simplest steps. Most vigorously, however, he demands that
proach to conducting research. Al-Haytham’s recommended pro- every researcher document design and conduct results of experi-
cedure listed seven steps of inquiry is still valid today: “共1兲 ments without leaving out any detail. The latter requirement is
observation; 共2兲 statement of problem; 共3兲 formulation of hypoth- based on his frustration caused by not being able to verify the
esis; 共4兲 testing of hypothesis using experimentation, 共5兲 Analysis work of other “discoverers” of the time as they commonly pro-
of experimental results; 共6兲 interpretation of data and formulation vided very little in way of descriptions beyond their claims. Des-
of conclusion; and 共7兲 publication of findings” 共Bradley 2006兲. cartes found many followers who published a steady stream of
While not the first in recognizing its importance, Ibn al-Haytham methods that improved the validity of experimental findings
mostly through proper design and evaluation of the measured
stipulated the use of experiments and the capturing of data as a
data. The basic premise, of course, was to make the experiment
means to prove or disprove a hypothesis.
reproducible so that the results could be reproduced if the descrip-
The final breakthrough for scientific methods was paved by the
tions were followed.
uneducated Galileo Galilei. After a long standoff, he forced the
The late 1920s and 1930s brought serious of behavioral ex-
establishment to accept “a posteriori knowledge” that had been
periments that revitalized Descartes’ call to skepticism concerning
discovered through the use of scientific methods but contradicted
the meaning of data. What is referred to as Hawthorne effect or
long established dogmas. Another “stepping stone” leading to
observer/experimenter bias began with a set of illumination ex-
modern experimental science was provided by a contemporary of
periments at the Hawthorne works of the Western Electric Com-
pany in Chicago. The goal was to find the lighting condition that
1
Professor, Dept. of Civil and Environmental Systems Engineering, allowed the workers involved in winding coils and assembling
Hanyang Univ., 426-791 Korea. E-mail: [email protected] relays to be more productive. After a long study that involved the
2
Professor, Dept. of Civil and Environmental Systems Engineering, workers themselves, the light bulbs for a test group were made
Hanyang Univ., 426-791, Korea. E-mail: [email protected]
brighter while a comparison group had to continue with the old
Note. This manuscript was submitted on August 6, 2008; approved on
April 30, 2009; published online on May 4, 2009. Discussion period open ones. Strangely, the production of both the test as well as the
until June 1, 2010; separate discussions must be submitted for individual control group improved. Equally unexpected was the drop of the
papers. This paper is part of the Journal of Construction Engineering productivity after researchers had left. In fact, the productivity
and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, ISSN before and after the change of the lighting system was exactly the
0733-9364/2010/1-26–35/$25.00. same. It was subsequently discovered that the researchers’ actions

26 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 1. Procedure to Conduct Experimental Research is backed up by evidence rather than established through intuitive
Main activity reasoning. In natural as well as in social sciences, the scientific
experiment is the main tool of empirical research not only to
1. Definition of a vital question for which no answer
discover or explain our world but also in proving new theories.
exists
The tenet of scientific discovery in engineering is a logic that is
2. Establishment of the state-of-the-art 共data,
based on the causality of nature where a phenomenon can be
information, expertise, etc.兲
explained by establishing relationships to its causes. As even
3. Formulation of subquestions each accompanied by a
hypothesis 共a priori兲
early thinkers have pointed out 共e.g., Ibn al-Haytham兲 those
cause-and-effect relationships can be extremely “tricky” to dis-
4. Design and development of experimental procedures
and tools cern without offering a critical mind to doubt it. As a result, Ibn
5. Design and calibration/validation of methods needed
al-Haytham’s 7 steps have been more clearly defined over time
to measure and observe and constitute the accepted procedure to conduct scientific re-
6. Execution of experiments accompanied by data search involving experiments.
collection Steps 4–6 in Table 1 are uniquely tied to experimental work
7. Statistically sound analysis of data collected during while other steps are part of other methods of research as well.
experiments Consequentially, the remainder of this paper will focus on dis-
8. Interpretation of data to establish cause-effect cussing the fundamental steps of: 共1兲 design of experiment, 共2兲
relationships that either prove or disprove the testing of measuring methods, and 共3兲 implementation of experi-
hypothesis ments with data collection.
9. Conclusions founded on the interpreted data and
publication of results
10. Stipulation of new questions that have arisen Elegant Case Study: Curing Scurvy

共e.g., the interactions with the workers兲 impacted the outcome of Medicine is a field that has probable the longest history in con-
the experiment. Feeling as being a part of the scientific study the ducting experiments, unfortunately until recently only on a trial a
workers changed their original behavior only to fall back when error basis. Nevertheless, the history of medical research has
the original factory management team returned. many excellent examples of experimentations that were not only
Finally, a “lethal” problem that every experimental researcher well designed but led to important discoveries. One such case is
faces is bias. Expecting and whishing for a certain outcome, a the experiment conducted by James Lind in 1747, who, as a Brit-
biased researcher tends to accept data, even false data, that sup- ish Royal Navy surgeon, was faced with a serious killer disease
ports the expected outcome while discounting what does not fit. for which so many tried to find a cure. His approach to this
Only a critical and open examination of the data by a range of problem was the first controlled scientific experiment that has
experts will provide an adequate barrier to biased conclusions. been thoroughly documented 共Simon 2002兲. He established a hy-
pothesis that at least one of six “talked about” therapies, each
involving the intake of some kind of food, would actually restore
Process of Empirical Research in Science to health of diseased sailors. For his next long voyage, he loaded
the food stuff for the six treatments that he had selected to have
As mentioned earlier, the object of empirical research is to estab- available when sailors would get the disease. Fig. 1 depicts his
lish “a posteriori” knowledge, meaning scientific knowledge that plan for finding a cure.

0!ABF H54
< 0DGF

DI '5BJ >A>KA4F 5H
34567 ; 4AL5MA4A@
(D!C54F E!BJ56B (L64MG 34567 % =!@A4 KI 34567F %N:O <
=5>>5? 55@

34567 8 +!B4!5C EA4A FB!CC F!LP

(D!C54F E!BJ (L64MG 8 88 34567 9 +!?A"D4
%8 8 8 34567 : (ADEDBA4
8
34567 ; &4D?"AF
34567 < 'D4CAG
& 0)1)/0)/-
./0)1)/0)/-

&'()*+,-.&/

*)1&*-./.3 &
=&/-*&22)0

-*),-R)/-N
)Q1)*.R)/-

+,*.,'2)(
ST1&-S)(.(

+,*.,'2)

*)(U2-(
+,*.,'2)
1*&=)((
0)(.3/ &

Fig. 1. Design of the first controlled experiment to find cure for scurvy

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 27

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 -4ADB>A?BN
(ACALB!5? C54PN145LAFF 145LAFF &6B76B
&K=ALBFN(6K=ALBF

!"#$%&#'()* ,)$%)-*# ..#/( 0. !"#$%&#'()* ,)$%)-*#

U?L5?B45CCDKCA +D4!DKCAF )HHALB 5H U?L5?B45CCDKCA +D4!DKCAF

=5?B45CCA@ +D4!DKCAF )HHALB 5H =5?B45CCA@ +D4!DKCAF

% 8 9

+DC!@DB!5? 5H .?FB46>A?BF &KFA4MDB!5?FNRADF64A>A?BFN0DBD =5CCALB!5? ,?DCGF!F 5H 0DBD =5?LC6F!5?F

Fig. 2. Architecture of experimental process to establish causality

The life on a navy ship in the 18th century provided for a very the James Lind used sick sailors as subjects of his treatment pro-
controlled environment in that every sailor faced pretty much the cess while the six different food supplements served as experi-
same hardship be it weather, food, or physical work. The con- mental variables. The controlled variables consisted of the daily
trolled experiment began when he had the first 12 sailors in the food rations and the un-controlled variables were the temperature,
sickbed. Why not start the experiment earlier? The surgeon’s ob- humidity, wind, etc. The surgeon made is daily rounds and re-
jective was not only to improve the reliability of the outcome, by corded what changes he observed in the 12 sailors. It is common
making six groups of two sailors, but also creating a uniform practice to record as much as possible during the experimental
environment for all subjects. All twelve were treated equally ex- process from independent “sensory” points. The outcome of the
cept the extra food that was randomly assigned to the sailors. Of treatment were 10 sick sailors and two who had recovered. The
course, other sailors who also got sick provided the control group. effect of the experimental variable was clear in that the treatment
Should both members of a group recover from scurvy it would be with oranges led to the recovery of both members of group 5.
hard to claim that it was a coincidence. Since all 12 sailors had experienced the same uncontrollable and
As Fig. 1 indicates, each group was given a daily portion of the one controlled variable, their effect on the recovery was nil.
common food plus a supplement from the assigned “medicine,”
called the independent variable. In this experiment, observing the
Measuring Construction Work
effect of the different input variables was straight forward. After
only six days, both members of group that was receiving the Measuring the effectiveness of a new device, method, or material
oranges had recovered. Unknown to the surgeon at that time, in the construction arena is not as straight forward a task as in-
vitamin C from the oranges was and still is the key ingredient to vestigating the effect of a food on curing scurvy. The most im-
prevent and fight scurvy. portant differences are the many extraneous factors that interfere
While this case comes from outside the construction arena, its with the independent variable and the difficulties in measuring
simplicity helps one to easily identify the key characteristics of meaningful values with sufficient reliability. Pioneers in this arena
controlled experiments. The next section of the paper will now were Frederick W. Taylor as well as Frank and Lillian Gilbreth.
expand this base model to the more complex world of construc- While Taylor focused on timing work elements while the Gil-
tion. breths categorized basic work motions as ineffective or effective
leading to what is now called motion studies or predetermined
time systems that calculate work time based on needed motions.
Process Model of Scientific Experiments L. H. C. Tippett later addressed one of the key weaknesses in both
namely the large amount of time needed for the studies to be done
Modelers of construction processes or supply chains recognize by highly trained personnel. Instead of measuring the continu-
the input-output nature of a scientific experiment. The goal is to ously he recommended to randomly pick time instances to take
measure the relationships between an input and an output charac- “snapshots” of what is going on at that particular time. Today, we
teristic. Fig. 2 presents a model of the core relationships of pro- refer to it as activity, work, or productivity sampling. Despite the
cess oriented experiments. emergence of digital tools that alleviate the burden of a continu-
Again, the goal of a scientific experiment is to establish em- ous time study, work sampling is still the main workhorse in
pirical evidence of a relationship between an independent or ex- industries where the pace of the work is not controlled by robots
perimental variable and a dependent variable that is being or a conveyor. Ergonomic studies, on the other hand, have devel-
effected by it. As Fig. 2 reemphasizes, the key to this process are oped standard procedures to quantify the amount of “stress” work
observations and measurements with calibrated instruments at has on humans while quality sampling can be used to measure
Stages 1–3 to establish causalities that are cleared of extraneous output quality.
effects caused by uncontrollable variables. Also highlighted in
Fig. 2 is the fact that measurements are not restricted to a before- Work Sampling
and-after scenario but that may involve the process itself. What A good description of this method can be found in a paper by
guides the final design of an experiment is the need for data that Salim and Bernold 共1994兲. The purpose of this method is to quan-
either proves or disproves the hypothesis “a posteriori.” tify the activities during the work process and is indicated in Fig.
The process model in Fig. 2 indicates three different data- 2 as Measurement 3. A critical step in the process is to define all
collection points for: 共1兲 pre, 共2兲 In, and 共3兲 postprocess measure- work-elements that are relevant to prove the hypothesis and orga-
ments. Applied to the scurvy healing experiment one can see that nize them into productive, contributory, personal or wasteful

28 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 2. Main Experimental Approaches in Construction
Method Object of study Procedure Results
1. Pilot test of devices and methods Device A Operation 共Lab-Field兲 → Measure -Performance
Method B Implement 共Lab-Field兲 → Measure -Performance
2. Passive observation 共case study兲 Subject-group or operation A 共Events兲 → Measure -Functioning of A/B
Subject-group or operation B 共Events兲 → Measure -Performances
-Variance
3. Controlled experiment
共direct comparison兲
Subject 共group兲 A
Subject 共group兲 A/B 兵 Pre-M→ TTrad → Post-M
Pre-M→ TMod → Post-M 其 -Improvement by A
-Variance
-Covariances

兵 兵 其
其
4. Randomized experiment Random subject 共group兲 X, Strata 1 Pre-M→ TTrad → Post-M -Correlations
共conventional兲 Random subject 共group兲 Y, Strata 2 Pre-M→ TTrad → Post-M -Improvement within strata
Random subject 共group兲 X, Strata 1
兵 其
-Variances
Pre-M→ TMod → Post-M
Random subject 共group兲 Y, Strata 2 -Covariances
Pre-M→ TMod → Post-M

兵 其
5. Four group experiments Random subject 共group兲 X Pre-M→ TTrad → Post-M -Improvements of Strata 1 members
共minimized Hawthorne兲 Random subject 共group兲 Y Pre-M→ TTrad → Post-M -Variances
Random subject 共group兲 Z -Covariances
TMod → Post-M
Random subject 共group兲 G
→Observ.

groups. The number of random observations can be calculated facing challenging problems in designing valid experimental tests
and are based mainly on the confidence level 共e.g., 95%兲 and the 关National Institute of Standards and Technology 共NIST兲 2009兴.
size of the allowed errors 共e.g., 5%兲. This method is easy to learn
and use, and it provides more operational detail than historical
data. Other work measurement techniques can be used in parallel Method 1. Pilot Test of Devices and Methods
to ensure that the work sampling results are reliable. Assuring the quality of materials or the precision of building
structures 共e.g., the slope of pyramids兲 has always involved ex-
tensive measurements and standardized tests. The rapid growth of
Taxonomy of Experimental Methods the well known ASTM, established over 100 years ago, was based
for Construction on its scientific approach and the involvement of all the constitu-
ents in the standardization process. New or improved construction
Collecting a lot of evidence is necessary but not sufficient to devices or methods, however, are not covered by such standards.
discovery and the establishment of new knowledge. As the Haw- Manufacturers are generally interested to pilot new technology
thorne tests that the experiments with placebo medicines show, not only to assess its performance but also verify its ease of use,
what looks like clear relationships between independent and de- etc. Innovating new technologies in construction also depends on
pendent variables is but a “mirage.” The productivity of the coil field testing of prototypes to demonstrate technical functionality
winders did not improve because the light was changed. Execut- and opportunities for further improvements. Such pilot tests re-
ing a series of repetitive experiments within a stable environment quire a creative researcher to develop solid test procedures. The
is only possible inside strictly controlled laboratories. Experi-
most common approach is to deploy and collect data related to the
ments involving test on a construction site or observing office
performance under laboratory or field conditions.
work require a different approach. One serious difficulty is the
many uncontrollable variables that create the “unwelcome” extra-
neous side effects. Safety and scheduling concerns makes it un-
likely that a contractor is willing to allow an experimental test of Laboratory Tests
a new device or method on a real construction project. Thus,
construction researchers have been challenged to develop experi- Tests in a protected environment with access to standardized large
mental procedures within the given constraints while finding testing apparatuses provide many advantages but limit in most
ways to collect data that address their hypothesis. Table 2 ar- cases the applicability to the real world of construction. Peng
ranges the various procedures that have been used in the recent 共2002兲 published an example where observed conditions on site
past into five clusters each with some unique features. were recreated in the lab to test the load capacity of shoring
It is important to recognize that only the first method focuses systems: “The experimental test results indicate that the base
solely on the experimental performance testing of a new tool or stiffness to the ground of shores is … the critical load of the
method. Methods 2–5 on the other hand focus on individual, shoring system directly measured from the tests varies from 76 to
groups of individuals or the output of entire operations. Before 80% of the tested critical load of an individual shore …” The test
exploring in more detail, the main experimental research methods results were compared to theoretical calculations in order to pro-
that reader must take notice that various statistical methods are vide recommendation for safe designs. A different type of perfor-
available to quantify the variances of data, to assess the variability mance investigation of a new tool in construction involved
of data sets or the goodness-of-fit. Many books and documenta- assessing the survivability of various bar code labels in the con-
tions are available on the internet while the Statistics Department struction environment. For this purpose, an assortment of label
at any university generally provides assistance to faculty who are types was attached to a random selection of hand-tools and gas-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 29

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
bottles. After incremental periods, the “health” of the labels was bridges,…” 共Jaselskis et al. 2005兲. The objective of the study not
inspected. only focused on the accuracy of the laser scanner in the six dif-
ferent setting but also on establishing the time and to cost for
creating those outputs. Not surprisingly, the learning curve in de-
Scaling from Laboratory to the Field veloping efficiency in operating such complex systems was con-
sidered very costly. An excellent example of a project that focuses
The common path for new devices, after they pass laboratory on a method rather than a device tested the effectiveness of
tests, is to be brought into the “real world” to assess performance ASTM C1074 共standard practice for estimating concrete strength
under field conditions. This nontrivial step requires not only an by the maturity method兲 to predict the concrete strength in a cold
upgrading of the human-machine interfaces, in order to facilitate environment recently presented by Bagheri-Zadeh et al. 共2007兲.
operation by field personnel, but also the redesign of power sup- Eight foundations of an actual project were equipped with 29 data
plies, hardening of the protective structure 共e.g., protection
loggers measuring the development of concrete temperature while
against potential rain兲 as well as mountings for safe transportation
casting of standardized concrete cylinders provided validations in
and interface with other systems on site. One such example is the
equipment mounted utility detection system discussed by Kolera relation to established testing methods 共breaking of concrete cyl-
and Bernold 共2006兲. They reported about extensive laboratory inders兲. After a statistical treatment of the different data sets the
work to develop control and validate the performance of an inno- researchers were able to conclude that: “… the concrete maturity
vative system that integrated mechanics, geophysics, electronics, method could produce a reliable and accurate prediction of in
and computing. In a second development, the hardened system situ concrete strength on a continuous real-time basis during cur-
was attached to a backhoe loader for field tests that demonstrated ing” 共Bagheri-Zadeh et al. 2007兲.
that it: “… not only detected the presence of the pipe but aimed
correctly at the midpoint of the pipe within the space of the
trench” 共Kolera and Bernold 2006兲. A second paper that falls in Method 2. Passive Observations
this category presents the work in testing a real-time asphalt den- The descriptor, case study, used in Table 2 indicates a one-shot
sity measuring device. The writers of the paper explain that: “Sev- nature of this quasi-experiment where the researcher is does not
eral field studies have been conducted … Additionally, laboratory
administer a test but remains a passive observer of what is hap-
experiments were conducted … Core samples were also taken to
pening. This method actually dates back to Aristotle who stipu-
relate the variance of the signals to the densities” 共Jaselskis et al.
lated the need for observations as the basis for knowing. Galileo’s
2001兲. A drastic upscaling of a laboratory to integrate with a
commercial device was necessary to test an intelligent controller study and descriptions of the Lunar surface using one of the first
on a shop rebar bender. The writers describe how a manufacturer telescopes of the time, but invented in Holland, is a good ex-
was willing to loan a bender but requiring them to develop a ample. The JCEM contains several examples including a paper by
unique interface that allowed the transfer of the control model Lee et al. 共2007兲 expounding on an effort to monitor and compare
developed in the lab: “Bending tests were conducted with both a the production rates of freeway rehabilitation projects in Califor-
laboratory prototype and an actual shop table bender … Spring- nia: “A comparison of production rates was based on the as-built
back model evaluations revealed that empirical statistical models, progress data … recorded during one 55 h weekend closure …
neural networks, and in-process relaxation performed equally Examination of the as-built progress data indicated that … A
well” 共Dunston and Bernold 2000兲. Two more recent example are higher production rate and a noticeable ‘learning-curve effect’
the development and field testing of a machine vision-assisted, was observed …” 共Lee et al. 2007兲 The writers also point to the
teleoperated pavement crack sealer 共Lee et al. 2006兲 and the in- limitation of the small sample sizes to create generalizable knowl-
strumentation and field testing of a roller compactor to monitor edge. Nevertheless, as another example demonstrates the collec-
vibration behavior during earthwork compaction 共Rinehart and tion of useful data may require an extensive effort. Here, the
Mooney 2008兲. quality of asphalt overlay application during day and night cycles
on a 8.85 km 共5.5 miles兲 stretch of urban highway was assessed.
Surface smoothness, in-place density, gradation, and asphalt con-
Many Demands of Field Tests tent were measured during two weekends of work by one con-
tractor for comparison with published averages. “Production rates
While still only having to deal with nonhuman test objects, or- were compared to those from a comparably sized nighttime
chestrating field experiments requires many human skills from the project. The investigation revealed that.… High paving produc-
director or manager including coordination, communication, and
tion rates resulted from … with respect to these parameters was
flexibility besides a thorough understanding of the technology to
decidedly better than average. Average shift production rates for
be tested and the construction process itself. How much may be
the weekend closure were higher than those documented for a
involved in testing a new device in the field was verified by
Jaselskis et al. 共2005兲 who explained the procedure and the out- similar nighttime paving project.” 共Dunston et al. 2000兲
come of testing the performance of a laser scanning device the While both cases present extensive amount of sampled data,
Cyra 2500 Laser Scanning Unit. The system was deployed in six mainly to measure work results, even employing standard instru-
different field situations to collect raw data while traditional sur- ments 共e.g., California Profilograph兲, it is impossible to reliably
veying approaches were used to create benchmarks for accuracy. draw cause-effect conclusions that are transferable. In cases
The preparatory work not only included hands-on training of the where the observed company is aware of the study arranged by
study team in the use of the hardware and software before it was the owner the surveyed laborers, operators, crew foremen, super-
sent to perform the entire process on its own: “… there were six intendents, as well off-site personnel most probably changed their
test areas involved in this pilot study: (1) An intersection includ- normal behavior much like the factory workers at Hawthorne
ing a railroad bridge, (2) a section of highway including a pair of Works in the 1930s.

30 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 +D4!DKCAF =5>>5?
U?L5?B45CCDKCA +D4!DKCAF effectiveness of different approaches to rebar placement for a
B5 '5BJ -AFBF
*A7ADBA@ (BDB!FB!LDC
multistory building 共Salim and Bernold 1994兲 and the laying of
&KFA4MDB!5?F 5H , *AF6CB concrete pipes with traditional and advanced technologies 共Lee et

=5>7D4!F5?
%I )?M!45?>A?B
05?A -4D@!B!5?DC 5H ,
8I RD?D"A>A?B
9I 2DK54<FP!CC<>5B
al. 2003兲. In the latter case, a utility contractor was willing to
:I 25"!FB!LF *A7ADBA@ (BDB!FB!LDC dedicate one and the same crew to lay nine pipe-sections tradi-
;I 0AF!"? &KFA4MDB!5?F 5H , *AF6CB 5H ,
<I =5>>6?!LDB!5? 05?A E!BJ Q E!BJ Q
tionally on one day and the same pipes using the new technology
during the next. Furthermore, these two operations took place in
the same field and under the same weather conditions. During
,FFAFF>A?B both operations, each repetitive task was timed allowing the cal-
5H =D6FDC
*ACDB!5?FJ!7F culation of mean task as well as cycle times. One cycle consisted
of 共1兲 excavation for one section; 共2兲 lowering of one pipe; 共3兲
jointing of new piece; and 共4兲 initial backfill. Because of the nine
Fig. 3. Design for comparative field experiment with minimal impact repetitions, the learning effect could be easily detected. Following
of extraneous variables
the design in Fig. 3 the operation was continuously observed in
order to measure the effect of uncontrollable variables on the
productivity. For example, during the experiment, an electric
Method 3. Controlled Experiment winch broke and had to be replaced with a chain and come-along
As we learned earlier, this method was first used by the British system drastically lengthen the time of rigging. Safety and quality
Royal Navy surgeon James Lind in 1747 共see Fig. 1兲. The differ- were constantly monitored as well. Due to the inherent simplicity
ence to Method 2 is the purposeful setup of a comparative test of the system, the crew showed an extremely steep learning
including the collection of relevant data before, during and after curve. Still, the Hawthorne effect could not be measured but as-
the ongoing experiment. Reviewing the JCEM one quickly real- sumed equally strong during both tests.
izes that this approach is heavily used albeit with several varia-
tions.
Multifactor Comparative Evaluation

Comparative Evaluation Human factor studies have shown that the performance of humans
may be effected by a range of causes some counteracting and
A hypothesis may state: “Procedure X will increase the produc- some synergistically supporting each other. Such “hidden” factor
tivity of construction process A by 30%.” The most common ap- combinations may result in such large variances when correlating
proach to prove or disprove such a hypothesis is a controlled cause-and-effect variables, that no sound conclusions can be
experiment to compare the productivity of process A with and drawn. Two experiments where this phenomenon surfaced in-
without with X. While this might sound simple, the difficulty lies volved the productivity assessment of a sensor-equipped nailgun
in the design of experiments that produce data that create clear 共Miller and Bernold 1991兲 and the speed and safety of different
and “clean” causal relationships between procedure X and the manipulator control concepts for bridge paint-removal 共Moon and
productivity of A. Since most construction takes place within an Bernold 1998兲. Fig. 4 depicts three graphs with hypothetical data
uncontrollable environment many extraneous influences may to demonstrate in a simplified manner the underlying principles
“sneak in through backdoors” to create misleading correlations. A and the merits of considering a multi-factor approach when ana-
recent paper by Shin and Dunston 共2009兲 shows how researchers lyzing experimental data.
reduced the complexity of the construction site by designing ex- Fig. 4共a兲 plots the average productivity of seven volunteers
periments that could be executed inside a controlled space. They testing a new device. A point represents the mean value for 10
correctly state this limitation in the conclusions and point to the observations, once working with the traditional and then with the
need for further work on site. new device. As shown, the surfacing improvements show large
In some situations, this risk may ask for a preexperiment to changes and range from 1.5 to 16 units per Person-hr. No linear
assert that all influential factor variables are being either con- relationships are apparent rather some small clusters. Is there a
trolled or their effect captured. The model in Fig. 3 is based on hidden factor that has extremely strong relationships with produc-
the assumption that all cause-effect causing factors are known, tivity? If data for all potential impact variables has been collected
captured and assessed. an ANOVA will find the strongest relationships and even the ef-
Fig. 3 highlights a strategy that does not eliminate uncontrol- fect of covariances. The critical nature of having “all” the neces-
lable variables but minimizes their conflicting impacts. As shown, sary data after all the experiments are completed reemphasizes the
the comparative experiments are executed within the same envi- importance of preexperiments.
ronment, management, labor, etc. The underpinning for this de-
sign is the expectation that hidden factors impact both
Preexperiment
experiments in the same manner, thus cancel each other out. It is
apparent, that this cannot necessary be true in all cases. For ex- Fig. 4共b兲 depicts graphically the result of one preexperiment that
ample learning skills, motivation or changes in health of persons had the objective to test if and how the years of experience effect
involved in comparative experiments can have drastic impacts on productivity. Twenty-two laborers with different years of experi-
the measurements. ence were separately observed nailing the plywood for the roofs
Four examples where this approach has been used involved the of single-family houses with the traditional nailgun. The only
comparison of automated with human control of a dredging op- experimental variable was the years of experience working with
eration 共Tang et al. 2009兲, the pollutant level and energy effi- nailguns. The hypothetical data is shown in Fig. 4共b兲 shows that
ciency of an intelligent controller for a tunnel ventilation system there exist three definite clusters or strata demarcated by 3.5 and
as compared to a conventional controller 共Chu et al. 2008兲, 6.5 years of experience. The importance of this data are not only

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 31

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 50 !"#!# ' !"#!# % !"#!# &
45 RAD? * :8$<

,MA4D"A U?!BF 7A4 1A4F5?NJ4

,MA4D"A U?!BF 7A4 1A4F5?NJ4

40 (-0 * 8$% 45 RAD? * :8$5
35 (-0 *42.7
6$,
RAD? * 99$5 40
30 (-0 * 5$8 RAD? * 9<$<
36.5
(-0 * %$9
35
25
20 30
15
RAD? * +$, 25
10 RAD? * 89$5
23.6
(-0 * ;$: (-0 * %$,
5 20
0 15
0 2 4 6 8 0 2 4 6 8 10
(a) +5C6?BAA4 (b)
TAD4F 5H )=7A4!A?LA
Traditional Tool New Tool Improvements Productivity

50 !"#!# ' !"#!# % !"#!# &

,MA4D"A U?!BF 7A4 1A4F5?NJ4

45
40 RAD? * :8$6
35
RAD? * 9:$;
30
25 RAD? * 8:$;
20
15
10
5
0
0 2 4 6 8 10
TAD4F 5H )=7A4!A?LA
(c) Traditional New Tool Improvements

Fig. 4. Experimental results with and without stratification 共a兲 results of initial experiments; 共b兲 identification of strata within labor-pool; 共c兲
initial experimental results stratified

in proving a correlation between experience and productivity but mial function with scale invariance of the form T共x兲 = T1 ⴱ x−␣. T1
more importantly, this information will be instrumental in design- represents the duration of the first cycle and ␣ the learning curve
ing the actual experiment. exponent. Task duration T共x兲 reduces for each repetitive cycle x
along the curve defined by x−␣ until it reaches a level that is
Experiment with Stratified Samples limited by extraneous factors. This curve is further bound by the
maximum number of cycles it will take to reach the minimum
In order to create generalizable results, conducting experiments time. Experience shows that learning curves reach from 90% for
with a representative sample of laborers is important. The preex- repetitive welding or 75% for more complex tasks, meaning that a
periment showed the existence of three experience groups show- cycle time will drop by 75%.
ing different productivities. It is apparent that the samples of
laborers to be invited to the comparative experiment should in-
clude representatives from each strata. In fact, the number of Method 4. Randomized Experiment
laborers in each strata should reflect the proportion of the strata in
Proving the hypothesis that: “Total Stations improve the produc-
the overall population. Fig. 4共c兲 plots the data from Fig. 4共a兲 as a
tivity of U.S. motor grader operators by 28%” will require experi-
function of experience with the oldest strata being represented by
three subjects. It can be observed that the average performances ments with a large number of operators, using many different
of the small samples are all close to the strata mean values in Fig. grader models, working under different conditions and regions of
4共b兲 although no subject participated in both experiments. It is the United States. As it is impossible to test every single operator
now immediately apparent that those with least experience benefit a sufficiently representative sample has to be selected. How big
most from the new device. This important conclusion gives im- that representative sample has to be depends on the required pre-
mediately raise to a new question: How quickly will the in- cision and the variability in the entire population. Various formals
experienced learn to handle the new device most effectively? have been proposed that allow us to calculate. For the sake of this
demonstration, expecting a confidence level of 95%, we shall use
the following simplified formula: Sample size n = N / 关1 + N共e兲2兴
Learning Effect where N = total population and e = acceptable error. If we assume
As mentioned earlier, in experiments that involve human exper- that there are 3,500 active grader operators in the United States
tise to coordinate manual operations one has to be aware of the and we consider the acceptable sampling error as 5% the cal-
learning effect. Learning effects the cycle time of an operation not culation of sample size looks as follows: n = 3 , 500/ 关1
dictated by the uniform pace of a machine because repeated ex- + 3 , 500共0.05兲2兴 = 359 operators. Reducing the confidence level to
ecution of that same motion leads to better hand-eye coordination, 10% would still require 98 operators, a large number considering
less experimentation, and time consuming mistakes. Based on the fact that each would have to be tested in various environ-
empirical data, the general form of the learning follows a polyno- ments. Stratification of the total population might help but would

32 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 /= (-,-) =)N8%: 1*)N-)(- )Q1)*.R)/-,2 1&(-N-)(- (-,-.(-.=,2
)/3./))*./3 (-,-.=( R),(U*)N -*),-)R)/-( R),(U*)N 0,-,
(&1SR&*)( E(ACH (ACALBI R)/-( E=5>>5? -ADLJA4I R)/- ,/,2T(.(

+DC!@DBA@ (64MAGF
(ALB!5? % (ALB!5? 8

=5?H!@A?LA (64M$
=544ACDB!5?F

(7AL!DCN(64MAGF

=5>>5? !?DCF
+DC!@DBA@ -AFBF
!"#$!% '$!() *+,-

=564FA 34D@A
!"#$% (ALB!5? 8 .$/01#1!"/% %/22 RAD?F
'( )*+" (BD?@D4@ 0AM$

31*O (,-

*ABA?B!5?
,+-./(/0+ (ALB!5? 9 (ALB!5? 9 +D4!D?LAF
3/"0245" *66- =5N+D4!D?LAF
(ALB!5? : 7$!89:# ;/290

.*' *AM!AEO (!"?!?" 5H

.?H54>A@ =5?FA?B 54>

Fig. 5. Design of an educational experiment with control group and self-selection

require the availability of statistical data about the different strata Informed Consent
共see previous section兲. This demonstration of sample size calcu-
lation should be sufficient in demonstrating that such projects According to the Code of Federal Regulations Title 45 Part 46,
would require funding much higher than is common in construc- Protection of Human Subjects, every institution that is involved in
tion engineering and management. On the other hand, selecting funded research is required to operate an office that reviews, ap-
operators working for one company only might skew the results proves, and documents that every project is following approved
that they may show similar and unique traits having been hired procedure regulations. At NCSU, this office was called Institu-
based on specific company criteria. The same holds true for lim- tional Review Board or IRB for short. In cases where research is
iting the sampling to one town, state or region of the United “conducted in established or commonly accepted educational set-
States. The only solution to this dilemma is to limit the hypothesis tings, involving normal educational practices … 共Code of Federal
to cover a population that can be sampled in a sufficient manner. Regulations 2009兲” Researchers will be exempt from continued
It goes without saying that the conclusion will only apply to the oversight by the IRB but will have to design an informed consent
targeted population. form that explains the goals, risks, and the context of the involve-
ment to be signed by both the student and the research director
共see Appendix I兲 The main hypothesis of the educational project
underlying Fig. 5 claimed that: “Hands-on project based teaching
Self-Select within a Limited Population
of statics will improve the grades of students who depend on
A more feasible alternative is to sample a smaller population or concrete experiences to understand and learn.” Thus, the objective
even a strata such as students attending a given university or a of the experiment was to establish strong correlations between the
program. For example North Carolina State University 共NCSU兲 learning preferences of students, 55 in the experimental and 70 in
has approximately 1,000 engineering freshmen every year. Apply- the control group, the method of teaching and the course grade.
ing the same formula as above would result in a still large sample Using a validated survey, the students were stratified into four
size n = 1 , 000/ 关1 + 1 , 000共0.05兲2兴 = 285. Instead of sampling from clusters of students who had similar learning preferences. It was
the entire population, considered homogenous, we might want to interesting to realize that the self-selection process produced dis-
stratify by matriculation and pick randomly from each cluster so tributions in each section that were close to the overall population
that each cluster is equally well represented in the final sample. at NCSU 共10, 22, 55, and 13%兲, taught by the same professor the
While such a procedure might be desirable, the reality of a uni- students faced similar tests but different methods of teaching and
versity prohibits that students are being “coerced” to participate in learning. The stratified comparison of the grades from both stu-
experiments that might influence their grades and with it their dent cohorts confirmed the hypothesis only partially. While one of
future. One way to circumvent this problem is through self- the two subgroups needing concrete experiences did in average
selection and student consent. Bernold et al. 共2000兲 presented the gain significantly higher course grades, did the second subgroup
result of such a study in engineering education where students still equally did poorly in both sections? For the results of ana-
self-selected into two sections of statics taught by the same pro- lyzing additional data, the reader is referred to subsequent papers
fessor at NCSU. in JCEM 共Bernold 2005, 2007兲.
Fig. 5 reveals that only a subset of all engineering sophomore
needed to take CE-214 共Engineering Mechanics-Statics兲 as a pre- Method 5. Four Group Experiments—Minimized
requisite for many other classes. While the students’ self-selection Hawthorne
from several parallel sections cannot claim to represent a perfect
randomization process one may be able to verify that it is suffi- This is considered the most powerful method to study the effect
cient for a particular experiment. It goes without saying that en- of any intervention on human performance because it considers
gineering students who become sophomores at NCSU do also not the previously discussed Hawthorne effect. As shown in Table 2,
represent a randomized sample of all engineering students in the four subjects or groups are randomly picked but only two of them
United States. Still, by comparing the representation of critical are subjected to pretest observations. The other two groups are
factors within the population 共e.g., female versus male兲 with the left without knowledge that they are being observed one using the
engineering at other universities may give sufficient indications traditional and the modified method. This design provides the
about the soundness of the sample. Fig. 5 involves a very impor- advantage that the consequence of knowing that one is part of the
tant topic that effects every research that involves people, namely experiment, due to the pretest, can be detected. However, this
protecting the safety of human subjects. design requires large samples while the federal requirement to get

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 33

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
the consent from anybody participating in the research project questions about your progress in class. The electronic journaling
will add an additional burden for the research team. Hence, it is will take approximately 15 min per week. Some of you will be
not surprising that the JCEM does not contain examples of four invited to group interviews at the end of each semester.
group experimental designs. 2. ..
BENEFITS
Considered a first in the nation, this study will provide impor-
Summary tant information about students’ learning preferences, how engi-
neering students study, and how it impacts their success and
Finding economic means to improving the quality, safety, and confidence.
productivity of construction or reducing waste and pollution re- CONFIDENTIALITY
quires the development of a posteriori knowledge about the effect The information in the study records will be kept strictly con-
of various factors. However, coming up with measurable evidence fidential. For example, your journal entries will be coded elec-
needed to support a hypothesis or theory is extremely hard to tronically. When data are given to us by the course instructors, it
come by in the construction industry. Furthermore, the large size will be stripped of identifying information. Data will be stored
of the industry puts extreme requirements on establishing re- securely and made available only to persons conducting the study
search findings that are applicable to the entire or even a section unless you specifically give permission in writing to do otherwise.
of the industry. The inherent problem is the cost involved in ob- No reference will be made in oral or written reports that will
serving sample sizes big enough to produce data points that are identify you as an individual.
statistically significant. PARTICIPATION
As the paper shows, published research results demonstrate Your participation in this study is voluntary; you may decline
that it is still possible to perform useful experimental projects in to participate without penalty. If you decide to participate, you
construction. The key to a successful project is the establishment may withdraw from the study at any time without penalty and
of a hypothesis that fits the available data and the data acquisition without loss of benefits to which you are otherwise entitled. If
capabilities. Likewise, a careful hypothesis will lead to a robust you withdraw from the study before data collection is completed
design of the experiment and consequential conclusions. your data will be returned to you or destroyed. If you decline to
The majority of experimental projects discussed in the JCEM participate or withdraw from this study you are still expected to
are pilot tests of devices or methods. It was discussed how con- submit the assigned journals to your instructor for you to earn
trolled experiments that compare the performance of one device your final grade, but your data will not be forwarded to us for
or method against another require extensive planning and data analysis.
collection procedures to ensure that extraneous effects are held at CONTACT
a minimal level or are at least being documented and considered If you have questions at any time about the study or the pro-
in the final analysis. The existence and the size of multiple impact cedures, you may contact Dr. Leonhard E. Bernold, at .. If you
factors is best established by performing preexperiments. Such feel you have not been treated according to the descriptions in this
preliminary results not only provide evidence if multiple factors form, or your rights as a participant in research have been vio-
exist they help to improve the design of the final experiment using lated during the course of this project, you may contact Dr. …
stratified sampling. Examples were used to demonstrate how such Chair of the NCSU IRB for the Use of Human Subjects in Re-
an approach could ensure the representative presence of every search Committee, Box.., NCSU Campus … or Mr. …, Ass. Vice
cluster existing within a larger population. Chancellor, Research Admin., Box .., NCSU Campus…
Experimental research in construction in most cases involves CONSENT
human subjects. Many unexpected things can and do happen de- I have read and understand the above information. I have
spite the best preparations. A researcher working with humans received a copy of this form.
within a laboratory or in the field should, in his own best interest
and in the interest of each participant, acquire the guidance and I AGREE TO PARTICIPATE I CHOOSE NOT TO PARTICIPATE
the approval of the IRB. This step will not only prevent accidents Subject’s Signature: Subject’s Signature:
but protect the researchers from the unconscious misuse of per-
sonal information.
References

Appendix I. Informed Consent Form Bagheri-Zadeh, S. H., Kim, H. K., Hounsell, S., Wood, C. R., Soleymani,
H., and King, M. 共2007兲. “Field study of concrete maturity method-
Research Study: Study Skills and Success of Engineering ology in cold weather.” J. Constr. Eng. Manage., 133共11兲, 827–835.
Students Bernold, L. E. 共2005兲. “A paradigm shift in education is vital for the
Principal Investigator: Dr. Leonhard E. Bernold future of our profession.” J. Constr. Eng. Manage., 131共5兲, 535–542.
You are invited to participate in a research project to better Bernold, L. E. 共2007兲. “Teaching evaluations for construction engineering
understand the interrelationships between learning strategies and and management: Opportunity to move us forward.” J. Constr. Eng.
academic success of students in the College of Engineering. This Manage., 133共2兲, 146–156.
Bernold, L. E., Bingham, W.L., McDonald, P.H. and Attia, T.M. 共2000兲.
information will be used to develop new educational material that
“Influence of learning type oriented teaching on academic success of
will designed to enable students to reach their academic and pro- engineering students.” J. Eng. Educ., ASEE, April, 191–199.
fessional goals…. Chu, B., et al. 共2008兲. “GA-based fuzzy controller design for tunnel ven-
BACKGROUND INFORMATION tilation systems.” Autom. Constr., 17共2兲, 130–136.
1. As part of your classwork for E101: Intro to Engineering Clarke, D. 共2006兲. Descartes: A biography, Cambridge University, Cam-
and Problem Solving, you will be asked to complete three stan- bridge, U.K.
dard survey forms and keep an electronic journal answering short Code of Federal Regulations. 共2009兲. “Title 45-Public Welfare.” Dept.

34 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 of Health and Human Services, Part 46, Protection of Human crack sealer.” Autom. Constr., 15共5兲, 616–626.
Subjects, 具http://www.hhs.gov/ohrp/humansubjects/guidance/ Miller, M. L., and Bernold, L. E. 共1991兲. “Sensor-integrated nailing for
45cfr46.htm#46.101典共Jan 12, 2009兲. building construction.” J. Constr. Eng. Manage., 117共2兲, 213–225.
Dunston, P. S., and Bernold, L. E. 共2000兲. “Adaptive control for safe and Moon, S., and Bernold, L. E. 共1998兲. “Graphic-based human-machine
quality rebar fabrication.” J. Constr. Eng. Manage., 126共2兲, 122–129. interface for construction manipulator control.” J. Constr. Eng. Man-
Dunston, P. S., Savage, B. M., and Mannering, F. L. 共2000兲. “Weekend age., 124共4兲, 305–311.
closure for construction of asphalt overlay on urban highway.” J. National Institute of Standards and Technology 共NIST兲. 共2009兲. “e-
Constr. Eng. Manage., 126共4兲, 313–319. Handbook of statistical methods.” 具http://www.itl.nist.gov/div898/
Jaselskis, E. J., Gao, A., and Walters, R. C. 共2005兲. “Improving transpor- handbook/index.htm典 共Jan 12, 2009兲.
tation projects using laser scanning.” J. Constr. Eng. Manage., Peng, J. L. 共2002兲. “Stability analyses and design recommendations for
131共3兲, 377–384. practical shoring systems during construction.” J. Constr. Eng. Man-
Jaselskis, E. J., Han, H. C., Grigas, J., Tan, L., and Fahrion, D. 共2001兲. age., 128共6兲, 536–544.
“Status of roller mountable microwave asphalt pavement.” J. Constr. Rinehart, R.V., and Mooney, M. A. 共2008兲. “Instrumentation of a roller
Eng. Manage., 127共1兲, 1–46. compactor to monitor vibration behavior during earthwork compac-
Kolera, B. T., and Bernold, L. E. 共2006兲. “Intelligent utility locating tool tion.” Autom. Constr., 17共2兲, 144–150.
for excavators.” J. Constr. Eng. Manage., 132共9兲, 919–927. Salim, Md., and Bernold, L. E. 共1994兲. “Effects of design-integrated pro-
Lee, E. B., Lee, H., and Ibbs, C. W. 共2007兲. “Productivity aspects of cess planning on productivity in rebar placement.” J. Constr. Eng.
urban freeway rehabilitation with accelerated construction.” J. Constr. Manage., 120共4兲, 720–737.
Eng. Manage., 133共10兲, 798–806. Shin, D. H., and Dunston, P. S. 共2009兲. “Evaluation of augmented reality
Lee, J., Lorenc, S. J., and Bernold, L. E. 共2003兲. “A comparative perfor- in steel column inspection.” Autom. Constr., 18共1兲, 118–129.
mance evaluation of tele-operated pipe laying.” J. Constr. Eng. Man- Simon, H. B. 共2002兲. The Harvard medical school guide to men’s health,
age., 129共1兲, 32–40. Free Press, New York, 31.
Lee, J. H., Yoo, H. S., Kim, Y. S., Lee, J. B., and Cho, M. Y. 共2006兲. “The Tang, J., Wang, Q., and Zhong, T. 共2009兲. “Automatic monitoring and
development of a machine vision-assisted, teleoperated pavement control of cutter suction dredger.” Autom. Constr., 18共1兲, 194–203.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 35

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Empirical Modeling Methodologies for Construction
Ian Flood, M.ASCE1; and Raja R. A. Issa, M.ASCE2

Abstract: The paper provides a review of empirical modeling and its application within construction engineering and management. The
scope of application and trends in use of this approach are first assessed, and the potential for its further development is identified. This
is followed by an examination of the key components of empirical modeling, namely: the structure and operation of the model and the
scheme used in its development. The paper then provides a rigorous methodology that must be followed to ensure the validity and value
of the end model, covering the steps: strategizing; data collation and assessment; model development; model evaluation and final
selection; final validation; and implementation. The methodology is designed to cater for all forms of empirical modeling including the
procedurally more demanding development algorithms that have become available in recent years, such as simulated evolution. Overall,
the paper is designed to provide researchers embarking on an empirical modeling study with an overview of when it is appropriate to use
this approach, what type of system to adopt, and how to ensure development of a successful end product.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000138
CE Database subject headings: Empirical equations; Dynamic models; Algorithms; Data collection; Data analysis; Validation;
Construction management.
Author keywords: Empirical modeling; Structured model; Static model; Dynamic model; Development algorithm; Data collection;
Data analysis; Model validation.

Introduction their output, and have no ability to extrapolate beyond the set of
observations used in their development.
Mathematical models are abstractions of systems described using The above characterization of empirical and theoretical mod-
the language of any branch of mathematics including, for ex- eling, while widely accepted, is an oversimplification of the truth.
ample, algebra, statistics, logic, and algorithms. They are devel- Ultimately, all mathematical models 共theoretical or otherwise兲 are
derived empirically and, contrary to popular belief, empirical
oped to serve either as experimental tools that can be used to
models can play a key role in developing a deeper and more
extend our understanding of a system or as predictive tools to
generalized understanding of a system. To illustrate these points,
assist in tasks such as decision-making and automated systems consider the problem of modeling the dissipation of heat in a
control. Certainly they have become significantly more powerful large concrete structure during curing. At the heart of this model
and accessible with the advent of desktop computing, finding im- would most likely be the following heat equation describing how
portant applications in almost all branches of science and tech- the temperature of a solid material changes with time
nology. Mathematical models can be classified in many ways,
though a common dichotomy relevant to this paper is that of ⳵ T/⳵ t = k · 共⳵2T/⳵ x2 + ⳵2T/⳵ y 2 + ⳵2T/⳵ z2兲 共1兲
empirically versus theoretically derived models. An empirical
model is one developed from observations of the response of the where T = temperature; t = time; k = thermal diffusivity of the con-
system under investigation 共or of an analog of that system兲 for a crete; and x, y, and z = spatial dimensions.
This would be considered to be the underlying theory from
range of situations. In contrast, a theoretical model is one devel-
which the heat dissipation model would be built; yet it can be
oped from what are considered to be the fundamental laws or
shown that ultimately every aspect of this model will have been
principles that govern the response of the system. From these
derived using empirical techniques. First, the broader form of the
definitions, it is often reasoned that empirical models provide equation 共which states that the rate of change of temperature over
compromised solutions to problems; they are relatively easy to time is directly proportional to the second derivative of the tem-
develop but are black box devices providing no explanation of perature distribution across space兲 must have been inferred from
experience of how heated objects behave. The use of experience
1
Associate Professor, Rinker School, College of Design, Construction as such to develop a model is an empirical procedure, requiring
and Planning, Univ. of Florida, Gainesville, FL 32611-5703 共correspond- reference to observations of the response of the system both for
ing author兲. E-mail: [email protected] inspiration in the design of the model, and for evolving that de-
2
Professor, Rinker School, College of Design, Construction and Plan- sign into a more accurate or complete definition of reality. Indeed,
ning, Univ. of Florida, Gainesville, FL 32611-5703. E-mail: raymond- developing a set of general principles about a system 共a theory兲
[email protected]
based on experience or observation is a branch of artificial intel-
Note. This manuscript was submitted on August 28, 2008; approved
on March 27, 2009; published online on August 20, 2009. Discussion ligence 共AI兲 that has significant potential for application in con-
period open until June 1, 2010; separate discussions must be submitted struction engineering and management 共CEM兲, and is a concept
for individual papers. This paper is part of the Journal of Construction that will be returned to later.
Engineering and Management, Vol. 136, No. 1, January 1, 2010. Next, the parameter k in the equation 共the thermal diffusivity
©ASCE, ISSN 0733-9364/2010/1-36–48/$25.00. of the concrete兲 is a constant that will have been determined

36 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
directly from laboratory experiments or derived from other mate- regression NOT neural neural NOT regression regression AND neural

rial constants 共the thermal conductivity, specific heat capacity, and 14

density of the concrete兲 that will themselves have been deter-

Occurrences in JCEM
12
mined from laboratory experiments. A constant derived from 10

laboratory experimentation is by definition an empirical model, 8

6
albeit the simplest form possible.
4
Finally, the complete model representing heat dissipation 2
within the concrete structure would require Eq. 共1兲 to be dis- 0
cretized in time and space, most likely using the finite-difference 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
Year of Publication
method. The discretized form of this equation would be set within
a spatial grid defining the shape and size of the concrete structure,
Fig. 1. Number of articles published in JCEM that consider the most
which would then be used to simulate the heat dissipation pro-
popular types of empirical model
cess. Importantly, the accuracy of the model’s predictions will
depend on, among other things, the distance between the discrete
elements within the grid. Appropriate distances would usually be cover almost every aspect of CEM. A small yet diverse sample of
determined by trial-and-error 共again an empirical modeling tech- some more recent applications includes: forecasting labor demand
nique兲 in which the proximity of the discrete elements would be 共Wong et al. 2008兲; predicting the performance of project manag-
varied until the behavior of the model is sufficiently close to that ers 共Ahadzie et al. 2008兲; predicting profit performance of
observed in the real system. Alternatively, the distances could be projects 共Han et al. 2007兲; studying dispute negotiation styles
based on rules-of-thumb; nevertheless, these rules-of-thumb 共Cheung et al. 2006兲; predicting construction costs 共Lowe et al.
would have been derived at some earlier point in time by experi- 2006兲; estimating productivity 共Ezeldin and Sharara 2006兲; as-
menting with similar models. sessing project safety 共Fang et al. 2006兲; estimating operation
Clearly empirical modeling has a broad potential, being at the durations 共Marzouk and Moselhi 2004兲; and assessing risk in bid-
heart of just about all forms of mathematical modeling. The ob- ding for projects 共Fang et al. 2004兲.
jective of this paper is to identify the extent to which this poten- Although there are many examples of the application of em-
tial has been realized within the field of CEM and to identify the pirical modeling to CEM, the majority of these consider very
most promising areas for future development and application. simple modeling systems that develop continuous functions that
This includes an evaluation of the application of the technique as map directly from a vector of inputs to a vector of outputs. The
reported within the ASCE Journal of Construction Engineering most common and simplest of these is regression analysis, which
and Management 共JCEM兲 and related journals, classifying the provides either a linear or simple nonlinear mapping between the
methods that have been considered, and identifying trends in their inputs and outputs, as illustrated in Fig. 2共a兲. More sophisticated
adoption and usage. Despite their diversity, the development of varieties of direct-mapping models include: Fourier analysis
any empirical model must follow the same basic procedures to which produces a compound of periodic functions that map from
ensure the production of a valid and useful end product. More- the inputs to the outputs, as illustrated in Fig. 2共b兲; and Feed
over, recent advances in model development techniques have forward Artificial Neural Networks which develop multiple layers
made this procedure more tortuous. This paper, therefore, pro- of compound functions, of any form, to map from the inputs to
vides a comprehensive review of the empirical modeling method- the outputs, as illustrated in Fig. 2共c兲. Example applications of
ology that recognizes the extended demands of these new tools. these various forms of direct-mapping model in CEM include:
early-design cost estimation using multivariate regression analy-
sis 共Stoy et al. 2008兲; determining truck loads from bridge strain
Application in CEM readings using Fast Fourier Transforms 共Gagarine 1991兲; and pre-
diction of project success using artificial neural networks 共Ko and
Empirical modeling is clearly one of the most widely used ana- Cheng 2007兲. Beyond this, there has been a growing interest in
lytical tools in CEM. For example, in the 17-year-period from simulated evolutionary methods, such as genetic algorithms, for
1991 to 2007, a total of 97 articles have been published in JCEM model development 关see for example Salem et al. 共2007兲兴, al-
that consider the two most widely used forms of empirical model: though the potential of this method has not been fully tapped
regression analysis, and artificial neural networks 共this was based having only been used to develop relatively simple model struc-
on a search for articles with the terms “neural” and/or “regres- tures. Another area that has attracted some interest is the devel-
sion” within their titles, keywords, and abstracts兲. This is compa- opment of empirical models that operate dynamically, providing
rable in popularity to simulation modeling, for example, which is insight into the time dependent behavior of a system. Examples
considered in 99 JCEM articles according to a similar search include simulating construction excavation processes 共Flood and
using the terms “simulation” AND “model.” Moreover, the last Christophilos 1996兲, and modeling dynamic heat-flow through
couple of decades have seen an increasing trend in the use of buildings 共Flood et al. 2004兲.
these most commonly used forms of empirical modeling, as is
evident from the distribution in Fig. 1. Indeed, the number of
articles that consider regression analysis and/or artificial neural Modeling Systems
networks has increased from an average of 1.8/year for the first
five years covered by the chart to 9.8/year for the past 5 years, a Empirical methods can be used to develop models that are far
5.4 factor increase. This can be explained in part by an increase in more sophisticated than the direct-mapping devices discussed
the number of articles published each year in JCEM, although this above, significantly extending the scope of application of the
has only doubled indicating that the proportion of published ar- technique, although to date this potential remains largely unex-
ticles dedicated to empirical modeling has also increased. plored. In fact, just about any type of mathematical model can be
The applications of empirical modeling reported in JCEM developed empirically. While a complete discussion of all model

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 37

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 output output output

input input input

output
output
add constant Σ sum
output
Σ sum apply weights
apply weight &
apply weights
add constant sum then apply
apply simple activation function
function apply
apply weights
periodic functions

input apply
activation function
apply weights
input

(a) Regression Model (b) Fourier Series (c) Artificial Neural Network

Fig. 2. Examples of direct mapping models 共a兲 regression model; 共b兲 Fourier series; and 共c兲 artificial neural network

types is beyond the scope of this paper, there are two broad issues the model developer. Careful consideration should be given to the
that must always be considered in an empirical study and thus design of these variables since they will have a profound impact
warrant discussion. These are: 共1兲 the structure of the model on both the value and accuracy of the model.
共which can be further broken down into the interface and internal A primary decision to be made when designing the structure of
structure兲; and 共2兲 the scheme used to develop the model. These the model’s interface is whether it will run statically 共generating a
broad issues, and their relevance to the success of an empirical single set of outputs in response to a set of inputs兲 or dynamically
modeling study, are discussed in the following subsections. 共producing a stream of values at the outputs兲. Almost all empirical
modeling studies adopt the static approach, due to its simplicity.
Interface Structure However, dynamic models provide additional information on the
A model’s interface comprises input variables which are used to behavior of a system, and greater modeling flexibility. The issues
define the specific instance of the problem being investigated and relating to this choice will be illustrated by reference to Fig. 3
output variables which register the models response to the values which shows the interfaces for a static and dynamic version of a
presented at the inputs. Depending on the modeling system being model of an excavation system comprising one excavator loading
used, these variables may be determined by default 共such as is the a fleet of dump-trucks. The static version of this model 关Fig. 3共a兲兴
case for many simulation modeling systems such as CYCLONE provides a single estimate of the production rate of the system,
共Halpin and Woodhead 1976兲 or they may have to be designed by averaged over an indefinite period of time, in response to a set of

INPUTS OUTPUTS INPUTS OUTPUTS

Truck type Mean production Truck type Production rate

Truck engine power rate Truck engine power (stream of values)
MODEL

Truck haul capacity Truck haul capacity

Excavator Excavator
Number of trucks Number of trucks
utilization factor utilization factor
Haul distance Haul distance
(stream of values)
MODEL

Haul road condition Mean truck queue Haul road condition

Excvtr. bucket size length Excvtr. bucket size Truck queue
Excavator utilization t-15 length
(stream of values
Excavator utilization t-10
with feedback)
Excavator utilization t-5
Queue length t-15
Queue length t-10
Queue length t-5

(a) Static Model (b) Dynamic Model

Fig. 3. Interface structure for alternative models of an excavator and truck based excavation systems 共a兲 static model; 共b兲 dynamic model

38 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
inputs ranging from “truck type” to “excavator bucket size.” Two Key: = problem domain * = observation data points
other outputs are provided in this model, “excavator utilization

Excavator bucket size (m3)

factor” and “mean truck queue length” that would help the user
determine whether the excavator performance and number of 2.00 n
* * * * * *

Queue length t-10

trucks in the system are balanced. The dynamic version of the
* * * * *
model 关Fig. 3共b兲兴 runs iteratively, providing a sequence of esti- *
mates of production rate at constant increments in simulated time * * * * * * *
共perhaps every minute兲. Most of the input variables are the same
* * * * * *
as the static version of this model, except that recent truck queue *
lengths and excavator utilization factors are fed back as input to 0.25 * * * * * 0 *
the model at each cycle, given that these will correlate with the poor excellent 0 n
future production rate. The advantages of the dynamic model are Haul road condition Queue length t-5
that: 共1兲 it is possible to see changes in the performance of the
(a) Uncorrelated Input Variables (b) Correlated Input Variables
excavation system over time; and 共2兲 changes to the specification
of the excavation system 共such as the number of operational Fig. 4. Affect of correlation between input variables on the observa-
trucks兲 can be made during the simulation to see how this impacts tions required to develop a model 共a兲 uncorrelated input variables; 共b兲
performance. On the downside, development of the dynamic correlated input variables
model will require more observations of the performance of the
system, and these observations will have to cover more variables.
An example of dynamic modeling of excavation system using Markov models may generate values stochastically as random
empirical modeling techniques is provided by Flood and Christo- events in time 共Zhang 2006兲. Whatever variable types are used,
philos 共1996兲. care must be taken to ensure that they are compatible with the
The next point to consider in the structuring of the model’s type of model being used. Consider, for example, the input vari-
interface is the selection of the input variables. Obviously, the able truck type in Fig. 3共a兲 which may assume values such as
only input variables that should be included are those that are “type 1,” “type 2,” and so on. It is an enumerated type, but unlike
significant in terms of either affecting the values to be generated the other variables in the model, its values cannot necessarily be
at the output or being correlated with them in some way. How- placed in a progressive order that has some functional relationship
ever, the significant variables are often not known at the outset of with the output variables. If a meaningful order can be identified
an empirical modeling study. Determining an appropriate set of 共perhaps based on the haul capacity of the trucks in this case兲 then
input variables requires: 共1兲 expert judgment by the model devel- this should be adopted; certainly an arbitrary order of values
oper, including a review of other work; and 共2兲 experimentation should always be avoided. The inclusion of unordered variables in
in which the performance of the model is compared for alterna- a model is usually problematic, particularly if the mapping from
tive sets of inputs. Consider, for example, the variables truck type, inputs to outputs is a continuous function 共such as is the case for
“truck engine power,” and “truck haul capacity” in the excavation regression models and many artificial neural networks兲, since it
models of Fig. 3. It is likely that truck type implicitly defines both will introduce discontinuities to the problem that are difficult to
the “engine power” and the “haul capacity” of the trucks, result- model. Other examples of unordered variables commonly mis-
ing in some redundancy in this set of input variables. In this case, used in this way include “type of construction project” 共assuming
a choice will have to be made between these alternative variables. values such as “residential,” “commercial,” and “industrial”兲,
Using truck type has the advantage that it may be associated with “country,” and “project manager.” A good way to handle unor-
other relevant information about a truck beyond its engine power dered variables is to develop a separate model for each value of
and haul capacity 共such as the weight of the truck and thus its the variable—for the excavation model, this would require devel-
traction兲 potentially increasing the accuracy of the model. This oping a separate model for each type of truck.
would have to be confirmed in a comparative study, developing There are a couple of practical issues that should also be con-
the model with the alternative sets of inputs and measuring which sidered when developing the interface for an empirically derived
version produces the most accurate output. A second advantage of model. First, the number of observations required to develop a
using truck type instead of “truck engine power,” and “truck haul model tends to increase geometrically with the number of input
capacity” is that there will be one less input variable in the model variables, and so there is a strong incentive to include as few
which will, in turn, reduce the number of observations required to inputs as possible. Fortunately, this is not true for input variables
develop the model. On the other hand, the use of truck engine that are correlated such as “queue length t-5⬘,” “queue length
power and truck haul capacity as the inputs will provide greater t-10⬘,” and “queue length t-15⬘” in the dynamic version of the
modeling versatility, allowing for any combination of values for excavation system shown in Fig. 3共b兲. The reason for this can be
these two variables. understood from Fig. 4 which shows the distribution of problems
Another point to consider in the development of a model’s that can occur for two uncorrelated input variables 关Fig. 4共a兲兴 and
interface is the type of value each input and output variable will two correlated input variables 关Fig. 4共b兲兴. Clearly, the correlated
assume. Referring to Fig. 3共b兲, variables such as “mean produc- variables have a problem range that is highly constrained and thus
tion rate” and “haul distance” assume real-numbered values, can be represented by far fewer observations.
“truck queue length” and “number of trucks” assume positive The second practical issue affecting the development of a
integers, while “haul road condition” is an enumerated type model’s interface stems from the quality and availability of the
which may assume values such as “poor,” “satisfactory,” and observed data used to develop the model. It could be, for ex-
“good.” Other systems of modeling may use radically different ample, that the observed data set does not provide a representative
ways of encoding information. Pulse frequency coding, for ex- distribution of values for a variable 共in terms of numbers and
ample, represents values by the number of pulses generated in a distribution of data points across the range of a problem兲, or it
given period of time 关see for example, Dayhoff 共1991兲兴, while may not be possible to measure a variable accurately. Conse-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 39

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Key: module are used to define the specific characteristics of the sec-
= finite element tion of waterway represented by that component, and the modules
mesh
are assembled to provide a model capable of estimating traffic-
= acoustic input
concrete beam
/output sensors flow through a complete waterway system. The approach has also
been applied to dynamic modeling by Flood et al. 共2004兲, in this
case to model transient heat-flow in buildings. The primary ad-
Fig. 5. Finite-difference model of the acoustic response of a concrete vantage of the modular approach 共compared to using a monolithic
member used to identify the location and size of reinforcing steel model to solve the problem兲 is an ability to model a wider range
of problems given that there is indefinite number of ways in
which the modules can be assembled. It is also extensible, allow-
quently, including that variable in the study may provide little ing an increase in the functionality of the modeling system
extra benefit or, in some circumstances, it could impede the mod- through the addition of new modules.
el’s ability to develop an accurate representation of the problem.
Development Schemes
Internal Structure
Determining a complete definition for an empirical model will
Some models have a rich internal structure 共such as discrete-event usually involve the expert judgment of a model developer
simulation models兲 while others may be very simple in form complemented by the application of a model development algo-
共such as regression models兲. This internal structure may be de- rithm. An artificial neural network, for example, may require the
fined implicitly by the type of model being used, it may be de- model developer to decide on the number and connectivity of the
rived automatically by a model development algorithm, or it may nodes in the network and the type of activation function used at
be crafted by the model developer. Where a developer has some each node, then an error-gradient descent algorithm 共such as
choice or control over the internal structure, this should be done backpropagation兲 may be employed to determine the weights on
with care to ensure the accuracy and value of the resultant model. the connections. In the extreme case, development of the model
These points will be discussed in detail in the following para- will be done entirely by the model developer without the use of a
graphs. development algorithm. An example of this would be the devel-
The objective of most empirical modeling studies is to develop opment of a simulation model in which the observed data are used
a device that can make accurate estimates or predictions about by the model developer to direct the design of the model and to
some aspect of a system—the focus is on the output generated by validate its performance. At the other extreme, the development
the model. However, a potentially powerful yet under exploited algorithm may perform all the model development work. For ex-
application of empirical modeling focuses on the internal struc- ample, in a regression study, the development algorithm may de-
tures that a model forms when developed to solve a given prob- termine both the type of function to use 共based on some statistical
lem. If the model is configured carefully, these structures can tell test such as the goodness-of fit兲 as well as the coefficients of the
us something important about the physical structure of the system function. Typically, the model developer has been responsible for
being modeled or provide a set of rules or principles that can be performing tasks such as selecting the basic type of model and
used to solve related problems. This type of application of em- determining its structure, while the development algorithm has
pirical modeling can be classified as a branch of AI, providing an been responsible for deriving the model’s coefficients. However,
automated approach to inductive reasoning. recent advances in computer-based search techniques have made
To illustrate this concept, consider the problem of detecting the automated development of models with complicated structures a
location of steel reinforcement in a concrete member from its viable option. Simulated evolutionary methods, for example, are
response to acoustic signals generated at discrete points on its sufficiently flexible to allow development of all aspects of just
surface. A finite-difference based mesh could be set-up that simu- about any type of mathematical model. The problems for the
lates the acoustic behavior of the structure as if it is composed model developer then become: determining the right type of al-
entirely of concrete, as shown in Fig. 5. This model could then be gorithm to adopt for a given problem; and coding the problem in
adapted by substituting elements representing concrete 共indicated a way that facilitates application of these techniques.
by the gray diamonds兲 with those representing steel or boundaries Most model development algorithms operate iteratively, ad-
between steel and concrete, or by adjusting the values of the vancing the model in discrete steps toward a design that is able to
coefficients, until it is able to replicate the acoustic response ob- replicate, to within an acceptable tolerance, the response of the
served in the actual concrete structure. This adaptation process system under investigation. Progress in model development is
may be achieved using, for example, genetic algorithm techniques measured and directed by an objective function, which measures
关such as adopted by Salem et al. 共2007兲兴. A critical decision for error at the outputs from the model for a comprehensive set of
the modeler in this example would be to select a mesh density for example problems. Often, this measure is the sum of the square of
the model that would be able to identify the location and size of the differences between the models actual output and the required
the reinforcing steel to the required precision. output for the examples. The objective is to develop the model to
In some cases, the basic internal structure of a model may be minimize this error. For static models, the objective function is
crafted by the model developer. The end product may take the based on the errors the model generates for a set of independent
form of a set of empirically developed modules that can be pieced observations of the problem, each mapping from an input vector
together as required to represent each new version of the problem. to a corresponding output vector, as illustrated in Fig. 6共a兲. For
Zhu et al. 共1999兲 adopted this approach for static modeling of the dynamic models, the issue is more complicated than this, since
flow of commercial traffic through inland waterways. The system the model will generate a stream of vectors at the outputs in
comprises a menu of neural network modules, each trained to response to each input vector. The concern is that when the model
estimate traffic-flow through a type of component in a waterway is running dynamically, it will be feeding information back 共either
system, such as a lock or stretch of canal. The inputs to each at its interface or internally兲 thus possibly leading to a compound-

40 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Key: * = target outputs;
= model’s output;
Key: = target for one example problem;
= model’s output for the problem;
Step 1: Strategizing
= errors.
= error. The first step in model development is to establish as much
familiarity as possible with the problem at hand, including:
output value output value
* 1. Establishing the application objectives of the model—
*
*
addressing issues such as what output variables are required,
* *
* ** * whether the output should be static or dynamic in form,
*
input value time/iteration in sequence whether the focus of the study is on the internal structures
(a) errors for multiple problems in a static model (b) error for 1 example problem in a dynamic model developed by the model or on the output values it generates,
and the level of accuracy required.
Fig. 6. Comparison of errors for static and dynamic models 共a兲 errors 2. Determining the input variables likely to be significant.
for multiple problems in a static model; 共b兲 error for 1 example prob- 3. Gaining a feel for how the system being modeled responds to
lem in a dynamic model different conditions, such as, whether it is linear versus non-
linear, or stochastic versus deterministic.
4. Determining the availability of data for developing the model
and the sources for additional data.
ing of errors. This is illustrated in Fig. 6共b兲 which shows that the The questions that should be answered at this stage are:
first output response for a dynamic model can have a small error, 1. What type and structure of model is most appropriate to
but with subsequent iterations, the error can start to increase. For solve the problem?
dynamic models, the error for each example of a problem should 2. What development algorithm will help derive the most effec-
be integrated across the full sequence of outputs from the model, tive model?
not just be based on the first output in the sequence. However, 3. What is the objective function?
operating an objective function in this way will require a lot of 4. What new studies will likely be required to acquire the nec-
computation, requiring a complete run of the model for each ex- essary data for development and validation of the model?
ample problem to be tested; there may be many such examples to In addition, the feasibility of the study in terms of its time,
be tested in this way and, moreover, this process will be nested cost, and availability of resources should be assessed. It is often
inside the model development algorithm which itself will require the case that several of the above questions cannot be answered
many iterations to fully develop a model. Consequently, it is often fully and with confidence until after some development and
not practicable to use full sequences of outputs from dynamic analysis work has been completed. If this is critical then it may be
models to measure error during the development stage 共the first necessary to conduct a pilot study developing a less advanced
outputs generated in the sequence may have to be used instead兲; version of the model to help answer these strategic questions
however, the final validation of the model should always be based more thoroughly.
on the errors measured over the complete sequence of outputs for One of the most insightful tasks that can be performed at the
each example problem. start of a modeling study is to gain a graphical understanding of
Empirical modeling offers many advantages over other mod- the problem using, if available, existing observations of the re-
eling techniques. Based on the experiences of the writers, there sponse of the system or of a similar system. There are two basic
are three main classes of problem to which the empirical ap- types of graph that should be considered here, the first plotting
proach is best suited, namely: those that are poorly understood; each output variable against each of the input variables, and the
those that are overstated; and those that require results to be pro- second plotting each of the input variables against each other. Fig.
duced quickly. An overview of these classes is provided in Table 7 shows examples of the first type of plot, demonstrating:
1, along with an explanation of the reasons why empirical mod- 1. The relevance of the input variables—for example, the
eling is successful in each of these cases, an illustration of poten- highly random distribution of the points for input Variable 1
tial applications within construction, and an identification of the in the figure indicates that it has little relevance to the prob-
corresponding limitations of the approach. As noted in the table, lem and thus might be excluded from the study.
the main constraint on empirical modeling is the fact that the 2. The complexity of the response of the system in terms of
number of examples of a problem needed to develop a model whether it is linear or nonlinear and, in the latter case, the
tends to increase geometrically relative to the number of input form of the nonlinearity—for example, input Variable 2 in
variables in the problem. This often places a practical limit on the the figure appears to have a nonlinear relationship with the
complexity of the problem that can be considered. Fortunately, output variable.
this limitation does not hold for problems where the values of the 3. The existence of unexplained variance in the response of the
input variables are strongly correlated with each other 共in this system, possibly due to noise or missing input variables that
case, the number of examples required may only increase lin- are significant—for example, input Variable 2 does not ap-
early兲. Moreover, as discussed later, it may be possible to circum- pear to fully explain the response of the system.
vent this limitation by developing models with more complex An example of the second type of graph 共plotting input vari-
internal structure. ables against each other兲 is provided earlier in Fig. 4. This type of
plot is useful for determining correlation between the input vari-
ables, which would suggest possible redundancy between those
Empirical Modeling Methodology inputs or at east a reduction in the distribution of the observations
required to represent those variables.
Development and implementation of an empirical model must The importance of developing an understanding of the re-
follow a rigorous set of procedures to ensure the validity and sponse of the system to selecting an appropriate model type and
value of the end product. These procedures can be divided into structure can be demonstrated by the simple example problem
the following six steps. shown in Fig. 8. The figure shows three different models of the

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 41

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Table 1. Problem Areas where Application of Empirical Modeling is Most Appropriate
42 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Application area Empirical modeling advantage Specific examples in CEM Issues

Problems that are poorly understood, A problem, even if it lacks a strong •Estimating concrete slump and cured strength The complexity of the problem that can be
that is: theoretic framework, may be from the properties and proportions of its considered will be limited by the fact
•where there is limited or no theory characterized by a comprehensive set constituents, including the profile of aggregate that the number of fitting data examples
quantifying the relationship between of examples of its performance or sizes, and the inclusion of admixtures. required to develop the model will increase
the input and output variables; behavior. Empirical modeling methods •Estimating project manager suitability for geometrically with the increase in the
•where a comprehensive list of relevant have the ability to develop a representation assignment to a given project, based on key number of input variables.
input variables has not been established; of a problem from such a set of examples. performance parameters measured from previous
and/or similar projects, and key descriptors of the
•where the intent is to base the model on current project.
correlations between the input and output
variables rather than on cause-effect
relationships.
Problems that are overstated, that is: Empirical modeling methods can •Predicting the duration for earth hauling equipment The number of fitting data examples
•where there is a lot of overlap/correlation determine a mapping from an overstated to travel along a haul road, based on parameters required may increase geometrically
between the values of the input variables; set of input variables to the output such as a series of gradients measured at increments with an increase in the number of input
and variables, which compensates for the along the haul road, as well as attributes of the variables, but only in relation to those
•where the problem can be solved with excess of information. Moreover, in these haul equipment. input variables that are not correlated
alternative subsets of the input variables. cases, an empirical model can work •Estimating the load on each axle of a truck, based with each other.
effectively even if some of the input on the series of strain readings it induces on a If there is the possibility of missing input
values are missing or contain bridge girder during the truck crossing event. values, then the empirical model will
significant errors. require sophisticated internal structuring in
order to solve the problem.
Problems that require rapid execution and Most empirical models provide a •Modeling dynamic heat-flow through composite Modeling more complicated and/or a
existing methods are unacceptably slow. simple function which maps directly materials, such as occurs for endothermic heat broader scope of systems requires a
In particular, acting as an alternative to from the inputs to the outputs. These generation and dissipation in curing concrete modularized approach. This can place a
numeric simulation techniques 共which functions are not very demanding structures. large burden on the programmers/developers
are notorious for taking several hours or computationally and can be executed •Modeling acoustic wave propagation in enclosed of the original modeling system, that can
days to run, especially if many simulations in a fraction of a second. spaces, such as from noisy construction equipment be time consuming and expensive.
must be executed to help determine an Aspects of a problem that require operated indoors causing health concerns.
optimal design solution or to gain a assembly from many components in a
statistical summary of stochastic processes兲. traditional simulation model can be
treated as a single variable in an
empirical model.

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 output output squares兲. The first model attempts to fit a straight line to the data,
*
* ** which appears to be quite successful except for some relatively
* * * * small errors indicated by the differences between the white dia-
* * * * *
* * * * * * * * * *** * monds and the straight line. The second model 关Fig. 8共b兲兴 recog-
* *
* * *
input variable 1 input variable 2
nizes a slight acceleration in the data 共probably resulting from
increased friction in longer piles兲 and thus attempts to fit a curved
Fig. 7. Plotting output against input for a set of existing observations line. This provides a better model in that the errors for the testing
of the response of a system data are reduced. There is still some unexplained variance be-
tween the data points and the curve, which could possibly be
removed by using a more complicated shaped curve or by includ-
same problem, that of estimating the duration to sink a pile given ing additional input variables 共such as soil type or rig operator
the length of the pile. Each model attempts to fit a line to a set of experience兲. Alternatively, the variance could be due to error
observations collected from various construction sites. These ob- 共noise兲 in the data in which case no improvement in the model
servations are represented by the fitting data set 共indicated by the could be achieved. In fact, using a more complicated curve than is
black diamonds兲 and the performance of each model is evaluated warranted by a problem can lead to a loss in performance. This is
relative to the observations in the testing data set 共the white seen in the third version of the model 关Fig. 8共c兲兴 which is able to

fitting data points testing data points fitting data points testing data points

30 30

25 25
Duration to Sink Pile (hours)

Duration to Sink Pile (hours)

20 20

15 15

10 10

5 5

0 0
0 5 10 15 20 25 30 0 5 10 15 20 25 30

Pile Length (m) Pile Length (m)

(a) straight line (b) simple curve

fitting data points testing data points

30

25
Duration to Sink Pile (hours)

20

15

10

0
0 5 10 15 20 25 30

Pile Length (m)

(c) convoluted curve (many degrees of freedom)

Fig. 8. Fitting functions of different complexity to a set of observations 共a兲 straight line; 共b兲 simple curve; and 共c兲 convoluted curve 共many
degrees of freedom兲

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 43

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
fit a curve 共with many degrees of freedom兲 very closely to the Key: = problem domain; * = observation data points
fitting data points. However, in this case the curve is too flexible under represented
共having more degrees of freedom than data points in the fitting region

set兲 and thus behaves erratically between the fitting data points. * * * * *
* *

input 1
As a consequence, the model performs very poorly when it is * * *

input 1
* *
evaluated with the testing data 共artificial neural networks, for ex- *
**
**
ample, can suffer from this problem since they allow the model * ***** *** * * *
input 2 input 2
developer to add an unlimited number of degrees of freedom to
(a) input variable 1 (b) input variable 2 (c) input variable 1 versus input variable 2
the solution兲. It can be seen that, even for this very simple ex-
ample, the type and structure of a model must be matched care-
Fig. 9. Distribution of twelve observations across the problem do-
fully to the problem.
main 共a兲 input variable 1; 共b兲 input variable 2; and 共c兲 input variable
Generally, decisions about the type of model to adopt for a 1 versus input variable 2
study will precede selection of the model development algorithm.
Nevertheless, for any given type of model, there will likely be
many choices of development algorithm available. Important fac- distribution across the problem domain. The number of observa-
tors to consider in selecting an appropriate algorithm are: tions required will tend to increase with each of the following
1. Whether the algorithm is able to develop the structure of the factors:
model or just its coefficients—if it cannot develop the struc- 1. The number of input variables 共generally, this will be a geo-
ture of the model, then this may have to be done by the metric relationship except where the input variables are cor-
model developer. related with each other兲;
2. The degree of expertise required in its use—for example, 2. The complexity of the shape of the function being modeled;
artificial neural network training algorithms often require a and
lot of experience in their use in order to get them to converge 3. The amount of error 共noise兲 in the observed data.
on a solution, while simulated evolution requires skill in de- While statistical tests exist for determining an appropriate
termining a functional coding mechanism for the problem. number of sample observations 关see for example, Fellows and Liu
3. The likely rate of model development—model development 共1997兲兴 these are limited to assuring the statistical properties of
algorithms can be computationally very expensive requiring the data such as its mean value. In general, a sensitivity analysis
much iteration to converge on an acceptable solution, and is the most effective way of determining an appropriate number of
may not be completed within a reasonable period of time. observations for model development—this will require the model
Finally, it should be remembered that modeling studies often to be developed for a range in the number of observations, start-
incur failures but, at the same time, provide insights that can ing with a relatively small number then increasing until the re-
stimulate new ideas for solving a problem. The study should, quired level of model performance is achieved.
therefore, allow for flexibility in its approach. The distribution of observations in an existing set should be
checked by plotting them as scatter diagrams for each input vari-
able. Figs. 9共a and b兲 show such plots for a data set comprising 12
Step 2: Data Collation and Evaluation observations in a two input variable problem. Both figures indi-
cate a good distribution of points between the limits of the prob-
Most empirical modeling studies require three independent sets of lem domain. However, plotting the distribution for individual
observed data: input variables can be misleading. Fig. 9共c兲 shows the same 12
1. Fitting data set—the observations that the model, through observations plotted for both input variables simultaneously, re-
development, attempts to replicate or approximate. vealing a lack of points in the lower right corner of the problem
2. Testing data set—the observations used to evaluate the per- domain. For similar reasons, plotting the observations for two
formance of the model during its development and to help input variables for problems comprising three or more variables
select between competing models. can also be misleading. Nevertheless, this type of visualization
3. Validation data set—the observations used to make a final will provide useful insight into the quality of the data set.
assessment of the performance of the model. If it is necessary to generate additional observations, or if the
The content of all three types of data set needs to be selected study requires the generation of a completely new set of observa-
carefully to ensure the value and performance of the end model. tions, then there are several alternative sources that may be
Ideally, the testing of a model under development should be done tapped:
in the context of its end application. For example, if a model is 1. Systems that cannot be controlled for the purposes of data
dynamic in operation then each observation used for testing the collection such as ongoing construction projects.
model’s performance should be a series of values representing a 2. Systems that can be controlled including laboratory based
complete simulation run. Likewise, if the purpose of a study is to physical models.
identify the internal model structures that form during the devel- 3. Computer-based models such as simulation—in this case, the
opment process, such as for the acoustic response model dis- purpose of the study may be to develop an empirical model
cussed earlier 共Fig. 5兲, then testing of the model should be made that can generate solutions significantly faster than the exist-
relative to the accuracy or utility of those internal structures, not ing model兲.
relative to the output from the model. This may not be feasible for The first of these sources provides the least control over the
testing during the model development stage due to excessive distribution of the data within the problem domain and thus may
computational demands, but it should at least be adopted when it require careful assessment of the collected data to make sure that
comes to the final validation of the model. it is well distributed—this may necessitate collecting observations
If existing observations are used for model development, they from several examples of the system to achieve a comprehensive
must be assessed to see if they are sufficient both in number and set of observations. Systems that can be controlled provide more

44 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Key: = problem domain; * = observation data points Key: = fitting data set error Key: = fitting data set error
= testing data set error = testing data set error
* = selected versions of model
region with more noise or controlled
more complicated response anchor points Error Error

*
* * * * * * * * * * ** * * * * * *** ** * * * * ** * * * * **
input 1

input 1

input 1

input 1
* * * * * * * * * * ** * * ** *** * * ** * ** * * * * * * **
* * * * * * * * * * ** * * * ** ** * ** ** ** * * * * * * * * *
* * * * * * * * * * ** * * * * * * ** * *** * * * * * ** **
* **
* * * * * * * * * * ** * * *** *** ** *** ** * ** * * * * *
input 2 input 2 input 2 input 2 * * * *
(a) regular grid (b) variable density grid (c) regular and random (d) random
Iteration # Iteration #

Fig. 10. Distribution of observations collected from controllable sys- Optimal version of Optimal version of
model model
tems 共a兲 regular grid; 共b兲 variable density grid; 共c兲 regular and ran- (a) error-gradient descent applied to model coefficients (b) simulated evolution applied to model structure
dom; and 共d兲 random
Fig. 11. Progress in model development for studies that use search
algorithms 共a兲 error-gradient descent applied to model coefficients;
flexibility in selecting the distribution of observations, but time 共b兲 simulated evolution applied to model structure
constraints and other limited resources can be restrictive.
Computer-based experiments provide the greatest control, al-
though simulation models can run extremely slowly thereby lim- density that corresponds to the likelihood of that situation being
iting the number of experiments that can be performed within the encountered. This can be achieved using Monte Carlo sampling
project’s time constraints. techniques provided that the distribution is known.
When no control over a system is available, the model devel- As a final point, it should be remembered that empirical mod-
oper should still be able to choose the aspects of the system els are often only valid when operating within the bounds of the
observed and the timing of the observations. For controllable sys- fitting data set, so it is imperative that this scope be set to em-
tems 共including computer-based models that are used to generate brace all problems that may be encountered in the end applica-
observations兲 the model developer must design a set of system tion.
input conditions that will ensure the entire problem domain is
covered. Within this domain, the model developer may choose
Step 3: Model Development
between an orderly grid of observations and a random set of
values, such as shown in Fig. 10 for a two input variable problem. A conceptual design for the model will have been determined in
Fig. 10共a兲 shows a regular grid, which is usually the most effec- Step 1. This is developed into a finalized design in Step 3, using
tive distribution strategy to adopt for the fitting data set. However, the observations in the fitting data set and the testing data set. The
sometimes more information is required about some regions of fitting data are the target observations used for model develop-
the problem domain, either because the response of the system is ment, while the testing data are the observations used to evaluate
more complicated in that area or because it contains more noise, the performance of the model at each stage in its development and
and so the grid for the fitting data set should provide a greater to select between competing models. A common mistake is to use
density of points at that location 关as illustrated in Fig. 10共b兲兴. An the same set of data for both fitting and testing. The problem is
alternative strategy is to select observation points at random, as that the model will be inherently biased toward the data used in
shown in Fig. 10共d兲. This strategy is useful for the fitting data set its development 共the fitting data兲 and so if these were also used to
when the collection of observations and development of the assess performance it would result in an overestimation of the
model is to be performed in stages. In this case, the number of accuracy of the model. The use of a second independent data set
observations collected at any stage may not be sufficient to form for testing of the model avoids this bias. This can be illustrated by
a regular grid. When a random distribution is used for a fitting inspecting the output from the three piling models considered
data set, it is still helpful to at least include anchor-point obser- earlier in Fig. 8. If the performance of these models was measured
vations at each corner of the problem domain as indicated in Fig. for the observations used to develop the model 共the black dia-
10共d兲, to ensure the full scope of the problem is represented. It is monds兲, then we would conclude that the version shown in Fig.
also possible to use a regular grid for some input variables and a 8共c兲 is the best 共having the least error between the model output
random distribution for other input variables such as shown in and these observations兲. However, measuring performance for the
Fig. 10共c兲. independent observations in the testing set 共the white squares兲
For the testing data set and the validation data set, the obser- concludes, correctly, that this is actually the worst of the three
vations should always be randomly distributed across the problem models.
domain. A mistake sometimes made by researchers is to generate Model development can be a straightforward process executed
a set of observations using a grid strategy 共or mixed grid and on a computer in a fraction of a second 共such as calculating the
random strategy兲 and then to reserve a random sample of these for coefficients in a linear regression兲, or it may involve a search
testing and validation. The problem with this approach is that algorithm that uses both computer-based methods combined with
these observations will all fall along a finite number of lines, human judgment and can take several days to complete. Search
along which the observations used to develop the model are also algorithms can be applied at any level in the development of a
located, thus biasing the assessment of the performance of the model from determining its coefficients through to forming its
model. complete structure.
Sometimes, it is desired to measure the performance of a For studies that use search algorithms, the testing data set
model in a way that is weighted toward variations of a problem plays an important role in helping determine when to terminate
that are more likely to be encountered in the end application. In the model development process. Fig. 11, for example, shows typi-
this case, observations in the testing and validation data sets cal progress in the development of a model where: 共a兲 the model
should be distributed across the problem domain with a varying parameters are adjusted using an error gradient descent technique

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 45

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 error when no initial

Error for Testing Data

Error for Testing Data

input variables are
excluded

Bucket size
Road condition
Haul capacity
Num. trucks
Truck type

Haul distance
Engine power
Input Variables 0 1 2 3 4 5
Excluded from Number of Historic Values
Model Used as Inputs to the Model

(a) alternative sets of input variables (see Fig 3(a)) (b) alternative numbers of historic values (see Fig 3(b))

Fig. 12. Searching for an input configuration for the excavation model that minimizes the testing error 共a兲 alternative sets of input variables 关see
Fig. 3共a兲兴; 共b兲 alternative numbers of historic values 关see Fig. 3共b兲兴

共such as error backpropagation applied to artificial neural net- models performance as indicated in Fig. 12共b兲. It is likely that
works 共Rumelhart et al. 1986兲兴; and 共b兲 the structure of the model increasing the number of historic values will increase the accu-
is adjusted using a heuristic search procedure 共such as simulated racy of the model asymptotically as shown in the figure. In this
evolution兲. In the latter case, milestone versions of the model that case, the decision about the number to include must be based on
provide an improvement in performance 共and are thus adopted as practical issues such as at what stage does the model reach an
the basis for the next iteration in the search兲 are indicated by the acceptable level of performance, when does the improvement in
asterisks. In both cases, when the performance of the model is performance become insignificant, or does the addition of new
measured for the fitting data set, the error curve 共shown as a solid inputs make the model operate too slowly? Abi-Shdid 共2005兲 pro-
line兲 appears to continue improving. However, when performance vides an example of this type of experimentation to determine the
is measured for the testing data set, the error curve 共the dashed number of historic values to use in an empirically derived dy-
line兲 reaches a low-point and then rises implying that the perfor- namic model of heat-flow in buildings.
mance of the model is starting to degrade. In other situations, the As a final point, it should be noted that changing the value of
error curve for the testing data may just level-out. Given that the one model parameter can affect the optimal value for another
testing data set should provide a less biased assessment of the parameter, and so the manual search process can require a con-
performance of the model, it is clear that the version of the model siderable effort from the model developer in order to home-in on
that is adopted should be the one where the testing error curve an optimal or at least acceptable solution.
共not the fitting error curve兲 ceases to improve.
Finding the minimum point on the error curve for the testing
Step 4: Model Evaluation and Final Selection
data 共as shown in Fig. 11兲 is only the first step in developing the
model. Model parameters that are not adjusted by the search al- The development step will have produced one or more candidate
gorithm need to be adjusted manually in a series of comparative models. The next task is to evaluate those models to see whether
performance studies. This should be done methodically, altering they perform satisfactorily and, where required, to select the best.
the parameters one-at-a-time and seeing how this affects the mini- This should be executed using the observations in the testing data
mum error that can be achieved by the development algorithm. set.
Examples of model parameters that are commonly adjusted in this The first set of tests should focus on the generalized perfor-
way are: the set of input variables included in the model, the type mance of the models 共integrated across the entire problem do-
and number of elements included in the model’s internal struc- main兲 using measures such as: the mean absolute error; the mean
ture, and the number of observations used for developing the actual error 共for determining positive or negative bias in the
model. model兲; the worst error; and variance in the error 共to assess the
To illustrate this point, Fig. 12共a兲 shows the results for a series distribution of the error兲. If the model is dynamic with each ob-
of performance experiments involving the static version of the servation representing a sequence of outputs, then the generalized
excavation model 关shown in Fig. 3共a兲兴 in which each input vari- tests would require the errors to be integrated across each se-
able is excluded in turn from the model development process. For quence before integrating across all observations. This may in-
comparison, the curve also shows the performance of the model clude, for example, calculating the mean absolute error for each
when none of the inputs in the initial set are excluded. From this, sequence and then finding the mean or worst of these for all
a decision can be made as to whether any of the inputs can be observations. Alternatively, a statistical measure such as Theil’s
permanently excluded. The study could proceed by testing the test 共Theil 1961兲 could be applied to each sequence of outputs to
exclusion of combinations of input variables. Sometimes this type assess its closeness to the target sequence, and then the mean and
of study will take the form of a sensitivity analysis. For example, worst of these results could be used as a generalized assessment
Fig. 12共b兲 shows a search for an acceptable number of historic of performance.
values to be included as inputs in the dynamic version of the An evaluation should also be made of the distribution of the
excavation model 关shown in Fig. 3共b兲兴. The model includes three errors in the model across the problem domain. This is a step that
historic inputs as drawn, measured at 5, 10, and 15 times steps in is neglected in most modeling studies, but is essential to assure
the past, although increasing this number will likely improve the that the model is performing sufficiently well in all regions of the

46 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 testing testing unacceptable
error acceptable error acceptable error
error limit error limit
* * 4.05

input B
* * ** 3.60
* ** * * * * 3.15
* * * *** 2.70
* ** * *
* * 2.25
* *
* * * * * * ** *** * *** 2.25
input 1 input 2 input A
(a) error plotted against each input variable (b) error plotted as a contour map

Fig. 13. Evaluating error across the problem domain 共a兲 error plotted against each input variable; 共b兲 error plotted as a contour map

problem domain. This can be done visually by plotting error 3. The usage of the model for problem solving.
against each input 关as shown in Fig. 13共a兲兴 or by plotting an error When possible, feedback on the performance of the model in
contour for two inputs at a time 关as shown in Fig. 13共b兲兴. Fig. practice should be obtained to reconfirm its validity and if neces-
13共a兲 shows a model that appears to be performing relatively well sary to assist in its further development.
across all values of its input Variable 1, but tends to perform
worse for higher values of input Variable 2—this may indicate
that more observations are required for development of the model
Conclusions
in this region of the problem. The approach of plotting an error
contour map, as shown in Fig. 13共b兲, provides additional infor-
mation about any correlation between pairs of input variables in Empirical modeling is one of the most commonly used analytical
terms of the occurrence of error. In this case, the error appears to methods in CEM, and is certainly the most diverse in terms of its
be worst where input A is low and input B is high. scope of application. The advent of fast, inexpensive desktop
This type of analysis should be performed for all input vari- computing, combined with recent advances in empirical modeling
ables. It may be found that overall the model performs well, but development algorithms, has helped make this possible.
there are certain regions of the problem domain where its perfor- A major criticism of empirical modeling is that it is a black
mance is unacceptable. Consequently, the model with the best box method, providing little explanation of its output results and
overall performance may not be the one that is selected for final no ability to extrapolate, and is therefore best used as a compro-
implementation. Alternatively, it may be worthwhile returning to mise in situations where a theoretical modeling framework is un-
Steps 2 and 3, to collect additional data focused on the poorly available. However, this characterization is somewhat unfounded
performing regions of the problem domain, with the intent of in that: 共1兲 empirical techniques are critical to the development of
developing an improved version of the model. any theoretical model, making the approach more prevalent than
may be realized; 共2兲 if used carefully, empirical modeling can
Step 5: Final Validation provide insights into the internal structure and principles that
drive a system; and 共3兲 modularization of a model can extend its
Once the final version of the model has been selected, it is nec- scope of application. That said, most applications have been lim-
essary to reevaluate its performance using a third independent set ited to simple input to output mapping devices that are not de-
of observations, the validation data set. This provides a final as- signed to identify the internal operation of a system, and often
sessment of the performance of the model and confirmation of its make use of correlated relationships as much as cause-effect re-
validity. Note that the testing data set cannot be used for this lationships between the system’s variables. This may be a reason-
purpose since it was used to select versions of the model during able approach for many studies but it leaves much of the potential
the development and final selection stages, and thus may have
of empirical modeling unexploited. Future efforts should focus on
some bias toward the final model. The same tests that were used
realizing this potential. Classes of problem that cannot be mod-
in Step 4 can be adopted for the final validation. However, it is
eled using simple input to output mapping devices, but could be
essential that the validation relate directly to the way in which the
solved by empirical models that have a more complex internal
model will be used in the end application: if the model is dy-
namic, then it must be validated for complete sequences of output structure, include those where:
values; alternatively, if the study is focused on the internal struc- 1. The set of input values may be incomplete 共eg., estimating
tures that form during model development, then it must be vali- the construction cost of a water treatment plant from a set of
dated in terms of the accuracy or utility of those structures rather its design parameters where, for some versions of the prob-
than the accuracy of the output from the model. lem, not all of these parameters will have been determined兲;
2. The set of input variables may differ from problem to prob-
lem 共eg., estimating the production rate of earthmoving sys-
Step 6: Implementation and Review tems where the number of features on the haul road that have
Implementation of the model requires education of the end users to be defined, such as slopes, will differ from problem to
共through documentation or tutorials兲 and should cover the follow- problem兲;
ing issues: 3. The objective of the study is to develop insights about the
1. The collection and organization of the data to be presented as system being investigated from the internal structure of the
input to the model, to ensure its validity and accuracy; resultant model 共e.g., identifying the location and diameter of
2. Interpretation of the output from the model; and steel reinforcing bars within concrete structures from the

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 47

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 acoustic response of the structure—see the example dis- Elsevier Science, New York, 307–320.
cussed earlier in Fig. 5兲; and Flood, I., Issa, R. R. A., and Abi-Shdid, C. 共2004兲. “Simulating the ther-
4. The important information defining a problem is part of a mal behavior of buildings using artificial neural networks-based
time or spatial series of values, but it is not clear where the coarse-grain modeling.” J. Comput. Civ. Eng., 18, 207–214.
Gagarine, N. 共1991兲. “Advance in weigh-in-motion method with pattern
key information starts or finishes within that series 共e.g., au-
recognition and prediction of fatigue life of highway bridges.” Ph.D.
tomated recognition of quantities of material in stockpiles dissertation, Univ. of Maryland, Dept. of Civil Engineering.
from video images where the location of the material within Halpin, D. W., and Woodhead, R. W. 共1976兲. Design of construction and
the frame of reference may vary兲. process operations, Wiley, New York.
Han, S. H., Kim, D. Y., and Kim, H. 共2007兲. “Predicting profit perfor-
mance for selecting candidate international construction projects.” J.
References Constr. Eng. Manage., 133, 425–436.
Ko, C. H., and Cheng, M. Y. 共2007兲. “Dynamic prediction of project
success using artificial intelligence.” J. Constr. Eng. Manage., 133,
Abi-Shdid, C. 共2005兲. “ANN coarse-grain based-model for simulating
316–324.
two-dimensional heat transfer in buildings.” Ph.D. dissertation, Univ. Lowe, D. J., Emsley, M. W., and Harding, A. 共2006兲. “Predicting con-
of Florida, Rinker School. struction cost using multiple regression techniques.” J. Constr. Eng.
Ahadzie, D. K., Proverbs, D. G., and Olomolaiye, P. O. 共2008兲. “Model Manage., 132, 750–758.
for predicting the performance of project managers at the construction Marzouk, M., and Moselhi, O. 共2004兲. “Fuzzy clustering model for esti-
phase of mass house building projects.” J. Constr. Eng. Manage., mating Haulers’ travel time.” J. Constr. Eng. Manage., 130, 878–886.
134, 618–629. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 共1986兲. “Learning
Cheung, S. O., Yiu, T. W. Y., and Yeung, S. F. 共2006兲. “A study of styles internal representations by error propagation.” Parallel distributed
and outcomes in construction dispute negotiation.” J. Constr. Eng.
processing, Vol. 1, D. E. Rumelhart and J. McClelland, eds., MIT,
Manage., 132, 805–814.
Cambridge, Mass.
Dayhoff, J. 共1991兲. “Pattern recognition with a pulsed neural network.”
Salem, O., Shahin, A., and Khalifa, Y. 共2007兲. “Minimizing cutting
Proc., Conf. on Analysis of Neural Network Applications, ASM Inter- wastes of reinforcement steel bars using genetic algorithms and inte-
national, Materials Park, Ohio, 146–159. ger programming models.” J. Constr. Eng. Manage., 133, 982–992.
Ezeldin, A. S., and Sharara, L. M. 共2006兲. “Neural networks for estimat- Stoy, C., Pollalis, S., and Schalcher, H. R. 共2008兲. “Drivers for cost
ing the productivity of concreting activities.” J. Constr. Eng. Manage., estimating in early design: Case study of residential construction.” J.
132, 650–656. Constr. Eng. Manage., 134, 32–39.
Fang, D., Chen, Y., and Wong, L. 共2006兲. “Safety climate in construction Theil, H. 共1961兲. Economic forecasts and policy, North-Holland, Amster-
industry: A case study in Hong Kong.” J. Constr. Eng. Manage., 132, dam.
573–584. Wong, J. M. W., Chan, A. P. C., and Chiang, Y. H. 共2008兲. “Modeling and
Fang, D., Fong, P. S., and Li, M. 共2004兲. “Risk assessment model of forecasting construction labor demand: Multivariate analysis.” J. Con-
tendering for Chinese building projects.” J. Constr. Eng. Manage., str. Eng. Manage., 134, 664–672.
130, 862–868. Zhang, X. 共2006兲. “Markov-Based optimization model for building facili-
Fellows, R., and Liu, A. 共1997兲. Research methods for construction, ties management.” J. Constr. Eng. Manage., 132, 1203–1211.
Blackwell Science, Oxford, U.K. Zhu, L., Schonfeld, P., Kim, Y. M., Flood, I., and Ting, C. J. 共1999兲.
Flood, I., and Christophilos, P. 共1996兲. “Modeling construction processes “Waterway network analysis with artificial neural networks.” Artif.
using artificial neural networks.” Automation in construction, Vol. 4, Intell. Eng. Des. Anal. Manuf., 13共5兲, 12.

48 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Quantitative Research: Preparation of Incongruous
Economic Data Sets for Archival Data Analysis
Gunnar Lucko, Ph.D., A.M.ASCE1; and Zane W. Mitchell Jr., Ph.D., M.ASCE2

Abstract: In the field of construction engineering and management, archival data sets are not always as correct and consistent as it would
be desirable. Between different sources that are studied, e.g., companies, they may differ in format or content and within them, they may
still be incongruous and require substantial preparation. This makes examining theories and extracting trends from historic data more
difficult than it is for carefully controlled experimental studies or for collecting new data. The purpose of this paper is not to review the
regression models that the writers developed during their research, but to focus on the data preparation that had to be applied before those
analyses. The objective is to outline various techniques that can be applied to archival data that are related to construction engineering and
management to give researchers a set of best practices on data preparation that can assist them in gleaning truths from them.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000078
CE Database subject headings: Construction companies; Construction equipment; Construction industry; Data analysis; Economic
factors; History; Computer software; Statistics; Databases.
Author keywords: Construction companies; Construction equipment; Construction industry; Data analysis; Economic factors; History;
Inflation, economic; Qualitative analysis; Spreadsheets; Statistical models.

Introduction data. These dissimilar data had to be integrated and prepared to

derive predictive equation. Lucko 共2003兲 focused on the residual
Construction engineering and management research is by its na- value of equipment. Data on four major manufacturers were col-
ture strongly related to field operations. Contracting companies lected from two different auction record publishers that covered
routinely measure numerous aspects thereof, especially time, cost, numerous types of equipment. While the data sets were in elec-
and productivity, to fulfill the project management functions of tronic form, their contents still had to be checked for errors,
planning, optimization, and control. This accumulates a valuable matched with other data that characterized e.g., the size or the
pool of data for researchers to explore. However, such data may macroeconomic context, corrected for inflation, normalized, and
be collected in a “home-grown” manner, lacking a standard for- finally purged of incomplete or redundant entries.
mat across different companies 共Mitchell 1998兲. Data from a For the purpose of this paper, archival data refers to data that
single source can still require substantial preparation to be usable, were created before being used for research or that are collected
especially if they are paper based. Since quantifying economic
concurrently but independent of the purpose of the research. The
phenomena is crucial to stay competitive in today’s industry and
type or contents of the data and the method of collection can vary
years of archival data can be at the researcher’s disposal, the
widely. Archival research can be employed for many research
challenge is to devise scientifically valid ways to compare “apples
questions. For the purpose of illustration, the writers present prac-
to apples.”
Both writers have extensive experience in preparing archival tical examples of data preparation techniques from their research
economic data. Mitchell 共1998兲 focused on repair costs of con- studies in construction equipment economics, but are hopeful that
struction equipment. Data were accumulated in various media these subject-unspecific techniques will assist researchers on
from four heavy/highway construction firms whose size, fleet, and other topics as well. Data preparation refers to systematically
computing infrastructure differed. The resolution ranged from de- collating and transforming unformatted, unconnected, or other-
tailed records of each individual repair to periodically aggregated wise initially unusable data into a consistent and coherent data set
that can be used for a specific analytical purpose. Yu 共2007兲 cited
earlier work that listed data cleaning, transformation, integration,
1
Assistant Professor, Director, Construction Engineering and Manage- reduction, and discretization as separate elements of data prepa-
ment Program, Dept. of Civil Engineering, The Catholic Univ. of ration. Rajagopalan and Isken 共2001兲 additionally noted that the
America, 620 Michigan Ave., NE, Washington, D.C. 20064 共correspond- “preparation involves enhancing and enriching” the data and de-
ing author兲. E-mail: [email protected] fined data quality to mean the “accuracy, relevance, complete-
2
Associate Professor, Chair, Dept. of Engineering, Univ. of Southern ness” of data with respect to their intended purpose, which was
Indian, Evansville, IN 47712. E-mail: [email protected] refined into dimensions including their accessibility, sample size,
Note. This manuscript was submitted on July 24, 2008; approved on
completeness, consistency, credibility, clarity, format, correctness,
April 3, 2009; published online on April 30, 2009. Discussion period
open until June 1, 2010; separate discussions must be submitted for indi- unbiasedness, and timeliness 共Pipino et al. 2002兲. Poor quality
vidual papers. This paper is part of the Journal of Construction Engi- data invalidate research studies 共De Veaux and Hand 2005兲 and
neering and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, “compromises decision making” in the business environment, a
ISSN 0733-9364/2010/1-49–57/$25.00. major competitive disadvantage 共Redman 1998兲.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 49

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Importance tical analysis, e.g., hypothesis testing or regression modeling.
Details on how to overcome problems in structure, format, label-
Zhang et al. 共2003兲 considered data preparation to be one of four ing, frequency, and other properties are illustrated with specific
steps of knowledge discovery, which they described as an itera- examples of economic data. The time and cost intensive effort of
tive process, alluding to it being part of the scientific method. developing and executing the data collection from scratch, which
They highlighted its fundamental importance by estimating “that could potentially take years to yield a sufficient sample size for
data cleaning and preparation takes approximately 80% of the analysis, is thus avoided. While the writers sought to address the
total data engineering effort. Data preparation is, therefore, a cru- broad range of problems that they encountered in their own re-
cial research topic. However, much work in the field of data min- search, there is no guarantee that this list is exhaustive or appli-
ing was built on the existence of quality data” 共Zhang et al. 2003. cable to all future studies. Considering that the characteristics of
Soibelman and Kim 共2002兲 cited earlier work with a somewhat existing data and the manners in which they can be examined are
different estimated effort yet drew the same conclusion: “60% of limited only by the researchers’ creativity, it is virtually impos-
the time goes into preparing the data for mining, thus highlighting sible to provide one general approach of data preparation that
the critical dependency on clean, relevant data. The actual mining would anticipate and mitigate all possible problems.
step typically constitutes about 10% of the overall effort. Thus,
the process of data preparation is one of the most important parts
of the entire process and one of the most time consuming and Data Mining
difficult.” It is further acknowledged that data preparation is in
fact indispensable, as hardly any archival data are directly usable Recent years have seen a tremendous growth in archival data
in an analysis. “One of the major obstacles to using organizational created through an increased use of information technologies in
data for mining and knowledge discovery is that, in most cases, it the business environment 共Rajagopalan and Isken 2001兲, particu-
关sic兴 is not amenable for mining in its natural form” 共Rajagopalan larly due to less expensive hardware, e.g., sensors and storage
and Isken 2001. Due to the essential nature of data preparation, media 共Soibelman and Kim 2002兲. This vast pool of data holds
research has not only sought to advance techniques, as is de- valuable knowledge for whose extraction scientific principles
scribed in “Data Mining” and the remainder of this paper, but has must be applied. This process is known in the literature as knowl-
also studied its positive impact on knowledge discovery in vari- edge discovery in databases 共Ng and Soibelman 2003兲 as a pro-
ous domains 共Zhang et al. 2003; Rajagopalan and Isken 2001兲. cess following knowledge storing and sharing 共Rajagopalan and
Isken 2001兲, or more specifically as data mining 共Stegemann and
Buenfeld 2004兲, which refers to the actual process of extracting
Applicability
trends or patterns 共Zhang et al. 2003兲. The objective of data min-
Types of construction engineering and management studies for ing is to enable extracting rules on clustering, classification, as-
which the data preparation techniques described in this paper can sociation 共e.g., linear regression兲, sequence, or time series 共Lee et
be useful include, but are not limited to, further research on al. 2008兲. Its application can confirm, refine, or expand existing
equipment economics 共Fan et al. 2008, 2007兲, productivity studies knowledge in a particular domain. Examples abound, e.g., in
based on field records such as schedules, logs, and other routine health care 共Goodall 1999兲, market research 共Hand et al. 2000兲,
records 共Thomas and Horman 2006; Thomas 2000兲 or based on aviation safety 共Jeske and Liu 2007兲, and prediction of hurricane
macroeconomic measures and industrywide cost records 共Rojas damages 共Nawari 2008兲. Engineering applications include pave-
and Aramvareekul 2003; Allmon et al. 2000兲, safety studies based ment management 共Attoh-Okine 1997兲, RC design 共Roddis and
on accident reports, databases of industry statistics, and hazard Zhang 2000兲, water reservoirs 共Bessler et al. 2003兲, contaminants
analyses 共Carter and Smith 2006; Arboleda and Abraham 2004; in concrete 共Stegemann and Buenfeld 2004兲, traffic flows 共Amado
Huang and Hinze 2003兲, metastudies that evaluate large quantities and Virkler 2006兲, and many others 共Fayyad and Smyth 1999兲. In
of previously published studies and possibly their data sets to construction engineering and management, knowledge discovery
explore the breadth and depth of knowledge on a topic 共Abu- has been used, e.g., to classify construction documents 共Caldas
dayyeh et al. 2006, 2004; Pietroforte and Stefani 2004; Chevallier and Soibelman 2002兲 and to support facilities maintenance 共Ng
and Russell 1998兲, data mining and sociology-type analyses of and Soibelman 2003兲 through mining text data. A recent study on
correspondence, internal memoranda, site diaries, and other writ- the residual value of heavy equipment 共Fan et al. 2008兲 continued
ten records of construction projects 共Zhu et al. 2007; Caldas and the earlier work of the writers with an autoregressive tree ap-
Soibelman 2006; Hajjar and AbouRizk 2000兲, and any other types proach. Current research continues to expand the analytical ap-
of studies that may in part or fully rely on archival records. Ad- proaches of data mining, e.g., by using neural networks, fuzzy
ditionally, studies that collect new data with paper records, espe- logic, and genetic algorithms 共Yu 2007兲. All studies had in com-
cially survey questionnaires that, e.g., explore the productivity or mon that they analyzed very large data sets whose imperfections
site safety 共Ezeldin and Sharara 2006; Zayed and Halpin 2004; required manipulation for data preparation.
Mohamed 2002兲 or the management or worker attitudes and mo-
tivations 共Cox et al. 2006; Ling 2002; Chang 2001兲 can also ben-
efit from exercising the cautious approach to ensuring data quality General Considerations
that is advocated and illustrated by this paper.
Scholarly publications typically contain a section on how data Central questions for the researcher at the onset of a study include
were collected 共Rojas and Kell 2008兲, but not necessarily describe the type, range, and source of the necessary data. It is assumed
what practical problems were encountered. The particular contri- that the researcher has already evaluated the advantages and dis-
bution of this paper is therefore to assist future researchers at the advantages of using archival data from one or several sources
critical interface between data collection and analysis, both of instead of collecting new observations in the field or the labora-
which are fundamental steps within the scientific method. It de- tory. This is not an easy decision. The researcher has great control
scribes successful preparation techniques for a meaningful statis- over the collection methodology for new data, but no influence on

50 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
how the existing data were created. Data from field sources are provide ideal realizations of economic value. Auction purchase
more prone to containing irregularities or noise, which can be prices are far superior to mere sales offers, which only reflect the
somewhat dampened by their potentially larger volume. Yet labo- expectation of a seller without ever having been substantiated in
ratory studies are often infeasible or impossible for examining the market. In some cases, their use is unavoidable, e.g., for nor-
economic phenomena due to their high cost, lengthy duration, and malizing the residual value with the manufacturers’ suggested re-
limited range. The challenge to extract quality data thus emerges. tail price, or list price, due to unavailability of actual purchase
Several questions need to be answered satisfactorily regarding the prices as is described next.
existing data 共Rajagopalan and Isken 2001兲: Besides comparison with many similar sales, other valuation
• Which data are required and which are available?
approaches exist and are used in various industries, e.g., consid-
• What are the type, range, and source共s兲 of the data?
ering the expected future income that can be generated by an
• Was the collection methodology of the data adequate?
asset. Some confusion is introduced by the concept of deprecia-
• How can incongruous data be matched consistently?
• How can various sources be merged into one data set? tion, which typically refers to a loss in book value for taxing a
• How are and should the data be formatted and labeled? capital investment with simplified accounting methods, e.g.,
• How can errors in the data set be detected and corrected? straight-line or declining balance, which may not reflect its actual
physical deterioration 共Perry et al. 1990兲. Mitchell 共1998兲 found
that companies were charged different amounts for the initial pur-
Variables and Sample Sizes chase of the same type of construction equipment. Trade-ins and
lease allowances clouded the valuation. The list price was used as
Even before existing data are compiled, the researcher must de- the initial value to mitigate this, regardless of what companies
cide what will constitute the “input” and “output” of the planned paid. This standardization was feasible because the study focused
model. The response, or dependent variable, is predicted from one on predicting recurring repair costs.
or several explanatory, or independent variables. Variables that
are not relevant to the study or that lack data points at hand
should not be included 共Soibelman and Kim 2002兲. Before the Recurring Cost Data
analysis it is unknown which ones of the possible explanatory Many different types of expenses can be included in economic
variables will contribute to the final regression model in a statis-
models. They can range from straightforward and tangible ex-
tically significant way. The number of explanatory variables
penses, e.g., fuel, to complicated and intangible expenses, e.g.,
should thus initially be kept as large as possible. Furthermore, the
the cost of obsolescence. Expenses can be broken down into three
central limit theorem dictates that a larger sample size should
yield a greater precision of the predictions that will be made with broad categories: Direct, provisional, and collateral costs. The
the model. The reason for this is that to accurately reflect the researcher must decide which types of costs the economic study
variability of the real phenomenon, the model needs to be built will address and remove any superfluous existing data from the
from data that contain a sufficient amount of such variability in all data set before analysis.
of their variables, as sparse data sets yield only insights of limited Direct costs are quantifiable, clear, and directly related to own-
validity 共Stegemann and Buenfeld 2004兲. Statistical equations are ing, operating, and maintaining an asset. They occur regularly and
provided in the literature 共e.g., Green 1991兲 to determine how predictably in a given accounting period and affect a company’s
many samples from the population are sufficient, depending on operating budget. They are offset by the revenue stream generated
the desired confidence level and on the number of explanatory by the asset. Examples of direct expenses for companies that op-
variables in the model. Acceptable and expected ranges of values erate heavy equipment include fuel, oil, tracks or tires, mainte-
for each variable should be determined a priori by the researcher, nance, repairs, financing interest and principal payments, taxes,
which defines the scope of the study. For example, the residual licenses, and insurance.
value study covered only common types of construction equip- Provisional costs are internal costs that are intended to cover
ment of up to 15 years of age to ensure a broad applicability of its the anticipated costs of continuous processes that are difficult to
results. Sample sizes outside this range were too small to be valid. quantify or discrete events that only occur a limited number of
Within each data point, the type, make, model, serial number, times during the life of the asset. For example, the costs of major
year of manufacture, and condition rating were reported, along repairs or rebuilds of a unit of equipment are handled by charging
with the auction firm, location, date, price, and a brief description a provisional hourly rate to build a repair reserve that is balanced
of the setup of the machine 共Lucko 2003兲.
across a particular unit, group, or fleet. Depreciation is another
type of provisional cost, which allows for the fact that the value
Evidences of Value of an asset decreases with the passage of time alone.
Collateral costs are more difficult to quantify and not always
Valuation is an important topic in economic studies and provides
a key input for investment decisions. Value is a somewhat elusive part of economic models in the literature, nor used much by man-
concept that seeks to indicate how much a good or service could agers in the industry practice. They can include obsolescence
be sold for in the market at a particular time. The “fair market costs, associated resource impact costs, lack of readiness costs,
value” is the price could be realized in an open transaction in a service level impact costs, and alternative method impact costs
competitive market between an equally informed and voluntarily 共Vorster and de la Garza 1990兲. Obsolescence costs are incurred
acting buyer and seller 共Lucko 2003兲. Valuation, or appraisal, is as an asset “ages” technologically versus new technologies that
only useful if it can occur without having to actually sell the item. provide increased productivity, reliability, or versatility. They can
Therefore, the data on which it is based must be “evidences of take the form of higher repair or production costs or may appear
value” 共Cowles and Elfar 1977兲, i.e., actual sales. According to as bids that are lost because of the then-higher costs of assets with
these criteria, public auctions with full information disclosure the old technology.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 51

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Data Sources be adjusted, averaged, or interpolated, e.g., to create an average
annual producer price index 共PPI兲 from monthly values to match
the years in which equipment age was measured. Another impor-
Proprietary Data
tant consideration is the data resolution, i.e., the significant digits
In the highly competitive environment of the construction indus- in their values, which analogously determines the possible nu-
try, contracts can be won or lost by very small margins 共Liu and merical accuracy of the response variable.
Ling 2005兲. Although construction firms often are willing to co- The equipment age was an explanatory variable in the repair
operate with researchers who develop and test theories to help cost study. It could be measured in units ranging from calendar
them improve their business, they are disinclined to share their age in years to cumulative usage in hours, the latter of which was
economic data in a form by which competitors might gain an chosen 共Mitchell 1998兲. One source presented the severe problem
advantage. Proprietary corporate data therefore require written that the explanatory and response variables were recorded in
permission to be usable for noncommercial scholarly purposes. separate databases at different frequencies; meter hour readings
Such confidentiality agreement regulates disclosing planned pub- were recorded whenever an oil change occurred but repair costs
lications to the source. Specifically, it should clearly establish were recorded with a lower frequency at the end of each month.
what type and extent of raw, aggregated, or processed data may These disparate data were matched via calendar date based on the
be published. If raw data are strongly protected, descriptions and assumption that events on or before the 15th day of a month could
illustrations can even use artificial data whose statistical proper- be grouped with its beginning and events after the 15th day with
ties are similar to the originals. The agreement should also permit its end, which may have introduced small errors but made the
discussing scientific questions of the project with other research- overall study possible. Repair costs that were associated with cu-
ers. Anonymity of participating companies or individuals and pro- mulative hours from oil changes early in a month may have been
tection of the raw data by limiting access and destroying them understated and the costs associated with later oil changes may
after a specified time are other typical precautions. have been overstated. Fortunately, the errors would be offsetting
in the long run if oil changes occurred approximately evenly dis-
Multiple Sources tributed over time. The matching process ensured that exactly one
data pair existed for the lowest data frequency, in this case
Ideally, economic data should be collected from several sources months. Other sources provided multiple data points for the same
to allow validating the order of magnitude, mean, and variance of cumulative hours of usage if equipment was idle for long periods.
the data and enable crosscomparisons that can include demo- All but the first value of multiple occurrences of the same data
graphic data about each source. However, data sources vary in were therefore eliminated, which was subsequently matched with
accuracy and reliability depending on their type and internal pro- repair costs as described previously.
cedures. Each additional source introduces new biases and pos-
sible errors. Governmental sources are generally considered of
high quality are consist data in a consistent manner while corpo- Storage Medium
rate sources often exhibit more incongruities. Different sources The media in which economic data are recorded range from in-
must be coded in the final data set, e.g., with integers or time formal papers via electronic office files to commercial data re-
stamps, to allow statistical testing for differences. While survey- positories. Actual paper records are vulnerable to transcription
ing the extent of the available data sets, it is important to ascertain errors that may occur during their initial recording or during ex-
if pertinent data fields are common and complete across all traction for research use. To be usable for analysis, data must be
sources. If an individual data set contains less than all of the carefully converted into an electronic format by manual transcrip-
attributes to address the research question it may need to be elimi- tion or by employing optical character recognition software. Ex-
nated, however extensive, unless the missing entries can be re- isting electronic data may be readily accessible on PC or may be
constructed in a valid manner. retrieved off central mainframe computers or distributed networks
with data mining software. Their quality must still be verified,
e.g., using various sorting functions to identify any fields that
Data Editing contain missing or impossible entries.
Spreadsheets are common for storing economic data and can
often be exported directly from local database software or online
Frequency and Resolution
databases such as Last Bid, formerly Green Guide Auction Re-
It is essential to understand the mechanism by which economic port, and Top Bid that cover most heavy equipment auctions but
data were collected and recorded, e.g., from technical descriptions require a subscription fee and various governmental sources of
or interviews with the respective staff. The terminology and defi- macroeconomic indicators that generally are free of charge.
nitions of what constitutes a data point depends on the internal Manufacturers and distributors provide additional valuable data,
procedures of a source. Data may be either continuous or discrete e.g., price lists, on paper, in text or spreadsheet files, Adobe por-
with a restricted or unlimited range of possible values. Since ex- table document format, or embedded in Web sites. These must be
isting data are not specifically collected for the purpose of the converted into a common medium and format with separators
particular study, their frequency and resolution may be inconsis- between each entry, so that data cleaning techniques can subse-
tent between different sources. Corporate data may be measured quently be applied evenly to the consolidated final data set.
ad hoc when a particular event occurs, e.g., an equipment repair. The data in the repair cost study initially were in paper
Macroeconomic data on the other hand often have a monthly or records, various PC files, and mainframe computers. PC and da-
quarterly frequency. Different frequencies that can span from cost tabase files required only little preparation by converting them
items of an individual unit of equipment to the economy at large into formats such as Microsoft Access or Corel Paradox into
must be matched in a valid response variable. The largest interval spreadsheets, or by only adjusting the layout if data were already
determines the frequency to which data of higher frequency must in Microsoft Excel or Corel Quattro format. Converting database

52 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
files was faster than PC files because queries extracted the data numerical data it is useful to add column headers that calculate
into the layout as desired for analysis. minimum, maximum, mean, and variance in the spreadsheet to
Preparing mainframe data can be more challenging because quickly gauge their range, centricity, and spread. Two types of
they may be housed in applications that were programmed spe- accounting errors had to be corrected in the repair cost study.
cifically for a company. Transferring them to unformatted Ameri- Most common were negative repair costs. Closer investigation
can Standard Code for Information Interchange 共ASCII兲 files is a with the respective companies showed that negative charges were
suitable solution 共Rajagopalan and Isken 2001兲 but adds the data due to either overcharges or mistakes that had occurred in an
preparation step of separating one row of unformatted entries. earlier month and were removed. Another error occurred due to
Converters within spreadsheet programs allow parsing data but replaced hour meters in equipment, which caused conspicuous
work only if they were originally in tabular format. Moreover, sudden decreases in cumulative hours of usage over time or non-
pertinent data may be disjointed across several mainframe reports, zero cost at time zero. After confirming that a meter had been
e.g., for the same unit of equipment. Data mining software such exchanged, the cumulative hours after replacement were cor-
as, e.g., DataWatch Monarch can help structuring data through rected to being continuous or the data point was removed entirely
creating custom templates and provides graphical reports. For ex- if the value directly before replacement was not known. Whatever
ample, cost data were extracted, filtered to include only repair the exact nature of the economic data, introducing checks and
costs, associated with a specific unit of equipment, subtotaled by balances that verify the numerical integrity in terms of dimension,
cost code for each month, and then exported. sign, sequence, intervals, and other parameters within or between
Paper records prove to be the hardest to prepare for a mean- entries can significantly enhance the data quality.
ingful analysis in the experience of the writers. Making them
usable is a labor-intensive process that is susceptible to transcrip-
Typographical Errors
tion, transposition, omission, and structural errors if files are il-
legible, unorganized, or incomplete. When the preferred Visually skimming the entire sorted data set for deviations can
electronic data did not exist for the repair cost study, manual help identifying and eliminating typographical errors. Lucko
transcription of loose leaf repair receipts into a spreadsheet and 共2003兲 was able to identify and correct errors that occurred be-
subsequent proofreading were required. tween similar looking letters and digits while transcribing the
handwritten records into electronic form:
• Letter or digit switches: 7XM instead of 7MX within a serial
Formatting and Labeling
number.
Individual data points typically consist of several columns. Since • Letter or digit changes: 1 = I, 2 = Z, 3 = 6 = 8 = B = S, 7 = F = T.
terminology for economic data are determined by the source, it is Others were spelling variations in alphanumeric entries, e.g.,
recommended that a consistent and intuitive labeling is applied to the model name of the equipment:
entire final data set during data preparation. Spreadsheet software • Incomplete model name: Excavator PC300 instead of
allows designating cells as text, numbers, dates, or prices accord- PC300HD-5.
ing to their contents. Superfluous formatting, e.g., printer symbols • Incomplete series name: Excavator PC100 II instead of
共Rajagopalan and Isken 2001兲, blank spaces, and tabulator stops PC100C Series II.
can be removed with the editing functions of text processing soft- • Missing hyphen: Excavator PW301 instead of PW30-1.
ware. An efficient method of removing formatting is pasting data Long numbers may contain superfluous initial zeros that can
as unformatted text into spreadsheet software, which unifies all of be removed. Finally, since serial numbers are awarded consecu-
their fonts. tively to newly manufactured equipment, sudden deviations in the
Sorting the columns of Lucko 共2003兲 residual value data al- year of manufacture between otherwise constant or sequential en-
phabetically brought blank cells and those containing symbols, tries were corrected to preserve the correct time sequence 共Raja-
e.g., “N/A” for not applicable or “-,” to the top, which were filled gopalan and Isken 2001兲. Correct spelling and grammatical
with “.” This symbol is commonly used in statistics software to variations need to be ascertained especially for studies that mine
represent missing values. Some data had to be calculated in addi- text data 共Jeske and Liu 2007兲.
tional columns, e.g., age as the difference between the year of
manufacture and the date of the auction at which the equipment
was sold. Age only had an accuracy of years, because the day or Conversion Process
month of manufacture was not known. The column of setup de-
scriptions was scanned for negations, e.g., “no,” “not,” and “in-
operable,” to isolate entries of equipment that was lacking parts. Reconstruction
Location-related data, e.g., postal ZIP codes and state name ab- Data sets need to be checked for completeness, as missing data
breviations, were converted by simple IF commands in the severely reduce the data quality and curtail the attainability of
spreadsheet into geographic regions that served as an explanatory research objectives 共Stegemann and Buenfeld 2004兲. If the overall
variable in the statistical model. For large data sets it is recom- data set contains redundancy or a predictable pattern, it is possible
mended to add a control column that automatically verifies the to fill gaps by carefully comparing the incomplete entry with
correct conversion of each entry with an adequate checksum, neighboring ones, provided that no data are fabricated a poste-
which can additionally be formatted to be easily visible. riori. Only in rare cases do data sets overlap to a significant de-
gree, such as for the equipment auction data from Last Bid and
Dimension and Sign Top Bid. They provided two independent records of the same
auctions at which a particular unit of heavy equipment had been
Sorting data numerically can reveal unreasonable extremes sold. Redundancy between the two data sets allowed verification
caused by partially complete entries, e.g., negative 共199_–1996兲 by comparing pairs of entries via unique identifiers such as date,
or large ages 共2002–199_兲 that must be corrected or deleted. For location, and serial number. Merging nonredundant entries can

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 53

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
increase the sample size, but should be verified to match the qual- Table 1. Conversion of Verbal Descriptors to Binary Numbers
ity of redundant ones. Machines appeared in more than one data Binary number
point if they were sold several times during their economic life. It
was assumed that such multiple entries were independent and Descriptor Number n1 n2 n3
could therefore be retained in the overall data set. Reconstruction A 1 0 0 1
required that all data points in the merged data set were sorted in B 2 0 1 0
the hierarchical order of model, serial number, and auction date. C 3 0 1 1
A computer macro FillGaps automated this process. If a cell con- D 4 1 0 0
tained “.,” then the macro matched several identifiers with pre-
ceding or succeeding entries to find redundancy, i.e., two records
of the same auction sale. The placeholder was then overwritten tween categories, so that either all or none should be included in
with the correct value from the adjacent entry as per the a statistical model. A drawback is that the conversion removes
pseudocode: any actual hierarchical information that may have been implied in
the integers.
//FillGaps 关ArrayEqData (Row, Column)]
for each Row in EqData
do if 共Row, Column1兲 = “ . ” Economic Indicators
then if 共Row, Column2兲. Macroeconomic phenomena are created by complex interactions
Date= 共PreviousRow, Column2兲.Date of many participants. Indicators attempt to capture one specific
and 共Row, Column3兲 = 共PreviousRow, Column3兲 aspect of a local, regional, national, or global economic situation
and 共Row, Column4兲 = 共PreviousRow, Column4兲 in a numeric measure. Vast amounts of data are available from
then 共Row, Column1兲 ← 共PreviousRow, Column1兲 government agencies, but also from financial news services, cor-
else if 共Row, Column2兲. porate publications, and independent research organizations, e.g.,
Date= 共NextRow, Column2兲.Date for business cycle indicators. They can provide useful explanatory
and 共Row, Column3兲 = 共NextRow, Column3兲 variables of the economic context within which construction com-
and 共Row, Column4兲 = 共NextRow, Column4兲 panies acted. But it is probable that they are correlated if they
then 共Row, Column1兲 ← 共NextRow, Column1兲 measure similar aspects of the economy 共Perry et al. 1990兲, e.g.,
inflation indicators, or the cost indices by Engineering News
A variation of this technique also identified the aforementioned Record magazine. It is necessary to determine which ones con-
sudden changes in an otherwise consecutive sequence. Remaining tribute significantly to the predictive power of the statistical
redundant entries had to be deleted once gaps were filled. A macro model. Numerous pairs of economic indicators have very high
DeleteDoubles—similar to the previous one—compared Pearson coefficients of correlation Rcorr, e.g., the building and
identifiers, allowing for a small variability between prices due to construction cost indices, gross domestic product and total retail
currency conversion, and deleted the first entry of each pair. sales, consumer price index and PPI for machinery and equip-
ment, and others for values from January 1980 to August 2002.
Lower frequencies were matched with higher ones by assuming
Verbal Descriptors them to be piecewise constant. This slightly decreased the corre-
A problem that is often encountered in existing data sets is that lation but allowed using all data points. Some indicators are avail-
they contain explanatory variables that consist of verbal descrip- able in regular and seasonally adjusted form to remove recurring
tors. All explanatory variables must be expressed in numerical swings from actual economic trends. The Bureau of the Census
terms to be usable in a statistical analysis. For example, the manu- provides, e.g., seasonally adjusted annual rates, which allows di-
facturer name, condition rating, and auction region were categori- rect comparisons between monthly, quarterly, and annual eco-
cal variables in the residual value study. Some are hierarchical, nomic values. Consistency in selecting seasonally adjusted or
e.g., condition 共excellent, very good, good, fair, poor兲; others are unadjusted indicators is recommended.
not, e.g., region 共northeast, south, midwest, west兲. Categorical
variables often serve as somewhat subjective proxies for model- Inflation Correction
ing complex phenomena. Condition is typically appraised visually
using a checklist and aggregates the result of wear and tear on The phenomenon of inflation is an imbalance between the supply
different parts of a machine, e.g., tires or tracks, undercarriage, of money and the supply of goods and services 共Bodie et al.
and engine, which is offset by maintenance and repairs. If at all 2002兲. Manufacturers’ suggested retail prices, auction prices,
possible, verbal descriptors should be systematically replaced maintenance and repair costs, and any other cost data must be
with meaningful numbers 共Soibelman and Kim 2002兲. However, corrected to an arbitrary common date before they can be ana-
simply assigning integers would only be permissible for a truly lyzed. Inflation adjustments can be made with an annual percent-
hierarchical phenomenon and yet would still be statistically prob- age rate, which creates an exponential growth. If data are
lematic, as it would assume a constant distance between catego- insufficient or unavailable to estimate such rate accurately, using
ries. governmental indices as per Eq. 共1兲 is a widely accepted tech-
A valid solution is to convert the integer into binary numbers nique
that serve as sets of “indicator variables.” Table 1 shows how four Index2
verbal descriptors are represented by an equivalent triplet of zeros Price2 = Price1 · 共1兲
Index1
and ones. Numbering begins with one, because starting at zero
would slightly favor the first category and de facto consider it the where Price= any price in U.S. dollars and Index= price index.
benchmark for any further variability in the particular explanatory They are ratios of current to past prices for standardized baskets
variable. Each binary provides only part of the distinction be- of goods or services. Indices with different levels of detail and

54 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
composition are available. Some apply to the entire economy; rating. It is recommended to create data sets that are as homog-
others are regional or cover only specific industries, commodities, enous as possible as long as the required sample size suffices.
and stages of processing. The PPI for finished goods was used to Predictive equations then seek to quantify the relationship be-
adjust costs related to heavy equipment 共Cross and Perry 1995, tween the response variable and the attributes. For example,
1996; Kastens 1997兲. Once an index has been used to correct for Mitchell 共1998兲 compared unlike machines by indexing their re-
inflation, it cannot be an explanatory variable. Otherwise, the pair costs to list prices with a cumulative cost index 共CCI兲 as per
model would suffer from multicollinearity, i.e., a very high cor- Eq. 共2兲
relation between them, which precludes obtaining a closed-form
solution. Ideally, all explanatory variables, including economic t
indicators, should thus be only weakly correlated. A more com- PP0 + 兺共Pt + Lt + Ot兲
0
plex method for calculating a composite price index was proposed CCIt = 共2兲
by Douglas 共1975兲. If specific indices can be applied to specific PP0
categories within the data, such weighted index can be beneficial. where CCI= cumulative cost index; P = cost of parts; L = cost of
For example, the not seasonally adjusted annual PPI for all fin- labor; O = other maintenance costs at time t; and PP0 = list price.
ished goods grew from 2002 to 2007 by 19.9%, 27.2% for mate- All of these cost items were cumulative.
rials and components for construction, and even 74.5% for steel
mill products 共Bureau of Labor Statistics 2008兲. A researcher fo-
cusing on steel-related construction would be ill advised to use
Statistical Preparation
the general PPI under this situation.

Outliers
Matching Data
Data may take on extreme values due to measurement or record-
If existing data are merged with other data, the question of how to ing errors. It is therefore prudent to purge the final data set of such
efficiently match them arises. For example, the auction records in outliers 共Soibelman and Kim 2002兲. They are defined as data
the residual value study needed to be categorized by adding size points that differ significantly from the basic relationship captured
parameters to the data set, normalized by adding list prices, cor- by the other data, either in their sign or magnitude, and can sub-
rected for inflation, and matched with economic indicators. If data stantially distort regression models. A manual technique for iden-
need classification, the categories should reflect common features tifying them is to graph all data in one or several scatterplots, but
of the item of interest, e.g., footprint area or number of floors for various statistical techniques also exist, e.g., residuals that mea-
residential construction. Equipment can be characterized based on sure how much a data point deviates from the model. It is recom-
its performance or capacity, e.g., standard operating weight mended to use scaled residuals due to their constant variance
共empty兲, general purpose bucket volume, and net horse power 共Montgomery et al. 2001兲. Their absolute value then determines if
共flywheel兲. A catalog of size parameters was assembled from data a data point is earmarked for removal, which can be implemented
by manufacturers and their distributors. Minor conversions and with an IF command in an additional column of the spreadsheet.
rounding between English and metric units were performed. The threshold should be set depending on the characteristics of
Rather than introducing an explanatory variable for size, a sepa- the data set in consultation with a statistician. After outliers are
rate model was created for each smaller, more consistent category, identified, it is best to try to identify the reason for the deviation.
which significantly improved the goodness-of-fit. A computer The statistical model then contains only coefficients that were
macro MatchData read unique identifiers in each row of the ex- calculated from the “cleaned” final data set.
isting data, looped through a given block of new data, and in case
of a match wrote the new data next to the existing. For list prices
and sizes, the identifier was the model name; for economic indi- Relative Dominance
cators it was the auction date. A statistical issue that manifested itself in the repair cost data is
an uneven distribution of data points between individual eco-
Normalization nomic assets that together compose a data set. Some machines
were represented with significantly more data points than others.
Normalizing variables by dividing them by a baseline value Reasons for this deviation included different usages but also the
makes them comparable across categories. Dollar values in the data collection style of the individual sources. Dominant ma-
residual value study were divided by manufacturers’ suggested chines would have had more of an influence on the regression
retail prices, i.e., list prices. Actual initial sales prices would have analysis than those with fewer data points relative dominance can
been ideal, but in their absence for proprietary reasons, list prices be addressed by interpolating data to discrete, evenly spaced in-
were assumed to be generated in a consistent manner across tervals, which are chosen to match the data at hand, e.g., 500 h of
manufacturers. They are devoid of discounts that specific compa- usage in the repair cost study. It is important to emphasize is that
nies may receive and unbiased, unlike purchase prices. Collected only one data point should be interpolated between any two actual
in analogy to the size parameters, list prices were interpolated ones, as otherwise the statistical integrity of the data set would
across gaps and, if necessary, extrapolated with an inflation cor- suffer from fabricating data.
rection before the matching.
Some economic research may investigate assets without iden-
Variable Selection
tical attributes, but costs can vary strongly depending on these
attributes. For example, units of construction equipment can vary Note that for observational studies, including archival research
in many regards. Differences between the machines may occur in that uses previously existing data from construction companies,
physical setup and in their usage, the latter of which are harder to no direct causality can be established statistically. Nonetheless,
discern and may be realized in a proxy measure, i.e., a condition carefully selecting the components and structure allows creating

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 55

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
predictive models with high confidence levels. Techniques for dential data, unifying the format, reconstructing missing values as
variable selection start the analysis phase after data of possible far as possible, checking the data set for various types of incon-
explanatory variables have been collected with a sufficient sample sistencies, matching incongruous data with each other, perform-
size and prepared. Pairs of variables 共Stegemann and Buenfeld ing an inflation adjustment on economic data, removing outliers,
2004兲, e.g., two explanatory variables or one explanatory and one and selecting what explanatory variables are included in the
response variable, are visually inspected in scatterplots to identify model. While each new study of archival data will present the
multicollinearity and any trends or patterns. Multiple explanatory researcher with new and somewhat unique problems, these gen-
variables yield a triangular matrix of scatterplots. These observa- eral techniques are hoped to serve as guidelines and as a source of
tions assist in determining the model composition, i.e., what inspiration in surmounting the crucial and often underestimated
mathematical function共s兲 to use, what order of variable terms, and phase of data preparation, which alone enables the subsequent
what possible interaction terms between them, if any. Plotting analysis and determines its quality and success.
data or derivatives thereof, e.g., the normal probability plot of the
residuals, also yields information whether important regression
assumptions are fulfilled by the data set at hand.
Acknowledgments
Both overfitting and underfitting a model with explanatory
variables reduces its predictive power. Three techniques are there-
The first writer thanks Joseph D. Lombardo of Learning Seed and
fore commonly used to select appropriate variables for being in-
Justin P. Molineaux of Computech for their advice on creating
cluded in a regression model, forward selection, backward
effective pseudocode.
elimination, and stepwise selection 共Montgomery et al. 2001兲.
Forward selection starts with an “empty” model, tests which vari-
able is most significant if added, adds it, and proceeds with testing
the remaining variables. Backward elimination is the reverse pro- References
cess that sequentially removes the least significant variables from
a “full” model and can assist in initially quickly discarding un- Abudayyeh, O., Dibert-DeYoung, A., and Jaselskis, E. J. 共2004兲. “Analy-
wanted variables. Stepwise selection extends forward selection sis of trends in construction research: 1985–2002.” J. Constr. Eng.
Manage., 130共3兲, 433–439.
and alternates between adding and removing to arrive at an over-
Abudayyeh, O., Dibert-DeYoung, A., Rasdorf, W. J., and Melhem, H.
all best-fitted model. While the statistical modeling can be aided 共2006兲. “Research publication trends and topics in computing in civil
additionally with an autoregression algorithm that creates and engineering.” J. Comput. Civ. Eng., 20共1兲, 2–12.
tests many possible regressions models 共Fan et al. 2008兲, it ulti- Allmon, E., Haas, C. T., Borcherding, J. D., and Goodrum, P. M. 共2000兲.
mately still depends on the conscious, experienced researcher and “U.S. construction labor productivity trends, 1970–1998.” J. Constr.
remains both an art and a science. Eng. Manage., 126共2兲, 97–104.
Amado, V., and Virkler, M. R. 共2006兲. “Using data mining to analyze
archived traffic related data.” Proc., 2006 9th Int. Conf. on Applica-
Outlook
tions of Advanced Technology in Transportation, K. C. P. Wang, B. L.
Messner 共2003兲 suggested organizing corporate data based on the Smith, D. R. Uzarski, and S. C. Wong, eds., ASCE, Reston, Va.,
organization, its processes and commitments, its facilities, and the 310–318.
environment. An accessible and well-organized data structure that Arboleda, C. A., and Abraham, D. M. 共2004兲. “Fatalities in trenching
operations—Analysis using models of accident causation.” J. Constr.
considers these recommendations facilitates their analysis at any
Eng. Manage., 130共2兲, 273–280.
point in time, including a posteriori research studies even and Attoh-Okine, N. O. 共1997兲. “Rough set application to data mining prin-
especially on to glean insights on aspects that are not part of the ciples in pavement management database.” J. Comput. Civ. Eng.,
routine daily business functions and their reporting requirements. 11共4兲, 231–237.
While researchers may have the time and training to transform Bessler, F. T., Savic, D. A., and Walters, G. A. 共2003兲. “Water reservoir
unsuitable data sets with the data preparation techniques, employ- control with data mining.” J. Water Resour. Plann. Manage., 129共1兲,
ees of construction companies may find it challenging to perform 26–34.
the necessary steps on initially unusable archival data. Addition- Bodie, Z., Kane, A., and Marcus, A. J. 共2002兲. Investments, 5th Ed.,
ally, pending the adoption of new data management systems, e.g., McGraw-Hill, New York.
accounting software, companies are likely to continue collecting Bureau of Labor Statistics. 共2008兲. “Producer price indexes: Databases,
data in the same way as they have before. Awareness is therefore tables & calculators by subject.” U.S. Dept. of Labor, 具http://www.
bls.gov/data典 共July 24, 2008兲.
called to the fact that data can be a singularly valuable pool of
Caldas, C. H., and Soibelman, L. 共2002兲. “Automated classification of
knowledge that is yet to be extracted by data mining techniques. construction project documents.” J. Comput. Civ. Eng., 16共4兲, 234–
The effort of setting up a data collection and storage system is 243.
therefore an investment whose future payoff in terms of savings Caldas, C. H., and Soibelman, L. 共2006兲. “A combined text mining
or productivity improvements is still unknown. In the meantime, method to improve document management in construction projects.”
the various data preparation techniques that this paper has de- Proc., 2006 Int. Conf. on Computing in Civil Engineering of ASCE, H.
scribed are hoped to serve as useful guidelines. Rivard, H. Melhem, and E. Miresco, eds., ASCE, Reston, Va., 2912–
2918.
Carter, G., and Smith, S. D. 共2006兲. “Safety hazard identification on
construction projects.” J. Constr. Eng. Manage., 132共2兲, 197–205.
Conclusions
Chang, S.-T. 共2001兲. “Work-time model for engineers.” J. Constr. Eng.
Manage., 127共2兲, 163–172.
This paper has described how different types of existing archival Chevallier, N., and Russell, A. D. 共1998兲. “Automated schedule genera-
data in the area of construction engineering and management can tion.” Can. J. Civ. Eng., 25共6兲, 1059–1077.
be prepared in a valid manner while preserving their integrity. Cowles, H. A., and Elfar, A. A. 共1977兲. “Valuation of industrial property:
Preparations included validating the sample size, handling confi- A proposed model.” Eng. Econ., 23共3兲, 141–161.

56 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Cox, R. F., Issa, R. A., and Frey, A. 共2006兲. “Proposed subcontractor- dissertation, Virginia Polytechnic Institute and State Univ., Blacks-
based employee motivational model.” J. Constr. Eng. Manage., burg, Va.
132共2兲, 152–163. Mohamed, S. 共2002兲. “Safety climate in construction site environments.”
Cross, T. L., and Perry, G. M. 共1995兲. “Depreciation patterns for agricul- J. Constr. Eng. Manage., 128共5兲, 375–384.
tural machinery.” Am. J. Agric. Econom., 77共1兲, 194–204. Montgomery, D. C., Peck, E. A., and Vining, G. G. 共2001兲. Introduction
Cross, T. L., and Perry, G. M. 共1996兲. “Remaining value functions for to linear regression analysis, 3rd Ed., Wiley, New York.
farm equipment.” Appl. Eng. Agric., 12共5兲, 547–553. Nawari, N. O. 共2008兲. “The role of data mining techniques in the predic-
De Veaux, R. D., and Hand, D. J. 共2005兲. “How to lie with bad data.” tion of hurricane damages.” Proc., 2008 Structures Congress, D.
Stat. Sci., 20共3兲, 231–238. Anderson, C. Ventura, D. Harvey, and M. Hoit, eds., ASCE, Reston,
Douglas, J. 共1975兲. Construction equipment policy, McGraw-Hill, New Va., 1–10.
York. Ng, H. S., and Soibelman, L. 共2003兲. “Knowledge discovery in mainte-
Ezeldin, A. S., and Sharara, L. M. 共2006兲. “Neural networks for estimat- nance databases: Enhancing the maintainability in higher education
ing the productivity of concreting activities.” J. Constr. Eng. Manage., facilities.” Proc., 2003 Construction Research Congress, K. R. Mo-
132共6兲, 650–656. lenaar and P. S. Chinowsky, eds., ASCE, Reston, Va.
Fan, H., AbouRizk, S. M., and Kim, H. 共2007兲. “Building intelligent Perry, G. M., Bayaner, A., and Nixon, C. J. 共1990兲. “The effect of usage
applications for construction equipment management.” Proc., 2007 and size on tractor depreciation.” Am. J. Agr. Econ., 72共2兲, 317–325.
ASCE Int. Workshop on Computing in Civil Engineering, L. Soibel- Pietroforte, R., and Stefani, T. P. 共2004兲. “ASCE Journal of Construction
man and B. Akinci, eds., ASCE, Reston, Va., 192–199. Engineering and Management: Review of the years 1983–2000.” J.
Fan, H., AbouRizk, S. M., Kim, H., and Zaïane, O. 共2008兲. “Assessing Constr. Eng. Manage., 130共3兲, 440–448.
residual value of heavy construction equipment using predictive data Pipino, L. L., Lee, Y. W., and Wang, R. Y. 共2002兲. “Data quality assess-
mining model.” J. Comput. Civ. Eng., 22共3兲, 181–191. ment.” Commun. ACM, 45共4兲, 211–218.
Fayyad, U. M., and Smyth, P. 共1999兲. “Cataloging and mining massive Rajagopalan, B., and Isken, M. W. 共2001兲. “Exploiting data preparation to
datasets for science data analysis.” J. Comput. Graph. Stat., 8共3兲, enhance mining and knowledge discovery.” IEEE Trans. Syst. Man
589–610. Cybern., Part C Appl. Rev., 31共4兲, 460–467.
Goodall, C. R. 共1999兲. “Data mining of massive datasets in healthcare.” J. Redman, T. C. 共1998兲. “The impact of poor data quality on the typical
Comput. Graph. Stat., 8共3兲, 620–634. enterprise.” Commun. ACM, 41共2兲, 79–82.
Green, S. B. 共1991兲. “How many subjects does it take to do a regression Roddis, W. M. K., and Zhang, L. 共2000兲. “Equation discovery in data-
analysis?” Multivar. Behav. Res., 26共3兲, 499–510. bases from engineering.” Proc., 2000 8th Int. Conf. on Computing in
Hajjar, D., and AbouRizk, S. M. 共2000兲. “Integrating document manage- Civil and Building Engineering, R. Fruchter, F. Peña-Mora, and W.
ment with project and company data.” J. Comput. Civ. Eng., 14共1兲, M. K. Roddis, eds., ASCE, Reston, Va., 890–897.
70–77. Rojas, E. M., and Aramvareekul, P. 共2003兲. “Is construction labor pro-
Hand, D. J., Blunt, G., Kelly, M. G., and Adams, N. M. 共2000兲. “Data ductivity really declining?” J. Constr. Eng. Manage., 129共1兲, 41–46.
mining for fun and profit.” Stat. Sci., 15共2兲, 111–126. Rojas, E. M., and Kell, I. 共2008兲. “Comparative analysis of project deliv-
Huang, X., and Hinze, J. 共2003兲. “Analysis of construction worker fall ery systems cost performance in Pacific Northwest public schools.” J.
accidents.” J. Constr. Eng. Manage., 129共3兲, 262–271. Constr. Eng. Manage., 134共6兲, 387–397.
Jeske, D. R., and Liu, R. Y. 共2007兲. “Mining and tracking massive text Soibelman, L., and Kim, H. 共2002兲. “Data preparation process for con-
data: Classification, construction of tracking statistics, and inference struction knowledge generation through knowledge discovery in da-
under misclassification.” Technometrics, 49共2兲, 116–128. tabases.” J. Comput. Civ. Eng., 16共1兲, 39–48.
Kastens, T. 共1997兲. “Farm machinery operation cost calculations.” Kan- Stegemann, J., and Buenfeld, N. 共2004兲. “Mining of existing data for
sas State University Farm Management Guide Rep. No. MF-2244, cement-solidified wastes using neural networks.” J. Environ. Eng.,
Kansas State Univ. Agricultural Experiment Station and Cooperative 130共5兲, 508–515.
Extension Service, Manhattan, Kan. Thomas, H. R. 共2000兲. “Schedule acceleration, work flow, and labor pro-
Lee, J.-R., Hsueh, S.-L., and Tseng, H.-P. 共2008兲. “Utilizing data mining ductivity.” J. Constr. Eng. Manage., 126共4兲, 261–267.
to discover knowledge in construction enterprise performance Thomas, H. R., and Horman, M. J. 共2006兲. “Fundamental principles of
records.” Journal of Civil Engineering and Management, 14共2兲, 79– workforce management.” J. Constr. Eng. Manage., 132共1兲, 97–104.
84. Vorster, M. C., and de la Garza, J. M. 共1990兲. “Consequential equipment
Ling, Y. Y. 共2002兲. “Model for predicting performance of architects and costs associated with lack of availability and downtime.” J. Constr.
engineers.” J. Constr. Eng. Manage., 128共5兲, 446–455. Eng. Manage., 116共4兲, 656–669.
Liu, M., and Ling, Y. Y. 共2005兲. “Modeling a contractor’s markup esti- Yu, W.-D. 共2007兲. “Hybrid soft computing approach for mining of com-
mation.” J. Constr. Eng. Manage., 131共4兲, 391–399. plex construction databases.” J. Comput. Civ. Eng., 21共5兲, 343–352.
Lucko, G. 共2003兲. “A statistical analysis and model of the residual value Zayed, T. M., and Halpin, D. W. 共2004兲. “Process versus data oriented
of different types of heavy construction equipment.” Ph.D. disserta- techniques in pile construction productivity assessment.” J. Constr.
tion, Virginia Polytechnic Institute and State Univ., Blacksburg, Va. Eng. Manage., 130共4兲, 490–499.
Messner, J. I. 共2003兲. “An architecture for knowledge management in the Zhang, S., Zhang, C., and Yang, Q. 共2003兲. “Data preparation for data
AEC industry.” Proc., 2003 Construction Research Congress, K. R. mining.” Applied Artificial Intelligence, 17共5–6兲, 375–381.
Molenaar and P. S. Chinowsky, eds., ASCE, Reston, Va. Zhu, Y., Mao, W., and Ahmad, I. 共2007兲. “Capturing implicit structures in
Mitchell, Z. W. 共1998兲. “A statistical analysis of construction equipment unstructured content of construction documents.” J. Comput. Civ.
repair costs using field data & the cumulative cost model.” Ph.D. Eng., 21共3兲, 220–227.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 57

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Ethnographic Theory-Building Research in Construction
Andreas F. Phelps, Ph.D., P.E.1; and Michael J. Horman, Ph.D.2

Abstract: Traditional construction research methods have enabled focused but narrow advances in our understanding of industry
phenomena. While contributing new insight, these methods are often not adequate to enable understanding of the complex interactions that
lead to many of the industry’s pervasive social and technical problems. One means of addressing these limitations is for the construction
research community to complement prevalent quantitative and case study methodologies with qualitative theory-building methodologies,
specifically ethnographic studies or those based on detailed and long-term observation of project environments. This paper identifies the
critical need for theory-building methods and their methodological conventions, challenges, and opportunities. Specifically, the paper
focuses on data collection in ethnographic studies and analysis of that data through the use of grounded theory. Through better under-
standing and more widespread use of theory-building methods, the construction research community can provide a needed complement to
the current prevailing methods and greatly aid the maturation of our important field.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000104
CE Database subject headings: Research; Construction management; Theories.

Introduction through methods such as grounded theory to develop theories that

explain behaviors observed in project teams. These overarching
Ineffective communication between project team members, lack theories produced through these methods can link several existing
of trust, inadequate adoption of new technologies, and inability to areas of research and create sets of targeted new research ques-
effectively integrate specialty contractors and consultants into a tions that can be explored using more traditional methods. The
coherent and functional project team are some of the most critical following sections of this paper outline the opportunities, meth-
issues affecting all aspects of the construction industry today. In odological conventions, and challenges in using these data collec-
large part, these challenges continue in the construction industry tion and analysis methods. Specifically, the first section focuses
despite extensive construction research into their various aspects. on data collection in ethnographic studies, the following section
This journal attests to the volume of this type of research. No shifts to how that data are analyzed through the use of grounded
matter the research subject—whether it is a new construction theory to build constructs and theories and finally how these
method, scheduling advances, lean construction, project delivery, methods can aid in the advancement of construction research.
information technology changes, or contracting techniques—the To tie these concepts more directly to capital projects, ex-
same inherent technical and social-technical issues abound. We amples from a 3-year ethnographic field study of a $100 million
argue that this is because construction research has been primarily cancer institute project and a $220 million children’s hospital
focused on research methods that attempt to simplify phenomena project are used 共Phelps 2008兲. Ethnographic studies have been
so that singular components can be observed, described, modeled, used in construction in the past but the roots of these methods lie
and predicted while controlling for extraneous variables. Al- in anthropology, and so it is important to address in this paper
though all research methods have their limitations, simplifying how these methods embrace the construction issues facing con-
constantly along the same lines leads to inherent biases. struction researchers. The children’s hospital study began with the
Ethnographic theory-building research is one type of method intent of understanding how sustainable design and construction
that offers the construction research community a complementary features could be more effectively implemented into complex
means of understanding phenomena observed in industry. Ethno- healthcare projects. The immersive nature of the study enabled
graphic studies involve long-term and extensive involvement with the researchers to expose fundamental issues that were not well
project team environments so that the details of social and tech- understood by the project team or the literature. The research
nical issues can be fully observed. The data can then be analyzed linked several previously disparate areas of research to create a
theory that helps to understand how project teams adopt sustain-
1 ability information in extremely complex projects.
Senior Engineer, Simpson Gumpertz & Heger Consulting Engineers,
Landmark Building at One Market Street, Suite 600, San Francisco,
CA 94105; formerly, Graduate Assistant, Penn State Univ., University
Park, PA 16802. E-mail: [email protected] Methodological Characteristics of Ethnographic
2
Associate Professor, Department of Architectural Engineering, Penn Theory-Building Research
State Univ., 104 Engineering Unit A, University Park, PA 16802 共corre-
sponding author兲. E-mail: [email protected]
The most important thing to understand at the outset about eth-
Note. This manuscript was submitted on August 19, 2008; approved
on June 15, 2009; published online on June 17, 2009. Discussion period nographic theory-building research is how it contrasts with cur-
open until June 1, 2010; separate discussions must be submitted for indi- rent research methods commonly employed in construction. The
vidual papers. This paper is part of the Journal of Construction Engi- top half of Fig. 1 shows how construction research is typically
neering and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, designed and performed. In this approach, the researcher uses
ISSN 0733-9364/2010/1-58–65/$25.00. literature to first develop a model 共e.g., by defining a specific

58 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 “We’re supposed to be getting the details for ‘Mike’s nuts’ any
day now so you should be able to start after the framing is up next
week.” “Mike’s nuts” is a term that the contractors use to refer to
the two oval shaped skylights in the lobby of the project named in
honor of the junior architect. The term has a commonly under-
stood meaning among the contractors in terms of its reference to
a given section of the project 共explicit meaning兲. However, the
term also gives a subtle poke at the design professionals 共who are
unaware of the reference兲 and creates a vernacular among con-
struction personnel that strengthens their association with each
Fig. 1. Comparison of traditional construction analysis methods and other 共tacit meaning兲. This example also speaks to a larger issue
theory-building methods 共Phelps 2008兲 of respect or lack of respect between design or office personnel
and field construction personnel. The foremen’s meetings usually
consist of individuals from 25 different specialty contractors. Al-
research objective, hypothesis, or relationship兲 and then structures though the meetings are open to anyone, the design professionals
the research to collect and analyze data 共e.g., an experiment, do not usually attend. Those that do attend are well respected by
survey, or case study兲 to prove or disprove the model. By con- the foremen and have no difficultly getting the foremen to make
trast, the bottom half of Fig. 1 shows the ethnographic theory- changes to their work. Those that do not attend the weekly meet-
building approach. In this case, the research begins without a ings, which also include a select few specialty contractors, are
model or preconceived explanation of the phenomena of interest. often the subject of jokes and sarcastic remarks. These elusive
Ethnography is then used to collect 共unbiased兲 observations over individuals also have a much harder time getting the rest of
a long period of time 共usually months or years兲. Analysis on this the contractors to accept changes that they need to make. As
collected data are performed to generate theories that explain this example illustrates, the richness of the data collected in
what is happening 共known as grounded theory兲. In this approach, ethnographic studies allows for insight into implicit or tacit infor-
the theoretical model that emerges from the analysis of the data. mation. This information provides useful insight into the interre-
Thus, ethnographic theory building uses ethnography to col- lation of trust, learning, and project performance that would not
lect data and grounded theory to analyze the data. These methods be available through more traditional means of construction data
of data collection and analysis are explained further in the follow- collection.
ing two sections.

Uses of Ethnographic Research Methods

Data Collection: Ethnographic Methods
Although used rarely in construction research, ethnographic data
One of the challenges with more traditional construction research collection methods have enjoyed more widespread and influential
methods is that they require collection of specific data that either use in the social sciences, organizational sciences, and informa-
supports or does not support a specific hypothesis. While this type tion sciences. Ethnographic research has been effective in socio-
of data collection enables controlled and focused research, it also logical research to understand issues such as the formation and
neglects many aspects of the construction environment. Alterna- organization of gangs 共Whyte 1993兲 and the treatment of skid row
tive methods of data collection, such as ethnographic studies, men by the jail and legal system in the 1960s 共Spradley 1970兲. In
strive to take in as much of the complexity of an environment as the study of organizational behavior and management, ethnogra-
possible and then refine future observations based on research phy has been used in numerous top business journal publications
questions that emerge from earlier observations. regarding studies in the following: the evolution of structure and
Ethnography, or the study and systematic recording of social control in newly formed self-managing teams 共Barker 1993兲;
environments through participant observation, is a method of ob- norms developed by bill collectors regarding expression of emo-
serving a specific population in their natural setting 共Denzin tions to debtors 共Sutton 1991兲; barriers and opportunities that
1978兲. In order to observe the natural behaviors of individuals, knowledge creates regarding innovation in new product develop-
these studies require longer observation periods 共i.e., months or ment 共Carlile 2002兲; cultural conflicts in multiculture organiza-
years兲 mostly to minimize the externally imposed variation tions 共Gregory 1983兲; and risk and blame in disaster sense
caused by having an observer present. The detailed experiences making 共Gephart 1993兲. Ethnography has been extensively used
collected during these long durations give researchers a rich un- in the information sciences to understand collaborative informa-
derstanding of complex phenomena that occur within that specific tion behavior and computer-supported cooperative work 共e.g.,
social environment so that researchers can develop an understand- McDonald and Ackerman 1998; Orlikowski 1992; Reddy et al.
ing of more than just the explicit meaning of words, actions, and 2001兲. In construction research, there has also been a number of
artifacts. By having such intimate knowledge of “society,” eth- innovative researchers used ethnographic methods 共Table 1兲.
nography enables researchers to understand the tacit meaning that
is implicit in the words, actions, and artifacts used by a society.
This is especially important in the building industry where, for Conventions of Ethnographic Data Collection
example, a majority of constructability knowledge 共i.e., 83%兲 is
not explicit but tacit 共Hanlon and Sanvido 1995兲. The overall goal of ethnographic methods is to collect the “richest
One example of how tacit and implicit issues can be observed possible data” 共Lofland and Lofland 1995兲. Richness of data
in ethnographic studies comes from the weekly foremen’s meet- comes from observing behavior in many different manifestations,
ing observed in our field study 共Phelps 2008兲. In one of these such as data collected through interviews, team discussions, inci-
meetings, the superintendent says to the glazing contractor, dental conversations, observations, documents, and nonverbal

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 59

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 1. Sample of Publications in Construction Journals Using Ethnography
Found that information sharing behaviors affected the development of trust and learning in project teams. Phelps 2008
Project environments characterized by trust and learning experienced convergence of values and mental models
and an improved flow of information and adoption of new information.
Classified various types of proactive and reactive means by which contractor issue claims for additional Rooke et. al. 2004
payments. They also found that the “culture of claims” stems from the high level of competition within the
industry and the need to bid low and make up profit elsewhere.
Proposed using ethnography to analyze global project to identify how institutions 共e.g., work practices, legal Mahalingam and Levitt 2004
regulations, and cultural differences兲 affect projects and how conflicting institutions interact and result in higher
project costs.
Found that innovations in coordination between firms in clean room construction projects resulted from Shields and West 2003
partnerships that continue onto other similar project and reduced complexity through replacing multiple formal
lines of communication with more direct informal lines.
Found that tensions between academic “engineering knowledge” and experiential “site knowledge” and lack of Seymour and Rooke 2001
understanding of existing mental models led to attitudes of suspicion or resentment in terms of the
implementation of lean practices in construction.
Studied women’s career under achievement in large U.K. construction companies and found that different initial Dainty et. al. 2000
perceptions of the construction industry and a strong male-oriented culture were key factors.
Found evidence that social networks research is applicable to the design process. Specifically, observed that Lloyd and Deasley 1998
informal structures determining work activity and the use of subtle “role” playing in problem solving
Found that CAD learning and experimentation in firms leveled off at suboptimal levels due to issues of Bhavnani et. al. 1996
communication and management. These findings were used to develop explicit recommendations to CAD users
and vendors.

communication 共Ball and Ormerod 2000兲. Specific data sources facts, and review of web-based project management systems.
are outlined in Table 2. As can be seen, these data collection These were conducted over a 3-year period. This diversity of data
techniques are not foreign to construction research and are di- allowed us to consider data that were formally captured in the
rectly employable in our field. project documentation, data from conversations between indi-
Using a combination of techniques allows for data from one viduals, and data regarding behaviors that could only be captured
technique to further support or enrich data collected from other through observation. In addition to gathering this data, it was also
techniques to create a rich description of the technical and social necessary to gain trust of the individuals involved with the field
processes that occur within the project team. By collecting data study. Trust reduces the risk of short-term variations in behavior
from both the systems level and individual level, the interactions by field study participants due to the presence of an outside ob-
between the various parts of the system can be analyzed in a more server, such as the Hawthorne effect 共Landsberger 1958兲. This
holistic way. Additionally, the use of multiple data collection was achieved through long-term involvement with the project and
methods 共e.g., formal interviews, contextual conversations, and having casual private discussions with project team members to
observation兲 allows for the comparison of rational, instinctual, gain their individual perspective on conflicts or other issues that
and unconscious behaviors to develop a comprehensive descrip- emerged during meetings or other project team interactions.
tion required to understand the phenomena 共Denzin 1978兲.
Our healthcare project field study involved over 1,000 hrs of
on-site observation of individual and collaborative activities, at- Challenges of Ethnographic Data Collection
tendance of over 150 coordination meetings, semistructured inter-
views with 12 key project team members, collection and review Ethnography faces the same challenges as other data collection
of 3,000+ pages of meeting notes, plans, and other written arti- methods, however, the means by which those challenges are ad-

Table 2. Data Sources Used in Ethnographic Research

Artifacts Project artifacts provide instances of project information that have been formally captured. They include the
drawings, specification, and reports that formally capture project decisions and information, meeting
minutes, requests for information, and change orders.
Critical incidents Critical incidents arise throughout the project and can significantly affect cost, schedule, quality, or other
critical goals. These incidents usually stem from project characteristics and data from events preceding or
following an incident and provide valuable insight to the system behavior.
Questions Questions arise because information that is needed is not available. Similar to the collection of critical
incidents, questions present insights to the shortcomings of the information management system.
Semistructured interviews Data from interviews allow for a better understanding of the everyday social experiences that members of
the project team encounter. Through structured interviews, team members have the opportunity to share their
rational thoughts about certain topics 共i.e., what they “should” do兲.
Open contextual interviews These less formal interviews capture less rational and more instinctual responses of team members on
certain topics 共i.e., what they “feel like” doing兲.
General observations Despite what team members think that they should do or feel like doing, they may act quite differently.
General observation is best suited for understanding what do members of the project teams actually do and
how they interact.

60 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
dressed differ due to the nature of ethnography. Specifically, that run counter to themes; 共6兲 spend significant time in the field;
the major challenges are 共1兲 maintaining consistency, rigor, and 共7兲 peer debriefing to enhance accuracy; and 共8兲 use an external
respect of these methods among researchers that may not be fa- auditor to review the project.
miliar with these methods and 共2兲 demonstrating validity and re- Reliability 共i.e., generalizability兲 is also critical because it pro-
liability of the data. vides stability and consistency of results under similar conditions.
This is a particularly important issue for construction ethno-
graphic studies given that the research is often conducted on a
Rigor in Data Collection
single site but needs to generate concepts that are applicable to
In terms of maintaining rigor and respect for ethnographic meth- the understanding of other sites 共e.g., commercial office buildings
ods, one of the pioneers in healthcare informatics ethnography, and residential construction兲. The three main types of reliability
Forsythe 共1999兲, outlined common misconceptions that outsiders are reliability of data using the same method for collection,
have regarding ethnography. Her main points are that: 共1兲 eth- reliability of data over time, and reliability of data during similar
nography is not a haphazard method of simply writing about points in time 共Kirk and Miller 1986兲. As with validity, the rich-
one’s observations; 共2兲 ethnography is actually quite counterintui- ness of the data provides an ideal environment for understanding
tive because it requires researchers to explore issues that insiders the same phenomenon by different means 共i.e., triangulation兲.
take for granted such as implicit constructs and tacit knowledge; Triangulation can be accomplished through different means,
and 共3兲 methodological rigor is essential to collecting and inter- specifically 共1兲 corroboration of observations at different times,
preting information from explicit 共i.e., verbal兲 and implicit 共e.g., in different spaces, or of different persons; 共2兲 having multiple
behavioral and nonverbal兲 sources to create empirically grounded researchers in the same investigation; 共3兲 arriving at the same
theories. interpretation of the phenomenon through different theoretical
Among the ethnographic studies cited in Table 1 concerning scheme; or 共4兲 using more than one method to gather data 共Den-
construction research, many of the studies that did not make use zin 1978兲.
of social scientists seasoned in ethnography suffered from lack of
methodological rigor. The common flaws in these studies were the
Validity and Reliability in Single Site Studies
following: deriving conclusions from limited time at the setting
共e.g., only a few meetings兲; stopping the analysis at open coding; It is important to understand that a single site can accomplish
and primarily focusing on the ethnographic interview rather than adequate validity and reliability. This is particularly important for
a spectrum of collection techniques. Of the studies that provided construction research given the size and length of projects often
strong “grounded” theories, there is a need to outline and perform provides practical limits on the number of studies that can be
follow-on research to test and strengthen the theories. included in the research. For an in-depth field study, a single site
may not provide a conflict between validity and reliability if both
issues are addressed through the richness of the data. Validity
Validity and Reliability
requires that the researcher provide enough detail to convince
The other major challenge in ethnography is demonstrating both others that the population has been sufficiently understood. Reli-
validity and reliability of the data from a single site, e.g., a single ability also requires significant detail but its purpose is to provide
construction project. Whereas validity is concerned with the data enough context and nuance that others will understand what as-
collected being accurate and meaningful, reliability 共or generaliz- pects of the unique situation are generalizable to similar situa-
ability兲 is concerned that the data collected and insights garnered tions.
have broader application than just at the field study site. Often Although field notes were only one method of data collection
validity and reliability are in tension with each other, e.g., one used in our field study, they serve as a useful illustration of how
detailed case study for validity versus broad multiple case studies rich descriptions can provide both validity and reliability. The
for reliability and the design of the research must address this following field note excerpt is taken from one of the building
tension early. envelope design coordination meetings 共Phelps 2008兲:
Validity is crucial because it is what gives the data its rel- “In the previous review meeting, the building envelope con-
evance and meaning. Golden-Biddle and Locke 共1993兲 described sultant had suggested that the concrete curb at the base of the
validity in terms of three factors: authenticity, plausibility, and curtain wall be made taller so that it extend a few inches above
criticality. Authenticity convinces readers that the researcher was the level of the exterior pavers. Instead, the project team decided
indeed part of the culture by using features such as vignettes and to keep the curb at the original height but use redundant seals—a
in vivo codes 共e.g., slang and industry-specific terms兲. Plausibility sheet metal sill pan at the exterior face of the curtain wall and
allows readers to accept the findings by having them “make another seal from the bottom of the curtain wall sill to the plaza
sense” to them. Finally, criticality further convinces readers by deck waterproofing membrane that turns up onto the top of the
causing them to reexamine their own assumptions that they had concrete curb 共see D.34 in the curtain wall shop drawings兲. There
possibly taken for granted. Developing a “grounded theory,” by was concern on whether the sealant would adhere to the plaza
its nature, provides significant opportunity to validate the re- deck membrane. During this discussion, the construction manager
searcher’s understanding of observations and implications. New 共CM兲 kept saying that the sealant was fine and kept referring to
observations either provide validation of the researcher’s under- the results of an adhesion test that had been performed. He
standing or present inconsistencies that challenge it. Although seemed to think that the test was between 关Sealant A兴 and 关Prod-
validation occurs naturally in the ethnographic research process, uct B兴 共which he thought was the uncured neoprene flashing for
Creswell 共2003兲 outlined additional strategies for validating data the plaza deck waterproofing system but is actually the protection
and constructs: 共1兲 triangulation using different data sources; 共2兲 layer兲. Finally, the consultant asked if the CM had the test report
member checking to determine the accuracy of findings; 共3兲 use on file somewhere. The CM went to his desk and brought the test
of a rich thick description to convey findings; 共4兲 clarify the re- report back to the meeting, the building envelope consultant
searcher’s bias through self-reflection; 共5兲 present discrepancies looked at it and realized that first of all the test was for the sealant

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 61

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
adhesion to the air and vapor barrier and that the results said that understanding phenomena in their full complexity, e.g., complex
the sealant only adhered with a 1 / 2 bond line under static weather social and sociotechnical phenomena regarding trust in teams,
seal applications 共i.e., only works for thick joints that see no information sharing behaviors, or obstacles to innovation. Theo-
movement兲. So, the adhesion test actually means nothing regard- ries related to such topics need to be derived from observation of
ing the conditions addressed in this specific detail.” the phenomena in their natural setting. Construction consists of
This excerpt provides useful insight as to how information is projects that involve some of the most complex technical systems
used or misused within project teams and how those behaviors that must be designed and constructed by numerous specialized
can influence trust and future interactions between project team people in social systems that we know as project teams. The
members 共this interaction was of major importance to the research emergent nature of this research method allows the findings to be
model developed by us兲. The richness of the description gives continuously developed, refined, and linked until a grounded
enough detail that readers have confidence that the observer ac- theory is built. Because these theories emerge from data collec-
curately captured the discussion 共i.e., validity兲 but it also provides tion of a social environment in its full reality, they result in insight
enough context that the situation can be generalized to other simi- and understanding that would not be possible through other
lar situations 共i.e., this could easily happen on any other construc- means.
tion project兲. To further establish validity and reliability, the field In the healthcare project field study, initial observations iden-
study also looked at four different scopes throughout the entire tified problems with incorporating innovation into projects that
delivery process 共i.e., building envelope, MEP, infection control were inherently complex. Subsequent observations identified the
risk assessment, and commissioning兲 to essentially create a root causes of these problems, e.g., ineffective management of
sample of four. The repeated observation of certain phenomena existing information, lack of trust between project team members,
during multiple meetings during each of the phases for each of and aversion to learning. Additional data collection led to greater
these scopes provided a level of reliability in the data collection understanding of each of these issues and the development of
and also helped to explain anomalies. constructs that link them to create a theoretical construct regard-
ing the cyclical nature of how information sharing behaviors af-
fect the development of trust and learning within project teams.
Method of Analysis: Grounded Theory

Once ethnographic data have been collected, it requires analysis

Uses of Grounded Theory
in order to develop theoretical constructs. One method of analysis
that can be used is grounded theory, i.e., the systematic develop-
ment of theory from the data through inductive and deductive Grounded theory allows researchers to study phenomena such as:
thinking. Grounded theory specifically stresses: 共1兲 the impor- 共1兲 behaviors that arise due to social conditions; 共2兲 behaviors that
tance of long-term field studies to discover what is really going are not directed toward the attainment of organizational or mana-
on; 共2兲 the existence of complexity and variability in human ac- gerial goals; and 共3兲 cognitive activities such as problem solving
tion; 共3兲 a realization that people act based on meaning and that and other team-oriented activities that otherwise would not be
the meaning is defined and redefined through interaction; 共4兲 a possible due to methodological limitations 共Ball and Ormerod
sensitivity to the importance of the evolving and unfolding nature 2000兲. This process of analysis relies on the researcher’s ability to
of events 共i.e., process兲; and 共5兲 an awareness of the interrelation- interpret the data, view social phenomena holistically, and use
ship between conditions 共i.e., structure兲, action 共i.e., process兲, and complex reasoning that is multifaceted, iterative, and simulta-
consequences 共Strauss and Corbin 1998兲. neous 共Creswell 2003兲. Table 3 provides a comparison of various
To better understand grounded theory, it is useful to contrast it types of methodologies.
with other methodologies. In quantitative studies, researchers iso- While all of the methods described in Table 3 are valuable for
late certain variables by use of controls to create repeatable con- various purposes, there is a need for complementary methods that
ditions. Those controlled conditions are used to provide data that allow for analysis of the full complexity present on construction
either proves or disproves a prefigured hypothesis. Although these projects 共Raftery et al. 1997兲. The following sections outline the
methods enable greater understanding of phenomena in controlled methodological conventions and challenges of using grounded
environments, they negate certain aspects that make up the full theory.
“reality” surrounding construction industry phenomena. These in-
clude situations where emotions, subconscious thought processes,
or complex interrelationships are involved. Conventions of Grounded Theory
Even qualitative research methods that are intended to capture
broad information and insights can be inadequate to describe
Grounded theory enables researchers to understand complex so-
some complex social or sociotechnical phenomena. Question-
cial and sociotechnical phenomena because the iterative nature of
naires and surveys only provide a characterization of one point in
the analysis allows for continuous focusing and refinement of data
time and do not capture how attitudes and values might change
collection. Coding and the ethnographic interview are the two
over time. Surveys have the additional limitations of requiring
prevalent techniques for ordering, refining, and linking data into a
respondents to fit their answers within a limited range of answers.
theory.
Observations of subjects in lab studies or other “artificial” envi-
ronment do not include all of the factors present in the subject’s
natural environment that may be critical to the behavior of the Coding
subjects. Even short-term observations of case studies do not pro-
vide enough context to understand the social and sociotechnical In addition to the richness of the data, the real value of ethno-
components of observed phenomena. graphic data is illustrated through coding. Coding is the process
In some instances the nature of the research problem requires of microanalysis that examines and interprets data at a minute

62 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 3. Comparison of Different Research Types
Research
type Methods Characteristics Logic for drawing conclusions
Quantitative Experiments, statistics, simulation, Short procedures, controlled for extraneous Measurement and quantification between
共deductive兲 surveys, questionnaires variables, often large sample sizes independent and dependent variables, establishment
of causal relationships between variables
Qualitative Observation, case study, Short-term studies, smaller sample sizes, Structured observation and qualitative analysis
共deductive兲 open-ended surveys selective control exercised over of data to either support or disprove a prefigured
extraneous variables hypothesis
Qualitative Observation and interaction with Long-term study of a single environment, Theory emerges from observations of populations
共inductive兲 participants, e.g., ethnography consideration of all variables to capture in their natural setting, known as grounded theory
full complexity

level 共e.g., at the word or phrase level兲. Open coding, axial cod- support for the development of those definitions. Through a pro-
ing, and selective coding are sequential methods of coding that cess of axial coding, additional data collection and analysis en-
build upon each other to develop a theory. abled these types to be realigned based on their relationships. By
Open coding is an analytical process used to identify concepts, better understanding the properties of each type of ambiguity, the
their properties, and dimensions that inform the data 共Strauss and 20 ambiguity types were grouped based on similarities into five
Corbin 1998, p. 101兲. Data are “opened” by breaking it down into more general types of ambiguity. Then each of those types was
discrete incidents or ideas and categorizing them based on their ranked within a hierarchy and used to create a model that de-
meaning so that additional incidents can be grouped based on scribes the observed phenomenon 共Fig. 2兲. This model could then
those categories. be used to tie these findings into a more overarching theory that
Once the data has been opened, it needs to be realigned project team discussions need to address the appropriate level of
through axial coding based on its properties and dimensions. ambiguity in order to not negatively affect trust among project
Axial coding is the process of relating categories to their sub- team members.
categories to form more complete and precise explanations of
phenomena 共Strauss and Corbin 1998, p. 124兲. Using these con-
Ethnographic Interview
ceptualizations of the observed incidents, researchers can begin to
discover relationships among categories 共e.g., why, how come, In addition to coding through microanalysis, coding can also be
where, when, how, or with what results兲. These relationships elu- accomplished through the “ethnographic interview.” Ethno-
cidate the interaction between structure 共i.e., why兲 and process graphic interviews go beyond traditional interviews in that they
共i.e., how兲 for a given phenomenon and serve as the basis for are not only used to collect data but also to begin ordering and
developing an overarching paradigm. The critical components categorizing that data. Interviewers use descriptive questions to
needed to develop a paradigm include the conditions, the actions/ draw out from the subjects 共i.e., informants兲 explicit and implicit
interactions, and the consequences. Conditions are sets of events cultural knowledge rather than asking specific questions. This
that act as catalysts 共i.e., causal兲, moderators 共i.e., intervening兲, or style of “tell me about . . . ” questioning allows informants to
mediators 共i.e., contextual兲 for a phenomenon. Actions and inter- speak in their own terms about issues that are pertinent to them.
actions demonstrate how an individual responds to conditions. From this descriptive knowledge, an ethnographer identifies do-
They can be either strategic actions 共i.e., purposeful兲 or routine mains in a given culture 共i.e., open coding兲 and asks follow-up
actions 共i.e., habituated兲 and may or may not be coordinated with structural questions that clarify how informants organize knowl-
other actions 共i.e., interactions兲. Last, consequences are responses edge 共i.e., axial coding兲. The refined domains can then be used to
to a situation that may affect the phenomenon. create a taxonomy of the domain and its subsets. Following the
After the data and relationships between the data have been development of a taxonomy, further follow-up questions can be
further explained, the theory can be integrated and refined used to systematically develop characteristics and meaning for
through selective coding processes. Explained phenomena are in- each component and outline relationships between components to
tegrated into a theory by identifying a central theme and relating create a complete paradigm 共i.e., selective coding兲. Because this
the other observed phenomena to the central theme. Strauss and
Corbin 共1998兲 outlined techniques for discovering the central
theme, including developing diagrams, writing a story line, or
reviewing and sorting through memos. As the explained phenom-
ena are integrated with the central theme, the theory can be re-
fined for internal consistency and gaps in logic.
In the field study, one theory that emerged was regarding
a hierarchy of ambiguity 共i.e., certain types of ambiguity need
to be addressed in order for project teams to have substantive
discussions regarding other types of ambiguity兲. From the field
notes and review of project documentation, there emerged 338
difference “instances” of ambiguity. Through a process of open
coding, those instances resulted in the formation of 20 types of
ambiguity. The process of developing definitions for each of
the types, classifying each instance as a type, and refining the
definitions they held for each instance in that category provided Fig. 2. Hierarchy of ambiguity 共Phelps 2008兲

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 63

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
interviewing technique requires significant skill on the part of the focused on the technical aspects. Ethnographic theory-building
interviewer, construction researchers may be well served to team methods provide an opportunity to address this deficiency by ob-
with social scientists in cases where this technique is used. taining intricate details about phenomena such as feelings,
thought processes, and emotions that are difficult to extract or
learn through more conventional construction research methods.
Challenges with Grounded Theory These methods allow the construction research community to de-
velop theories related to trust, commitment, perceptions of fair-
Although they lack the statistical rigor of more quantitative stud- ness, conflict, information behavior, and other sociotechnical
ies, theories built using qualitative methods need to demonstrate issues. They also provide a methodological bridge with other so-
their strength by providing empirical grounding. Empirical cial science fields such as sociology, organizational theory, infor-
grounding ensures that enough detail has been provided and that, mation sciences, and organizational behavior. By aligning with
given similar sets of conditions and similar processes, will be social sciences in addition to more technical academic fields, the
observed and explained by the theory. It allows for future studies construction research community has an opportunity to pursue
to validate and build upon the original study. The critical ques- translational research that makes use of theories and methodolo-
tions that researchers need to address consist of 共Strauss and gies that have been already extensively researched and proven in
Corbin 1998, p. 270兲 the following: more established academic fields.
• Are concepts generated? What are their sources? Because the resulting theories emerge from observations of a
• Are concepts systematically related? natural setting, ethnographic studies facilitate closer links be-
• Are there many conceptual linkages and are the categories tween industry and academia. Ethnographic theory building re-
well developed? Do categories have conceptual density 共e.g., quires close interaction with industry because industry is the
many properties and dimensional variations兲? subject of analysis, and there is no other way of doing this type of
• Is variation built into the theory 共e.g., the study of multiple research than immersion in industry often for a long period of
phenomena兲? time. Other research might be accomplished by other means and
• Are conditions under which variation can be found built may in fact require separation between the research and industry,
into the study and explained? For example, economic factors, e.g., testing a new fire repellant material must be done in a con-
policies, regulations, social movements, trends, culture, social trolled, safe, and isolated environment rather than on an existing
values, etc. or newly constructed facility. The benefit of ethnographic theory-
• Has process been taken into account? building research, thus, is to hold the research to the “flame” of
• Do the theoretical findings seem significant, and to what industry often at many points throughout the research process and
extent? therefore helps to lower the barrier between research and industry.
• Does the theory stand the test of time and become part of the The resulting close partnerships between industry and academic
discussions and ideas exchanged among relevant social and programs is likely to mutually strengthen and provide greater in-
professional groups? teraction and awareness of what is happening in each.
Regarding the healthcare field study, there were two means of Another way in which ethnographic theory-building research
providing empirical grounding. The first was through an internal can benefit construction research is by linking existing research
process of validation where the researchers asked the critical efforts into broader-level integrated paradigms. These paradigms
questions listed above regarding the constructs that emerged. The can act as roadmaps for comprehensive research programs that
second was through external validation from presentations of the address complex interrelated problems that for a single researcher
research findings to the field study participants and nonrelated would be too overwhelming to address. By providing the over-
industry members to gain their thoughts regarding the empirical arching structure for coordinated research, these paradigms can
grounding of the constructs. identify critical gaps in the research and unify related sectors of
The strengths of ethnography and grounded theory rely on the academic construction community to create powerful theories
effective collection of rich data and systematic processing of that fully understand the industry and have the potential to fun-
data to develop empirically grounded theories. The conventions damentally and holistically change the way that the construction
detailed above provide the structure for researchers to perform industry functions.
the academically significant and methodologically rigorous re-
search needed to improve our understanding of complex phe-
nomena in construction. Path Forward

In today’s ever-changing information-heavy environment, there

Opportunities for Ethnographic Theory-Building are numerous phenomena that need to be studied to enable the
Methods construction industry to effectively function in the future. These
include information flow and integration throughout the delivery
Ethnographic theory-building methods provide significant oppor- process, performance effects of innovative delivery methods, and
tunities for the construction research community to address issues the social and technical factors affecting communication, produc-
that in the past have been neglected due to methodological limi- tivity, and the implementation of innovation. Ethnographic
tations. The major areas include: 共1兲 understanding of social pro- theory-building methods provide a tremendous opportunity for
cesses; 共2兲 understanding the interaction between social and construction research to address these issues in their full complex-
technical processes; 共3兲 providing a direct link between academia ity by providing a methodology to study complex issues closer to
and industry; and 共4兲 linking related research areas into overarch- their entirety rather than in parts using conventional construction
ing paradigms. research methods. This is particularly appropriate for research
Construction, by its nature, is both social and technical. that aims to engage sociotechnical issues in construction such as
However, to date, the overwhelming majority of the research has implementing new scheduling methods and evaluating alternative

64 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
contracting incentives on project performance and worker produc- Dainty, A. R. J., Bagilhole, B. M., and Neale, R. H. 共2000兲. “A grounded
tivity issues. theory of women’s career under-achievement in large UK construc-
The use of ethnographic theory-building methods provides tion companies.” Constr. Manage. Econom., 8共2兲, 239–250.
great opportunities for the construction research community to Denzin, N. 共1978兲. Sociological methods: A sourcebook, 2nd Ed.,
understand complex phenomena. In order to capitalize on these McGraw-Hill, New York.
opportunities, there are a few measures that the construction re- Forsythe, D. E. 共1999兲. “It’s just a matter of common sense.” Ethnogra-
search community can take. First, construction researchers can phy as Invisible Work, Computer Supported Cooperative Work, 8,
begin partnering with social scientists in business schools, sociol- 127–145.
Gephart, R. P., Jr. 共1993兲. “The textual approach: Risk and blame in
ogy programs, and information system programs. These academic
disaster sensemaking.” Acad. Manage. J., 36共6兲, 1465–1514.
communities are already well versed in ethnography and
Golden-Biddle, K., and Locke, K. 共1993兲. “Appealing work: An investi-
grounded theory methods. Equally, construction researchers can gation of how ethnographic texts convince.” Org. Sci., 4共4兲, 595–616.
begin training themselves in these techniques. Second, academics Gregory, K. L. 共1983兲. “Native-view paradigms: Multiple cultures and
can also begin to develop longer-term partnerships with the con- cultural conflicts in organizations.” Adm. Sci. Q., 28共3兲, 359–376.
struction industry to gain much needed access to projects for data Hanlon, E., and Sanvido, V. 共1995兲. “Constructability information classi-
collection as well as test beds for applied theory. fication scheme.” J. Constr. Eng. Manage., 121共4兲, 337–345.
Finally, ethnographic theory-building research methods are Kirk, J., and Miller, M. 共1986兲. Reliability, validity and qualitative re-
meant to complement, rather than replace, the traditionally more search, Sage, Beverly Hills, Calif.
quantitative methods used in construction research. These quanti- Landsberger, H. A. 共1958兲. Hawthorne revisited, Ithaca, New York.
tative methods are not well suited to address the detailed nuances Lloyd, P., and Deasley, P. 共1998兲. “Ethnographic description of design
of cause and effect relationships that require strict controls and networks.” Autom. Constr., 7共2兲, 101–110.
measured variables. However, the research outcomes of ethno- Lofland, J., and Lofland, L. H. 共1995兲. Analyzing social settings: A guide
graphic studies and theory-building methods can be employed to to qualitative observation and analysis, Wadsworth.
provide structure for more focused and quantitative studies to fill Mahalingam, A., and Levitt, R. E. 共2004兲. “Challenges on global
in missing details. Construction is a sociotechnical science that projects-an institutional perspective.” Proc., Int. Symp. of the CIB
requires the use of empirical methods to quantifiably simplify W92 on Procurement Systems.
reality in order to model and predict its processes and phenomena. McDonald, D. W., and Ackerman, M. S. 共1998兲. “Just talk to me: A field
However, without an equal emphasis on social, behavioral, and study of expertise location.” Proc., 1998 ACM Conf. on Computer
cultural factors to build a rigorous understanding of these phe- Supported Cooperative Work, ACM Press, New York, 315–324.
nomena, the results from empirical and quantitative studies alone Orlikowski, W. J. 共1992兲. “The duality of technology: Rethinking the
concept of technology in organizations.” Org. Sci., 3共3兲, 398–427.
will be chronically deficient. Striking this balance can be accom-
Phelps, A. F. 共2008兲. “Lean information flow in complex healthcare
plished through pursuing an agenda of a more integrative intuitive
projects and the role of the ‘integrator’ as information manager.”
synthesizing of research and appropriate methodologies that take Ph.D. dissertation, Pennsylvania State Univ., Pa.
place in natural settings. Through these complementary strategies Raftery, J., McGeorge, D., and Walter, M. 共1997兲. “Breaking up method-
the construction research community can organize and address the ological monopolies: A multi paradigm approach to construction man-
complex pervasive sociotechnical issues that plague our industry agement research.” Constr. Manage. Econom., 15, 5–18.
today and position our field to realize the full significance and Reddy, M., Dourish, P., and Pratt, W. 共2001兲. “Coordinating hetero-
power of its potential. geneous work: Information and representation in medical care.”
Proc., European Conf. on Computer Supported Cooperative Work
(ECSCW’01), 239–258.
References Rooke, J., Seymour, D., and Fellows, R. 共2004兲. “Planning for claims: An
ethnography of industry culture.” Constr. Manage. Econom., 22共6兲,
655–662.
Ball, L. J., and Ormerod, T. C. 共2000兲. “Putting ethnography to work: The
Seymour, D., and Rooke, J. 共2001兲. “The role of ethnography in the
case for a cognitive ethnography of design.” Int. J. Hum.-Comput.
Stud., 53, 147–168. implementation of lean construction.” Proc., 9th Int. Group for Lean
Barker, J. R. 共1993兲. “Tightening the iron cage: Concertive control in Construction Conf., 61–76.
self-managing teams.” Adm. Sci. Q., 38共3兲, 408–437. Shields, R., and West, K. 共2003兲. “Innovation in clean-room construction:
Bhavnani, S. K., Flemming, U., Forsythe, D. E., Garrett, J. H., Shaw, D. A case study of co-operation between firms.” Constr. Manage.
S., and Tsai, A. 共1996兲. “CAD usage in an architectural office: From Econom., 21共4兲, 337–344.
observations to active assistance.” Autom. Constr., 5共3兲, 243–255. Spradley, J. P. 共1970兲. You owe yourself a drunk: Adaptive strategies of
Carlile, P. R. 共2002兲. “A pragmatic view of knowledge and boundaries: urban nomads, Little Brown, Boston.
Boundary objects in new product development.” Org. Sci., 13共4兲, Sutton, R. I. 共1991兲. “Maintaining norms about expressed emotions: The
442–455. case of bill collectors.” Adm. Sci. Q., 36共2兲, 245–268.
Creswell, J. W. 共2003兲. Research design: Qualitative, quantitative, and Whyte, W. F. 共1993兲. Street corner society: The social structure of an
mixed methods approaches, 2nd Ed., Sage, Thousand Oaks, Calif. Italian slum, 4th Ed., Univ. of Chicago, Chicago.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 65

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Charrettes as a Method for Engaging Industry in Best
Practices Research
G. Edward Gibson Jr., P.E., F.ASCE1; and Donald A. Whittington, P.E.2

Abstract: Gaining innovative and useful research findings concerning construction industry best practices requires an interaction and
feedback mechanism between industry respondents and academia. Typical research methods such as surveys, source document reviews,
and structured interviews will work, but suffer from barriers which can hamper results. Examples of these barriers include low response
rates, asynchronous communication, time commitment of the researchers and respondents, access to project data, and travel costs.
Structured workshops 共research “charrettes”兲 are a unique and useful method for facilitating data collection between industry respondents
and academic researchers. They combine the best tenets of surveys, interviews, and focus groups in an accelerated time frame. This paper
will explain how these workshops provide a critical avenue for industry interaction. Characteristics leading to successful charrettes will be
outlined. The paper will conclude by describing the benefits of these workshops to researchers including lessons learned from successful
workshops.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000079
CE Database subject headings: Best Management Practice; Construction management; Construction industry; Research;
Methodology.
Author keywords: Best management practices; Construction management; Construction industry; Lessons learned; Research methods.

Introduction 共TxDOT兲 and others. Findings, practices, and tools developed as

a result of these workshops are widely used in industry today. The
For academic researchers to discover innovative and truly useful paper will explain how these charrettes provide a critical avenue
findings concerning construction industry best practices, an effec- for gaining data and in determining the applicability and useful-
tive interaction and feedback mechanism between industry re- ness of newly created products or practices intended for industry
spondents and academia is needed. Typical research methods such use. It will illustrate how the interaction among industry profes-
as surveys, source document reviews, and structured interviews sionals and academic researchers provides feedback and open
provide that mechanism, but suffer from barriers such as low forum discussion concerning critical investigation topics, while
response rate, long time for responses, asynchronous communica- providing direction for future research efforts. The paper will also
tion, time commitment of the researchers and respondents, access discuss how to conduct these workshops.
to project data, and travel costs which can hamper results. A Special attention will be devoted to characteristics leading to
unique and useful method for facilitating data collection between successful workshop outcomes for both the industry participants
industry respondents and academic researchers is the use of struc- and the researchers alike, including issues such as: who to invite,
tured workshops 共research “charrettes”兲, which combine and in- how to provide motivation for industry practitioners to attend,
tegrate the best tenets of surveys, interviews and focus groups in selection of location and venue of the charrette, optimum number
an accelerated time frame. of participants in attendance, sample size considerations, and
This paper will discuss how research charrettes have been ap- so on. The paper will address when these types of workshops are
plied to several research projects over the past 14 years by the best applied, as well as when they may not be as useful. The
writers to investigate best project practices for the Construction paper will conclude by describing the benefits of charrettes to
Industry Institute 共CII兲, Texas Department of Transportation researchers including lessons learned from successful workshops.

1
Professor, Program Chairman, Del E. Webb School of Construction,
Arizona State Univ., P.O. Box 870204, Tempe, AZ 85287-0204 共corre- Background of Planning Charrettes
sponding author兲. E-mail: [email protected]
2
Advanced Engineer, ExxonMobil Research and Engineering Com-
pany, 3225 Gallows Rd., Rm. 4A-1726, Fairfax, VA 22037; formerly, A planning charrette is a tool used by designers and facility plan-
Graduate Research Assistant, Dept. of Civil, Construction and Environ- ners to elicit ideas and leverage industry experience in an intense
mental Engineering, Univ. of Alabama, Box 870205, Tuscaloosa, AL and targeted workshop format. Based upon the French word for
35487. E-mail: [email protected] “cart,” charrette became part of the architectural vocabulary in the
Note. This manuscript was submitted on August 15, 2008; approved early 1900s at the Ecole des Beaux-Arts in Paris 共Healey 1991兲.
on April 2, 2009; published online on April 30, 2009. Discussion period
open until June 1, 2010; separate discussions must be submitted for indi-
Nearing the end of a 12 or 24 h studio project, architectural stu-
vidual papers. This paper is part of the Journal of Construction Engi- dents would frantically race to finish their work and place it on a
neering and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, collection cart as it was being wheeled along the aisles of drafting
ISSN 0733-9364/2010/1-66–75/$25.00. tables. It has been said that some students would actually walk

66 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
along side the cart to make a few last-minute changes before the 关Board on Infrastructure and the Constructed Environment
projects were collected by the instructors. 共BICE兲 2003, 2006, 2007兴. Research charrettes have been used to
Currently, planning charrettes are being used for a variety of test software 共Clayton et al. 1998兲. The term “charrette’s” genesis
activities including facility programs, master plans, strategic op- was in teaching architecture and it is still used in that regard today
erational plans, primary architectural designs, and more 共Carter & 共Walker and Seymour 2008兲. Charrettes can be used to develop a
Burgess, Inc. 2002; Gibson and Gebken 2003; Lindsey et al. research agenda for a funding agency.
2003; Lennertz and Lutzenhiser 2006兲. Organizations as diverse In the marketing and health care industries, these types of
as the Federal Highway Administration, the U.S. Department of meetings are typically called focus groups and have been success-
Energy, the U.S. Green Building Council, the U.S. Army Corps of fully used since the 1930s to determine opinions about manu-
Engineers, and the American Institute of Architects espouse the factured goods, marketing campaigns, political concerns, and
benefits of these workshops in the planning environment. The health care options prior to broader introduction 共Stewart and
value of these sessions is found in gathering the key stakeholders Shamdasani 1990; Morgan 1997; Krueger and Casey 2000兲.
in a short, intense work session, gaining input and making deci- Calder 共1977兲 clarifies the relationship between qualitative re-
sions that will lead to better project planning, alignment toward search and focus groups, pointing out that in some circles quali-
project goals, and ultimately better outcomes. tative research has become almost synonymous with the focus
Planning charrettes typically last one and one-half to four days group interview. Kitzinger 共1995兲 defines the purpose of a focus
or longer, with key project stakeholders captive during that time. group as a group process that assists people to explore and clarify
The leader and a facilitator focus on project objectives, critical their views in ways that would be more difficult in a one to one
decisions, development of scope documents, and team alignment. interview. When done effectively, participants work alongside the
These efforts are facilitated by the use of structured checklists, researcher, often taking the research in new, unexpected direc-
open brainstorming, diagramming methods, small group break- tions. The usefulness of focus groups can be found in the study
outs, multivoting, three-dimensional visualization and so on to performed by Parasuraman et al. 共1985兲 where the responses of
help make effective use of the participant’s time. At the end of the participants in focus groups were consistent and allowed common
charrette, a path forward set of documents have been developed themes to emerge.
that facilitate completion of the scope and design of the facility or The literature review found few published works referencing
master plan. research charrettes or structured workshops as a social science
A key to planning charrette success is the efficiency with data collection method in the construction and engineering arena;
which the meeting time is planned and conducted, including of those discovered, none provided guidance on how to plan and
having a defined agenda, structured and unstructured exercises, conduct a research charrette.
use of tools, and so on. Another key is the integrated involvement The typical research charrette combines several types of data
of project participants. The key players involved in planning collection methods into a four to six-hour-session to focus on a
charrettes include the project sponsor共s兲, the project engineer or single thread of the investigation. The charrette is managed by the
architect, the key discipline leads, the technical representatives, researcher or research team, and is very organized in its focus. At
the user representatives, key stakeholders, the design/builder or the end of the workshop, all data necessary to complete a research
construction manager, and a facilitator. This all-encompassing investigation phase has been collected.
participant list helps ensure the right voices are heard in the de- The writers have been involved in conducting research char-
velopment of the project’s scope definition, hopefully leading to rettes over the past 14 years as outlined in Table 1. A total of 29
less iteration in detailed design. Another key is the organization charrettes have been conducted with approximately 400 partici-
of the session and the facilitation and guidance by the person pants, representing over 100 organizations.
overseeing the effort 共Gibson and Gebken 2003; Lennertz and
Lutzenhiser 2006兲. Types of Research Investigation Methods

Engagement of industry in “best practices” research requires so-

Research Charrettes cial data collection and statistical evaluation methods. The data
captured is typically a combination of both qualitative and quan-
A research charrette is a derivative of the planning charrette. It is titative input by respondents and observations of work practices
a unique marriage of the planning charrette with social data col- or interaction of practitioners. Many times, these efforts are ex-
lection methods to elicit ideas and leverage industry experience ploratory in nature, trying to determine the best methods, pro-
in an intense and targeted workshop format. Its major advantages cesses, and controls needed to gain a successful outcome to a
are the synergy gained by having a group of industry experts defined problem. For instance, one may want to understand the
interacting in a structured manner, the ability to use multiple data best methods of planning an industrial project in order to favor-
collection strategies in a single setting, and the fact that when ably influence the success of the business venture, a subject with
the workshop is over, the data have been collected and are avail- limited or conflicted past research studies. There are many vari-
able for analysis. It also has a secondary benefit of helping to get ables and influences to the success of the effort, some under the
“buy-in” from industry participants to use or rely on the results control of the team and some not. How to study this area is a
of the charrette and to solicit follow-on involvement for valida- question that is difficult to answer. Typical issues to consider in
tion or testing of any methods, findings, or “best practices tools” choosing the investigation methods, along with how the research
developed. charrette method integrates into these issues are given below.
Research workshops 共or charrettes兲 in construction or project
research are not new. As an example, the National Research
Collection Methods
Council regularly convenes groups of experts and uses their col-
lective knowledge and experience to determine findings and rec- Data collection strategies must be tailored to the research problem
ommendations about specific issues in the construction industry statement and are typically a combination of several methods

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 67

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 1. Sample of Recent Projects Employing Structured Research Charrettes
Number of Number of Number of
Research investigation research charrettes participants organizations
Project rating definition index for industrial projects 2 54 31
共Dumont and Gibson 1996兲
Alignment during front end planning 共Griffith and Gibson 1997兲 3 38 19
Project rating definition index for building projects 共Cho et al. 1999兲 7 69 35
Expediting highway construction while retaining quality tool 6 62 6
共Simon et al. 2002兲
International project risk assessment 共Walewski et al. 2004兲 4 44 25
Advance planning risk analysis 共APRA兲 for transportation projects 3 62 12
共Caldas et al. 2007兲
Shutdown turnaround alignment review tool 共Whittington et al. 2008兲 4 67 39

共Gibson and Hamilton 1994; Yin 1989兲. Babbie’s taxonomy Drawbacks to Typical Data Collection Methods
共Babbie 1992兲 lists five modes of classifying the collection of
social data: Survey research methods are employed extensively to study con-
1. Experimental research involves taking action and observing struction industry best practices or other study variables, however
the consequences of that action and is usually associated with surveys have several potential barriers that can cause concern.
the physical sciences. Response rates can range as low as the single digit percentages
2. Survey research involves collecting data through asking for national surveys 共questionnaires兲; many times response rates
people questions and is perhaps the most frequently used as low as 15% are considered acceptable. Low response rates can
mode of observation in the social sciences. be problematic from two perspectives. The first is the amount of
3. Field research involves the direct observation of social phe- resources needed to send out many questionnaires, only to receive
nomena in natural settings. a few responses; and, if the survey is attempting to be represen-
4. Unobtrusive research involves investigation without the re- tative of a study population, uneven response can bias the results.
searcher intruding into whatever is being studied. It can con- The research charrette method has a captive audience 共group of
sist of content analysis, analysis of existing statistics, or respondents兲 and, if handled properly, surveys will be received
historical/comparative analysis. from all attendees by the end of the session, although not a ran-
5. Evaluation research seeks to evaluate the impact of social dom survey as explained later.
intervention by using experimental and quasi-experimental Another problem with questionnaires is the response time. In
methods. many situations, and especially with busy construction industry
Any of these methods can be used in a standalone or combined practitioners, it may take months to receive completed question-
manner. A research charrette will typically use more than one of naires, even with extensive follow-up correspondence. When the
the methods listed above, specifically Items 2: survey research, 3: research charrette is complete, the survey instruments are in hand.
field research, and 5: experimental Methods. Typically, the char- Finally, questionnaires must be well developed and tested be-
rette will use one or more surveys 共perhaps with the participants forehand in order to ensure that the intended meaning is trans-
bringing a partially completed instrument to the event兲. Most ferred to the respondent. The communication is typically
charrettes include an interactive component. This component al- asynchronous from the respondent to the questionnaire. In a re-
lows the researchers to observe the participants interact with each search charrette, the questionnaire can be explained by the re-
other and the research team in terms of specific problems or re- searcher to the group and questions can be asked by respondents
search questions, and to relay their observations about the re- for clarification, ensuring that the right answers are given for the
search investigation to date. Finally, if multiple workshops are intent of the question. If several charrettes are performed using
used with a mixture of demographic respondents, some inferences the same facilitator, consistency of response can be maintained
can be gained from the impact of the research investigation on the across the data collection effort.
attitudes of the participants. Typical data collection methods that Another survey method is the structured or semistructured in-
can be used in a research charrette are given in Table 2. terview. This is an excellent method, but again, setting up an

Table 2. Examples of Data Collection Methods Used during or after Research Charrettes
Type of collection strategy Examples
Surveys/questionnaires Background information about the participants; opinions of relative importance
of best practice strategies
Observation Participant’s interaction with peers in the charrette or their opinions about the
subject matter
Structured interviews/facilitated discussion Collective group or small group structured questions and responses
Focus groups Collective or small charrette group response to research products or findings to date
Informal interviews Interaction of the researchers with charrette participants during breaks or lunch
Result validation Participants volunteering to test, use or employ research findings subsequent
to the charrette

68 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
interview either in person or telephonically can take weeks or teams involved in the planning and execution of industrial capital
months. These interviews are typically performed one at a time facility projects,” so that the results can be applied specifically to
and are very resource intensive. In a research charrette, structured the investigation domain of interest.
questions can be asked in a collective setting or a small group, Once the population has been defined, the question of taking a
thus gaining the insight of quite a few practitioners in a short sample must be addressed. There are basically two methods of
period of time. selecting a sample from a defined population; probabilistic and
Field research and evaluation research again are resource in- nonprobabilistic. With a probability sample, each element has a
tensive. In the research charrette, small group deliberations and known nonzero chance of being included in the sample 共Kalton
structured prioritization exercises can be used to determine the 1983兲. In order to select a sample by a probability mechanism, a
general opinion of the participants to particular proposed strate- sample frame is required. A sample frame is essentially a list of
gies or methods. The data gathered in these settings can provide all members of the study population and a sample is selected from
keen insight into problems of interest and potential solutions to this list using some form of a random selection procedure.
the research hypotheses.
Purposive nonprobability sampling is characterized by a se-
lection process that does not result in every member of the
Exploratory Research population having a chance of being selected. Examples of non-
probability sampling include 共Kalton 1983; Griffith and Gibson
Sedlack and Stanley 共1992兲 discuss exploratory research as a situ- 1997兲:
ation in which a research investigator is exploring the basic di- • Volunteer subjects for studies;
mensions or possibilities of a complex subject or where little has • Time and motion studies of workers or equipment on a few
been previously recorded. They suggest that there are three basic construction sites;
questions under which exploratory research is conducted: • The clients of a given contractor;
1. Are there any variable relationships that appear to exist? • The employees of one firm;
2. What more detailed and systematic investigation is war- • Interviews conducted on a specific job site;
ranted? • Respondents to a pull-out questionnaire included in a trade
3. What methods are most likely to generate profitable data? magazine; and
If the hypothesis undergoing research has a lack of previous • Persons calling in response to an e-mail solicitation requesting
studies, unclear past results, or has many influences, they recom-
opinions.
mend using the data collection strategy of triangulation, which is
Probability sampling permits the use of statistical theory to
“the process of comparing the measurement of the same variable
develop estimators of little or no bias and to evaluate sample
from a variety of measuring instruments” 共Sedlack and Stanley
error. Nonprobability sampling methods do not permit the appli-
1992兲. Triangulation is the combination of multiple methodolo-
cation of sampling statistics. These types of samples can only be
gies in a study of the same subject area such that the multiple
assessed by subjective evaluation 共Kalton 1983兲.
sources indicate key parameters that can be ascertained by study
of all results. However, some populations do not permit probability sam-
Indeed, the capability to implement a strategy of triangulation pling due to the lack of a sample frame. The study population
means that evaluators must include in their research plan the abil- within the construction industry is typically such a case. There are
ity to use qualitative methods 共Patton 1989兲. Patton’s support of few lists of capital facility projects from which a random sample
the use of multiple and proven research methods in the explora- can be taken. In some cases, respondents want to remain secretive
tion of new areas of research has the apparent aim of minimizing for proprietary reasons. The construction industry is fragmented,
the chance of researchers making the common mistake of over- with many smaller companies versus a few large companies, mak-
simplification and overgeneralization. ing it more difficult to find representative companies and projects.
Research charrettes are typically very effective in exploratory Some projects are completed in-house and are never publicized.
studies. Combining the different methods used in the charrette Other projects are completed through negotiated contracts which
with other data collection strategies outside of the charrette are not advertised for bids and not part of the local permitting
provides a ready method to accomplish the triangulation needed process. There are some commercial databases available that at-
to conduct one of these studies. This can be accomplished by tempt to identify current construction contracts which could be
perhaps using specific case studies, structured interviews, statisti- used as a sample frame. However, these databases are expensive
cal database analysis, and so on in addition to the charrette to use and still are biased because they are missing many in-house
method. and negotiated contracts. Despite the obvious drawbacks in non-
probability sampling, the study population requires its use for
almost all construction best practices studies.
Sample Selection The research charrette uses the nonprobability sampling
One of the first steps in designing a research project is to define method focused on volunteer experts. Participants in the charrette
the population to be studied. The unit of analysis in construction 共some of which may have traveled long distances to attend兲 are
studies can be the project team, a group of individuals, a process generally invited based on past contacts with the researcher, rec-
step, a project, or a database of existing data. The entire popula- ommendations from the research sponsor or oversight team, or
tion of construction data in most cases would be too large and are actively involved in an organization such as a department of
diverse to study, producing questionable results. Conclusions transportation or a research consortium or trade group and re-
reached by a study of such a wide mix of project types would spond to an advertisement. Generally, the charrette will have cer-
provide overly general information that would not prove useful to tain parameters that are applied to screen the participants based
any specific subset of the population of project teams. As a result, on qualifications related to the study. For instance, the number of
the sample is generally narrowed to a subset, such as “project years experience that a person has been in industry, the types of

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 69

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
projects that he or she has been involved with, educational back- input at the beginning of an exploratory investigation. The rela-
ground and so on which allow the researcher to focus in the tively rapid manner that the charrette can be conducted, and the
proper area of assessment for the study. bisynchronous manner in which the researcher can gain insight
If the population of potential respondents is known, a random into the problem are unique.
sample could be chosen to participate in the charrette. This situ- The research charrette is not as effective in situations where a
ation is unlikely for the construction industry, but could happen rich data set is available for review, a national study is required
within very defined subsets. 共although it could be used to test the questionnaire兲, or where
The participants need to be able to focus not only on their own observation of methods or field procedures are dictated by the
organization’s short-term goals but also the overarching improve- study parameters. As previously discussed, the charrette should
ments to the construction industry as a whole. This separation of always be combined with other data collection strategies, such as
short-term views from long term issues needs to be addressed in field observation, database assessments, or case studies to ensure
the make-up of the data collection instruments. study triangulation.
Several charrettes may need to be conducted at different geo-
graphic areas to improve participation and participant diversity.
This dispersion will typically allow a more robust response Past Research Charrettes
set, but geographic differences cannot be inferred based on the
locations. As indicated in Table 1, the research charrette method espoused in
The fact that the research charrette uses a nonprobability this paper has been used on several research investigations. A
sample limits the ability of the researcher to infer broad claims brief discussion of each of these studies is given in this section to
beyond the sample surveyed. It also means that research char- illustrate the method’s flexibility, and its evolution; however ex-
rettes should typically be used as one component of a research tensive evaluation of the specific data sets collected and actual
framework. statistical analysis conducted is beyond the scope of this paper.
One example data collection effort will be provide later in the
Types of Analyses paper. Additional information and more detailed data analysis for
each charrette are located in the cited references.
As with any type of social data study, the types of analysis em- The first use of the research charrette method by the writers
ployed based on data received during research charrette would was during a study funded by CII in 1994. The research problem
include: was to gain insight into planning applications in industrial con-
• Statistical descriptive data summarizing averages, medians, struction projects. Based on data from a previous research effort
modes and frequencies of the participant’s demographic infor- 共Gibson and Hamilton 1994兲, and input from a research team
mation; composed of 17 industry representatives, a detailed draft checklist
• Pattern matching on written verbiage responses; for scope definition call the project definition rating index 共PDRI兲
• Compilation and analysis of comments and suggestions; was developed. Two research charrettes were used during the ef-
• ANOVA on stratified subsample results; fort to refine the checklist and obtain feedback from 54 partici-
• Regression analysis of dependent variables based on indepen- pants concerning the relative importance of the elements in the
dent variable input; and checklist. This activity used the respondent’s experience and in-
• Frequency analysis on ranges of response. sight based on their current projects. The respondents were a pur-
posive volunteer, expert sample. The developed tool and method,
which included relative values of the scope definition elements
When to Use Research Charrettes established in the charrettes, was subsequently tested on 40
projects, with some of the charrette participants providing test
Research charrettes are an effective method of gaining industry cases 共Dumont et al. 1997; Dumont and Gibson 1996; CII 1997兲.
input and insight in studies focused on determining best practices This tool has been used globally by many organizations for plan-
or understanding key parameters affecting project performance by ning industrial projects.
tapping the extensive expertise of volunteers. They are particu- A second series of research charrettes were used in an explor-
larly effective when the researchers are working with oversight atory study investigating project team alignment. Three charrettes
committees from the research sponsor. These committees are im- were held to narrow a list of 54 key alignment strategies which
portant in helping conduct the charrettes and also in developing had been identified by an industry oversight committee. Multivot-
the background research hypotheses data collection strategies for ing and small group breakouts were used to refine and narrow the
the charrette. list to the top issues affecting project team alignment during front
A major benefit of the research charrette method is the ability end planning. These issues and refinements were used in a sub-
to overcome time constraints around delivery of results. Research sequent series of project case studies and structured interviews
charrettes can typically be developed, organized, held, and ana- with company executives 共Griffith and Gibson 1997, 2001兲.
lyzed within a few months, if necessary. In 1999, the method was applied again on a project sponsored
Research charrettes should be part of the data collection by CII to develop a PDRI for building projects. Seven research
process for the research investigation, not the entire collection charrettes were used in this development effort, employing a very
effort. It is important to combine the data collected in the similar methodology to the first study cited above. Again, the tool
charrette with other methods such as case studies, follow-on vali- was refined and developed with input from industry participants,
dation of charrette findings or theories, source document reviews with the charrettes being widely dispersed around the United
or databases. Combined together, the different collection strate- States. The developed tool and method were subsequently tested
gies allow the researchers to triangulate on overarching critical on several dozen real projects as another source of data 共Cho et al.
findings. 1999; CII 1999; Cho and Gibson 2001兲. This developed tool has
The research charrette is also an effective method to provide been widely adopted by organizations in industry.

70 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 In 2001, the research charrette was used for a slightly different Workshop Agenda
purpose. TxDOT was concerned with the speed with which its
Shutdown Turnaround Alignment Review
projects were being delivered. A list of likely strategies were de- (STAR) Tool Prioritizations
veloped by the research team and then a series of charrettes were
held to refine this list, identifying whether these methods were Construction Industry Institute
feasible for TxDOT to pursue, and through a series of multivoting April 18, 2008
DuPont Chestnut Run Plaza, Wilmington, DE
exercises to recommend strategies that could be pursued immedi-
ately and others that required additional research. These six char-
8:00 am – 8:30 am
rettes involved 62 practitioners from six organizations 共and Continental Breakfast and Informal Networking

included 24 of the 25 regional construction management districts Introduction 8:30 am – 9:00 am

in Texas兲, and showed the flexibility of the charrette method  Introduce participants
 Research Project background
共Haas et al. 2003; Simon et al. 2002兲. This effort lead to several  Objectives and STAR Tool format
additional funded research efforts focused on scheduling, right-  Prioritization and scoring methodology
of-way acquisition, utility adjustment, and benchmarking.
Three additional studies have used the research charrette Review of STAR Tool and Category Prioritization 9:00 am – 9:45 am

method effectively in developing approaches to solving specific Break 9:45 am – 10:00 am

industry problems. In 2003, four research charrettes were used to
develop a risk management tool for international projects 共CII STAR Tool Category Prioritizations – Breakout Session 10:00 am – 11:30 am
 Category Reviews 1-9
2003; Walewski et al. 2006, 2004兲. In 2006, three charrettes were  Element Reviews
used to develop a risk management tool for highway projects
共Caldas et al. 2007; Le et al. 2009兲. Each of these used similar Break 11:30 am – 11:45 am
methods as outlined previously. The final study was recently Breakout Session Reports 11:45 am – 12:00 pm
completed and focused on developing an alignment and risk as-
sessment tool for shutdowns, turnarounds, and outages in the in- Lunch and Open Forum Discussion 12:00 pm – 12:45 pm
dustrial sector and is outlined in the next section. Prioritization Results 12:45 pm – 12:50 pm

Conclusion and Wrap-up 12:50 pm – 1:00 pm

Research Charrette Example Fig. 1. Example of research charrette agenda

In the fall of 2007 and spring of 2008, the writers used research
charrettes in order to prioritize and test the shutdown turnaround on-site analysis of the collected data, including comparison to
alignment review 共STAR兲 tool, a new planning tool for use in results from previous charrettes.
shutdowns, turnarounds, and outages 共STO兲 sponsored by CII. Each participant was asked to fill out a background datasheet
An initial survey was conducted of all CII members in 2006 at the start of each workshop. The datasheet included contact
concerning the extent of renovations projects within their project information, experience levels, and questions related to their im-
portfolio, including high level discussion of management con- pressions of how their company handles renovation and revamp
cerns and risks for these types of projects. One identified area of 共R&R兲 and STO projects. In addition, each participant was asked
importance and concern was the area of shutdown-turnaround- to provide information on a recent STO they had conducted to be
outages of industrial projects. Evaluation of these results led to used as their reference project for prioritizing the STAR Tool.
the realization that these types of projects should be studied more Table 3 shows information on the participant background data.
closely from a front end planning perspective as well as providing One of the participants used a $1.6 billion STO, a major refinery
a starting list of issues of concern 共Whittington et al. 2008兲. upgrade, as their basis for evaluating the STAR Tool. While
Along with the writers, the STAR Tool was initially drafted by the average project size listed is greater than $80 million, this
10 research team members 共industry practitioners兲 who specialize number was skewed by the one extremely large project. The
in the planning and implementation of STOs on an industrial site. median project value for all 67 projects within the sample was
Nine summary categories 共given later in Table 4兲 and 94 element $30 million and the nominated projects totaled approximately
descriptions 共not shown in this paper兲 were refined; these catego- $5.5 billion in total installed costs.
ries and subservient descriptions can be used for a joint assess- Each of the respondents was then asked to indicate the relative
ment of the multiple projects that typically converge during the importance 共priority兲 of each of the nine categories to the plan-
STO. ning process for STO projects 共on a scale of 1 to 100, with 100
Once the initial tool was drafted, a series of four research total priority points assigned兲 at a point just prior to beginning of
charrettes were held to prioritize the tool as well as to provide detailed design/construction. The respondents used their nomi-
input and suggestions for refinement. The agenda from one re- nated projects as a basis for these prioritizations. The details of
search charrettes conducted in this study is given in Fig. 1. The the method used to develop the prioritizations are fully described
charrettes allowed interaction between the industry professionals by Whittington et al. 共2008兲.
while also improving the STO planning tool. The workshops in- The nine STO categories are listed in Table 4 along with rel-
cluded an overview of the research effort, prioritization of the evant statistical results from the four workshops. Each respondent
critical elements in the tool under development, and written feed- was asked to give a relative value to the nine categories on a zero
back. The charrettes also included a detailed vetting of the indi- to 100 point scale, based on their career experience and their
vidual elements within the STAR Tool as well as assessment of nominated STO sample project. Given input from the 67 respon-
tool appearance, ease of use, and overall usefulness. At the con- dents, the priorities of each category are listed along with the
clusion of each charrette, the researchers were able to present standard deviation of each. The table includes the total number of
real-time prioritization results to the participants by performing respondent outliers from each category.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 71

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 3. Workshop Participant Information
Percentage of Percentage of Project cost
work in career work in career for STO
Results Years of Years of on R&R on STOs in question
N = 67 PM experience R&R experience 共%兲 共%兲 共$兲
Sum 1,323 1,107 5,471,100,000
Average 20 17 56 42 81,658,209
Minimum 2 2 5 5 250,000
Maximum 41 41 100 90 1,600,000,000
Standard deviation 10.32 8.56 28.74 24.35 203,331,135
Note: PM⫽project management; R&R⫽renovation and revamp; and STO⫽shutdown/turnaround/outage.

The research team classified the value of a priority outlier as team with immediate feedback which was used to modify the
plus or minus 1.5 standard deviations away from the overall approach 共including slight modifications to future charrettes兲 and
mean. The validity of the individual respondents was then scruti- a rich source of insight into how to use the tool. An excerpt from
nized. Any participant whose responses included three or more selected comments is given in Table 5.
priority outliers were classified as respondent outliers. Of the 67 In total, the workshops allowed the researchers to obtain 67
total participants, seven were determined to be respondent outliers unique perspectives representing 39 individual companies fo-
and were excluded for final prioritization calculations. This vet- cused on the STAR Tool and method. Although not reported here,
ting process is discussed in more detail in Whittington et al. other analyses such as charrette to charrette, owner to contractor,
共2008兲. and specific industry sector differences were also evaluated. The
In the final state of the tool, each of the nine categories has participants were able to interact with peers in similar industries
been assigned a transformed “value” or “prioritization” based on and learn novel approaches to common challenges 共Whittington
a total score for all categories of 1,000 points as shown in Table 4. et al. 2008兲. In addition to the data and input given earlier in this
This transformation was done to provide tool users a simple
section, the researchers were able to observe the interest and en-
method of assessing the readiness of their projects during the
gagement of the participants in the charrette setting. Nothing in
planning process. Each of the categories is assessed by the plan-
the observations and during the interactions at breaks and lunch
ning team in terms of its readiness based on a sliding scale. This
gave an indication that the research efforts were off base. To the
overarching assessment score gives a subjective method for de-
contrary, the enthusiasm and continued discussions provided
termining areas which need additional planning effort. A higher
total score 共up to 1,000 points兲 indicates poorer definition of plan- some assurance that this tool and data set were needed to help
ning preparedness. manage STO projects.
The structured workshop setting also allowed for facilitated In addition to research charrettes, this investigation also used
discussions in order to determine strengths and weaknesses of the the initial survey discussed above, 25 case study analysis of com-
new tool and planning method, a process that would not have pleted renovation projects using pattern matching software, and
been possible with traditional survey methods involving industry evaluation of an existing database of project data totaling over
feedback. In each of the workshops, from two to four small work $35 billion, focusing on practice use and project performance
groups 共5–8 individuals兲 was constituted and a facilitated discus- metrics. Results from these four data collection strategies allowed
sion ensued evaluating the entire tool and approach. Each work triangulation of key findings. As part of a follow-on effort to test
group reported back to the entire charrette group and a general the STAR Tool, it was effectively used on four STOs, with testing
discussion occurred. Comments were collected on a flip chart and still ongoing 共Whittington et al. 2008兲. The final report and tool
summarized. This input provided the researchers and steering were provided to charrette participants after publication.

Table 4. STAR Tool Categories and Statistics

Prioritizations Standard Priority outliers Relative importance
Category 共%兲 deviation 共exceeding + / −1.5 std dev兲 共0–1,000兲 scale
1. Design for constructability analysis 13.4 5.94 5 134
2. Permit requirements 6.3 3.55 4 63
3. Equipment and material status 14.0 5.48 7 140
4. Procurement procedures and plans 8.8 3.62 10 88
5. Project control requirements 12.0 4.99 8 120
6. Engineering/construction plan and approach 14.2 5.74 9 142
7. Shutdown/turnaround/outage requirements 16.3 6.66 6 163
8. Commissioning 7.3 4.29 8 73
9. Startup requirements 7.7 3.57 7 77

72 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 5. Selected Small Group Comments, February 2008
Comment category Specific issues
Overall observations • Tool will be usable early in planning, not just end of process
• Tool needs to emphasize safety more
• Leadership focus during this process is sometimes problematic
• Team needs to address timing of the evaluations with the tool
• Switch order of category 9. Commissioning with category 8.
Startup in the tool and so on
Engineering/construction plan/approach • Manpower density/loading should be addressed
• Craft timing is critical to this process
• Quality hold points should be established for dismantling and erection
of equipment
• A strategy for shared resources must be developed
• Quality assurance, inspections, cranes, safety, lay down/warehouse
scaffolding need to be emphasized
• Flexibility to adapt to market changes 共retention of labor兲 should be part
of considerations
• Setting priorities between projects that are within the STO should be part
of the consideration
• What are the STO drivers? Time versus cost versus quality and so on.

Recommendations for Conducting Research

Charrettes percentages of women and men currently participating in the
construction industry as participants in a research charrette. An-
When planning and conducting research charrettes for industry other possibility would be quotas of professionals from differ-
feedback, there are several important considerations to ensure that ent regions or jurisdictions to test relative understanding or
the process will obtain the appropriate results. Fig. 2 illustrates differences.
the process for developing the charrette and the method is dis- The research charrette also could be used as the first stage of a
cussed briefly below. At the conclusion of each process step noted larger effort focused on a probabilistic survey. In this scenario, the
in the figure and between each charrette, the academic researchers research charrette experts would be used to develop or test a
should discuss the results with the industry sponsors in order to questionnaire for wider usage in a random survey. CII used this
validate the previous steps and obtain buy-in for the subsequent technique in some of the studies cited earlier to develop a bench-
steps. The feedback will provide assurance that the plans, meth- marking questionnaire for wider contribution of project data from
ods, and procedures employed are the most appropriate to the its membership, although again not a random survey.
research objectives of the study. The number of participants depends on the intent of the study.
Most important in ensuring a successful charrette is the pre- In general, involvement of greater than 30 total participants will
work. The topic, objectives, issues for discussion, a developed ensure that sufficient input has been received to elicit consistent
and tested data collection strategy, and good background docu- viewpoints. This number is considered a large sample in statistical
ments need to be in place prior to the workshop. Involving the terms, and the writers have observed that the outcome of the
sponsor and any oversight committee to the research can greatly data collection effort will only change slightly once this number
enhance this preparation. has been reached and more data are added. However, the com-
The researchers need to develop a list of minimum criteria for ments and observations in larger total samples can provide a rich
attendees. Soliciting volunteer participants typically requires the source of additional insight if planned correctly. In terms of in-
assistance of individuals currently practicing in industry, and the dividual charrette sessions, from experience, the writers recom-
researchers must have a working relationship with a team of in- mend that no more than 25 individuals should participate in order
dustry practitioners to facilitate this step. When developing an to keep the session members engaged and to manage the effort
invitee list for the workshop, overbooking is necessary because as effectively.
many as 40% or more of the confirmed attendees will be unable The topic of the research must be interesting and the value of
to attend from personal or professional reasons. the workshop to the attendees must be communicated prior to the
Another possibility, discussed in the focus group literature invitation being issued. The “hook” in place to gain participation
cited earlier, involves randomly selecting participants from a is typically the opportunity to interact with other practitioners
group of nominated 共nonrandom兲 participants, thus mollifying concerning the subject, and the ability to gain insight into a topic
some of the bias, although not all. This was not done in the for professional growth. In almost all cases, the writers have pro-
studies discussed in this paper, because it is difficult to get vol- vided takeaways 共generally partially complete research products
unteers to participate in research studies in engineering and con- and findings to date兲 during the charrette for attendees and lunch
struction. In effect, all qualified participants were asked to attend as another enticement.
in order to get a large sample of diverse inputs. Findings from The meetings must be convenient in timing as well as location
the study should reflect the nonprobabilistic nature of the partici- for the proposed attendees. Therefore workshops may need to be
pation. Theoretically, a random sample of participants could be distributed across a broad geographic area. This also assists in
attracted. limiting the potential for a geographic bias in the data, although it
In addition, quota sampling could be used for specific types of does not obviate the fact that this is a nonprobabilistic sample.
studies. For instance if the study was focused on gender issues The industry practitioners can also assist the researchers as
in construction, the researcher could hope to match the relative workshop hosts and coordinators including help with creating and

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 73

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Develop industry
not only for several periods of informal networking among the
specific study participants but also periods of structured discussion. The re-
hypotheses and searchers should present a brief background on the history of the
methodology research topic as well as expected outcomes of the workshop.
The use of facilitated small group discussions in individual
break out rooms allows for a free flow of ideas between the in-
dustry practitioners and the research team. These groups should
Research No be made up of 6–10 participants. Unlike surveys and question-
charrette(s) End naires, this time allows for ideas to be discussed and the research-
warranted?
ers to interact with the respondents to clarify thoughts and points.
The agenda should allow time for feedback by these small groups
Yes
to the entire charrette to communicate developed thoughts and
ideas to everyone.
With industry input,
plan and develop Additionally, at least preliminary analysis of the collected data
multiple data should be presented to the workshop participants so that they get
collection strategies immediate feedback concerning how they responded compared
with participants at other workshops held a different times and
locations. Appropriate appreciations, long-term follow up, and
Develop charrette validation needs to be performed after the workshop has been
agenda and data
collection concluded and final results presented to participants. Although not
instruments done in any of the studies cited in this article, charrette partici-
pants could also be contacted after a period of time has elapsed in
order to gain additional insight.
Set up meeting
logistics and Lessons Learned
solicit participation

After conducting a variety of research charrettes, several lessons

have become apparent. Among these
Conduct • Performing prework and gaining industry input in preparation
research to hold the charrette is imperative to ensure that the time spent
charrette(s)
during the charrette is efficient.
• Providing clear and concise logistics information, including
premeeting information such as location, travel suggestions,
Collect, analyze, agenda, session objectives, and so on will make the session
and synthesize participants better prepared to participate effectively.
data • Using a local active session sponsor who can set up meeting
logistics and ensure that several participants can attend
helps in getting the appropriate number of participants to the
charrette.
Validate findings Produce research • Pretesting all data collection strategies prior to holding the
and follow up if report
charrette is essential. Again, sponsor representatives can pro-
necessary
vide a good test set. The first charrette held for the interna-
tional project risk study used an unworkable comparison and
collection strategy 共Walewski et al. 2004兲. This mistake caused
End the researchers to employ a remediation strategy to collect
additional information after the charrette was completed.
• Developing the list of invitees is not easy. It is difficult to
Fig. 2. Research charrette development process map
confirm that the participant’s backgrounds are appropriate for
the particular research topic ahead of time. The participants
comprise a nonprobability sample and researchers must state
communicating the specific workshop logistics such as material caveats in any data conclusions to account for this issue. In
and equipment needs, room layouts and seating arrangement, the some cases, participants must be removed from the data
presence of breakout rooms, food and drink availability, handouts, sample after completing the charrette if they did not meet a
nearby hotels, and parking and meeting location access/security screening parameter, such as years of experience.
arrangements for attendees and hosts. Packets of handouts for the • Employing the research charrette as part of a multiprong re-
attendees must be carefully created with defined takeaways and search effort is best. Case studies, source document review,
data collection forms which are given at the beginning of the additional questionnaires, follow-on use of the findings, and so
charrette. A useful method to ensure that the appropriate forms are on should be used to triangulate or validate the findings.
used and returned is to color-code the handouts. • Involving industry participants in the charrette gives the re-
On the actual day of the meeting, the strategy and agenda searcher access to follow-on data input and also sets the stage
for the workshop must be fully formed. The agenda should allow for buy-in and usage of the results by industry participants.

74 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Conclusions
Dumont, P. R., Gibson, G. E., and Fish, J. R. 共1997兲. “Scope management
The charrette has been used effectively for planning capital using the project definition rating index 共PDRI兲.” J. Manage. Eng.,
13共5兲, 54–60.
projects for many years. More recently, as related herein, research
Gibson, G. E., and Gebken, R. 共2003兲. “Design quality in pre-project
charrettes have proven to be an effective method for data collec-
planning: Applications of the project definition rating index.” Build.
tion in industry best practices studies. It allows researchers to Res. Inf., 31共5兲, 346–356.
gather data relatively quickly, collect valuable input from experi- Gibson, G. E., and Hamilton, M. R. 共1994兲. “Analysis of pre-project
enced practitioners, make excellent industry contacts, and gain planning effort and success variables for capital facility projects.”
insights with several collection strategies as outlined earlier in Rep. Prepared for Construction Industry Institute, Rep. No. 105, Con-
this paper. This method has some limitations, however, such as struction Industry Institute, Austin, Tex., 239.
nonprobabilistic samples and limitations on the type of data that Griffith, A. F., and Gibson, G. E. 共1997兲. “Team alignment during the
can be collected. pre-project planning of capital facilities.” Rep. Prepared for Construc-
Researchers should consider the charrette as one part of an tion Industry Institute, Rep. No. 113-12, Univ. of Texas at Austin,
overall data collection effort for industry best practices studies. Austin, Tex., 339.
Effective prework, involvement of sponsors, good logistics prepa- Griffith, A. F., and Gibson, G. E. 共2001兲. “Alignment during pre-project
planning.” J. Manage. Eng., 17共2兲, 69–76.
ration, and sound data collection strategies during the charrette
Haas, C., Gibson, G., O’Connor, J., Anderson, C., Somali, B., and Zhang,
provide the basis for its successful application. Z. 共2003兲. “Development and validation of a method selection tool for
expediting highway construction.” Rep. No. 4386-1, Center for Trans-
References portation Research, Texas Dept. of Transportation, 230.
Healey, E. H. 共1991兲. “Planning a library in one week.” American Librar-
Babbie, E. R. 共1992兲. The practice of social research, 6th Ed., Wad- ies, 22共4兲, 354–356.
sworth, Belmont, Calif. Kalton, G. 共1983兲. Introduction to survey sampling, Sage, London.
Board on Infrastructure and the Constructed Environment 共BICE兲. Kitzinger, J. 共1995兲. “Introducing focus groups.” BMJ, 311, 299–302.
共2003兲. Completing the “Big Dig”: Managing the final stages of Krueger, R., and Casey, M. 共2000兲. Focus groups: A practical guide for
Boston’s central artery/tunnel project, The National Academies, applied research, 3rd Ed., Sage, London.
Washington, D.C., 60. Le, T., Caldas, C., and Gibson, G. E. 共2009兲. “Managing risks in
Board on Infrastructure and the Constructed Environment 共BICE兲. the project development process for transportation infrastructure
共2006兲. Proc., Workshop to Review PATH Strategy, Operating Plan, projects.” ASCE Transportation Construction Management Conf.,
and Performance Measures, Division on Engineering and Physical Orlando, Fla.
Sciences, The National Academies, Washington, D.C., 108. Lennertz, B., and Lutzenhiser, A. 共2006兲. The charrette handbook,
Board on Infrastructure and the Constructed Environment 共BICE兲. American Planning Association, National Charrette Institute, Port-
共2007兲. Assessment of the results of external independent reviews for land, Ore.
U.S. Department of Energy Projects, The National Academies, Wash- Lindsey, G., Todd, J., and Hayter, S. 共2003兲. A handbook for planning
ington, D.C., 74. and conducting charrettes for high-performance projects, National
Caldas, C., Le, T., Gibson, G. E., and Thole, M. 共2007兲. “Identifying Renewable Energy Laboratory, Golden, Colo.
right-of-way requirements during the project development process.” Morgan, D. 共1997兲. Focus groups as qualitative research, Sage, London.
Rep. Prepared for Center for Transportation Research, Rep. No. Parasuraman, A., Zeitham, V. A., and Berry, L. L., 共1985兲. “A conceptual
0-5478-1, Center for Transportation Research, Texas Dept. of Trans- model of service quality and its implications for future research.”
portation, 224. J. Marketing, 49, 41–50.
Calder, B. 共1977兲. “Focus groups and the nature of qualitative marketing Patton, M. Q. 共1989兲. Qualitative evaluation and research methods, Sage,
research.” J. Mark. Res., XIV, 353–364. Newbury Park, Calif.
Carter & Burgess, Inc. 共2002兲. “A common sense approach to design.” Sedlack, R. G., and Stanley, J. 共1992兲. Social research: Theory and meth-
Water Eng. Manage., 149共10兲, 26–28. ods, Allyn and Bacon, Needham Heights, Mass.
Cho, C., Furman, J. C., and Gibson, G. E. 共1999兲. “Development of Simon, E., Gibson, G., Haas, C., O’Connor, J., Somali, B., and Zhang, Z.
the project definition rating index 共PDRI兲 for building projects.” Rep. 共2002兲. “Development of a tool for expediting highway construction
Prepared for Construction Industry Institute, Rep. No. RR 155-11, while retaining quality.” Rep. No. 4386-P1, Center for Transportation
Construction Industry Institute, Austin, Tex., 296. Research, Texas Dept. of Transportation, Austin, Tex.
Cho, C.-S., and Gibson, G. E. 共2001兲. “Building project scope definition
Stewart, D., and Shamdasani, P. 共1990兲. Focus groups: Theory and prac-
using project definition rating index 共PDRI兲.” J. Archit. Eng., 7共4兲,
115–125. tice, Sage, London.
Clayton, M., Kunz, J. and Fischer, M. 共1998兲. “The charrette test Walewski, J., Gibson, G. E., and Dudley, G. 共2004兲. “Development of the
method.” CIFE Technical Rep. No. 120, Construction Industry Insti- international project risk assessment 共IPRA兲 tool.” Rep. No. 181-11,
tute, Palo Alto, Calif. Construction Industry Institute, Austin, Tex., 366.
Construction Industry Institute 共CII兲. 共1997兲. “Pre-project planning tools: Walewski, J. W., Gibson, G. E., and Vines, E. W. 共2006兲. “Risk identifi-
PDRI and alignment.” Rep. No. 113-1, Construction Industry Institute, cation and assessment for international construction project.” Global
Austin, Tex. project management handbook, 2nd Ed., D. Cleland and R. Gareis,
Construction Industry Institute 共CII兲. 共1999兲. “Pre-project planning tool: eds., McGraw-Hill, New York, 1–17.
PDRI for buildings.” Rep. No. 155-1, Construction Industry Institute, Walker, J., and Seymour, M. 共2008兲. “Utilizing the design charrette
Austin, Tex. for teaching sustainability.” Int. J. Sustainability Higher Educ, 9共2兲,
Construction Industry Institute 共CII兲. 共2003兲. “Risk assessment on inter- 157–169.
national projects: A management approach.” Rep. No. 181-1, Con- Whittington, D., Gibson, G., and Cui, Q. 共2008兲. “Analysis supporting
struction Industry Institute, Austin, Tex. front end planning for renovation and revamp projects. Part 2.” Rep.
Dumont, P. R., and Gibson, G. E. 共1996兲. “Project definition rating index No. 242-12, Construction Industry Institute, Austin, Tex.
共PDRI兲.” Rep. Prepared for Construction Industry Institute, Rep. No. Yin, R. K. 共1989兲. Case study research, design and methods, Sage,
113-11, Univ. of Texas at Austin, Austin, Tex., 214. London.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 75

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Implementing Observational Research Methods to Study
Team Performance in Construction Management
Robert M. Leicht1; Samuel T. Hunter2; Chitwan Saluja3; and John I. Messner, M.ASCE4

Abstract: The evolution of research in the area of construction engineering and management has moved toward the use of quantitative
metrics and research methods for identifying and evaluating the impact of events or procedures on the construction process. While
quantitative research methods can be very important for answering certain research questions, there are important reasons for adopting
qualitative or mixed methodology studies to gain a better understanding of complex phenomena. This paper explores the use of obser-
vational studies and their potential use within the field of construction research. Observational studies can provide answers to “what”
phenomena occurred, particularly when people are involved in a process, along with gaining insights into “why” the phenomena occurred.
This paper discusses two types of observational studies, structured and unstructured, and provides a procedure for their implementation
within construction research. To clearly demonstrate the methodology, a case study focused on the investigation of two different versions
of an educational simulation application, the virtual construction simulator 共VCS兲, is used to illustrate the benefits and challenges of
implementing mixed methodology observational studies. The case study involved the video recording and analysis of interactions between
student team members when using the VCS application for a construction sequencing task. The video recordings were analyzed, and
important insights were identified, both qualitative and quantitative. Through content analysis, it was determined that the improvements
made in a new version of the VCS application were beneficial, and the detailed observational studies identified insights into why the
revisions in the application yielded improved results. This case study details the steps and considerations involved in planning an
observational study, as well as the benefits and challenges that researchers may encounter when using observational research
methodologies.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000080
CE Database subject headings: Research; Methodology; Construction management; Productivity; Teamwork.
Author keywords: Research methods; Observational studies; Content analysis.

Introduction Since these historical uses, the methodology and application of

observation as a form of research has grown and developed into a
The use of observation as a means of scientific research is in regular practice in many areas of research ranging from docu-
many ways one of the oldest methods recognized. Darwin used menting physical symptoms of diseases to behavioral studies, to
sketching and personal observation of Galapagos tortoises as the studies of communication. Developments within computing and
backbone for his Origin of Species 共Darwin 1859兲. Newton de- audio-visual technology have led to the utilization of better cam-
veloped the concepts for the laws of universal gravitation from eras, microphones, and software for capturing activities for obser-
his own “observation” when hit by a falling apple 共Conduitt vation as well as more thorough and accurate analysis
2006兲. Archimedes used simple observation for determining con- capabilities. The value of the use of observational studies, using
cepts of material density through his “Eureka” moment in a bath these new tools, lies in the inherent mixing of the methods. The
tub 共Biello 2006兲. The use of observation is indelibly linked with use of observation is inherently qualitative, but by integrating
most major advances in science. content analysis of the video it provides the ability to generate
quantitative data to support the inferences from the qualitative
1 observation. While the use of observational methods is not un-
Graduate Research Assistant, Dept. of Architectural Engineering,
Pennsylvania State Univ., University Park, PA 16801 共corresponding au-
known in construction and dates back to work on themes such as
thor兲. E-mail: [email protected] worker efficiency 共Taylor 1903兲, the use of these methods is fairly
2
Assistant Professor, Dept. of Industrial Psychology, Pennsylvania rare in construction research. Also, the use within construction is
State Univ., University Park, PA 16801. focused on simple tasks like that of Taylor, which focuses on
3
Graduate Research Assistant, Dept. of Architectural Engineering, fairly simple repetitive tasks rather than more complex and dy-
Pennsylvania State Univ., University Park, PA 16801. namic issues such as means of effective collaboration, interdisci-
4
Associate Professor, Dept. of Architectural Engineering and Engi- plinary teaming, or the use of new technology in construction.
neering Design, The Pennsylvania State Univ., University Park, PA With recent trends in the delivery of projects focusing more on
16801.
integrated delivery of facilities, means of closer collaboration,
Note. This manuscript was submitted on August 8, 2008; approved on
April 2, 2009; published online on April 30, 2009. Discussion period
and the use of technology for virtual teaming, this methodology
open until June 1, 2010; separate discussions must be submitted for indi- could potentially move to the forefront in research related to
vidual papers. This paper is part of the Journal of Construction Engi- teams and technology as these tools become more widely used in
neering and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, the industry.
ISSN 0733-9364/2010/1-76–86/$25.00. This paper demonstrates an overview of the different manners

76 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
in which observational studies may be used, citing examples to Quantitatively Coding Qualitative Data
clarify the unique implementation and variety of uses. Following
this background and explanation, a case study of implementation Termed mixed-methods research, current ethnographic studies are
is presented to clearly demonstrate the steps and the potential for often characterized by collecting qualitative data and quantifying
use in construction research contexts. it in such a way as to allow for the application of traditional
statistical techniques 共Plano Clark et al. 2008兲. Better character-
ized as a quantitative analysis of qualitative data, mixed-method
approaches have a number of unique advantages over the use of
Use of Observational Methods in Social
qualitative and quantitative approaches alone. For example, al-
Science Research
though a quantitative researcher may observe a correlation be-
tween variables X and Y, supplementary qualitative information
It is difficult to determine precisely when observational research
may provide the researcher with the context to better understand
methods were first used 共Wax 1971; Bauer et al. 2000兲. These
the processes involved in that relationship. On the other hand,
techniques are, in many ways, natural steps in the research pro-
applying traditional quantitative methods to qualitative data al-
cess. Simply stated, if we are curious about why something oc-
curs we are likely to seek observations of that phenomenon to lows researchers to more strongly establish generalizability to
more fully understand it. Thus it would appear more fruitful to other, similar contexts. It is not difficult to see how gaps between
consider when the observational method gained formal recogni- theory and practice are bridged by providing both numbers and
tion as a research tool rather than when it was initially employed. context to more completely understand a given phenomenon.
In doing so, we find some early use in studying the London work- The use of observational methods within these contexts is di-
ing class during the 1880s. The technique became more formal- vided into two main categorizations, usually identified as struc-
ized, however, in the United States—particularly in Chicago tured and unstructured observation 共Yin 2003兲. Structured
during 1920s and 1930s 共Gill and Johnson 2002; Wax 1971兲. It observations rely on frameworks of predefined actions, discussion
was during this period that a group of sociologists formed the content, or even body language to fit the activity within the vari-
Chicago School and trained some of the most influential and im- ables and scope of the research question. By using an existing
portant qualitative researchers. Termed ethnographers, these re- framework or developing a specific framework to track observed
searchers sought to understand the Chicago working class by activities, the researcher has chosen the architecture around which
examining them in their own environment—that is, observing they are fitting the hypothesis and they utilize the data from the
them in their natural context. Ethnography has gone through a observations to create the correlations between the pieces. Buch-
number of changes since its formal inception. In fact, Denzin and holz and colleagues observed 17 ironworkers for a period of 13
Lincoln 共1994兲 suggest that there are five historical stages the days during construction of a tunnel construction project in Bos-
approach has gone through: 共1兲 traditional 共1900–1950兲; 共2兲 mod- ton 共Buchholz et al. 2003兲. They were focusing on ergonomic
ernist 共1950–1970兲; 共3兲 blurred genres 共1970–1986兲; 共4兲 crisis of concerns of specific tasks performed while installing the reinforc-
representation; and 共5兲 the post modern challenge/the present. No- ing in preparation for the concrete placement. The coding was
tably, Denzin and Lincoln suggest that recent stages were brought performed live with a very detailed coding framework focused on
about by technology such as video cameras and tape recorders. It posture, activities, tools, and handling of materials 共PATH兲. The
should be noted that the distinctions and current status of these detailed structured framework created for the study used exact
stages has been challenged by some 共e.g., Atkinson et al. 1999兲. calculations about the loads, material weights, and tool use tech-
What is clear, however, is that observational methods have had niques to incorporate how the workers physical forms were dis-
both high and low points with regard to their approach and favor tributing the stresses of the construction tasks to their backs, legs,
with researchers. At present, it is safe to say the method is being and arms. The detail and structure of the framework enabled the
welcomed by an emerging group of researchers eager to under- researchers to relate the data collected about times and frequen-
stand their phenomena in a more complete fashion. cies of activities to the safety measures and longer term physical
Qualitative methods such as observation have been adopted by impacts on the workers of performing specific tasks. In this case
a number of groups within the social sciences including, most the researchers had very specific concerns regarding body posi-
notably, the sociologists out of the Chicago School mentioned tion and movement. By coding the field durations and frequencies
above. Other groups include anthropologists, who have often em- of the various construction activities, the researchers could draw
braced the “naturalistic” component of ethnography with some conclusions about these variables in the changing setting typical
choosing to wholly embed themselves in the culture they are to construction, which previously had been considered too dy-
studying in an attempt to gain a comprehensive understanding of namic a setting for this type of study as compared to the typical
that culture as well as limit their own influence, or bias, in their factory floor studies.
observations. Ethnography is used to some degree in psychology, In contrast to structured observations, an unstructured obser-
most notably in the study of children as well as the workplace vation utilizes no preset framework. The concept driving unstruc-
共Mintzberg 1973; Gill and Johnson 2002兲. Historically, however, tured observation is to have the researcher enter into the
many areas of psychology have been skeptical of the method observations with no preconceptions regarding the expected out-
viewing it as unscientific and lacking empirical strength. Instead, comes. The researcher simply records and notes the behaviors in
experimental and survey-based techniques have been emphasized question and the trends in the behaviors will emerge with time.
in an attempt to obtain perceptions of scientific rigor. This nega- Along with these, nonfrequency techniques and contingency
tive perception has begun to wane, however, as researchers begin analysis, coding whether or not something is used, rather than
to dissolve the distinction between “softer” qualitative research how often, or the absence or presence of an attribute may be more
and “harder,” quantitative research. In fact, most researchers significant than the relative frequency of other characteristics. In a
today are open to and understand the benefits, as well as costs of, study of lab processes in science classes, Dov and colleagues used
combining the two methodological approaches when studying a unstructured observation of classrooms to identify the different
phenomenon. processes used in conducting lab activities in science classes 共Dov

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 77

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
2005兲. The identification of seemingly unique activities offered examples 共good and bad兲 of what they will be rating. The second
insight into areas that set apart those teachers and their methods is a series of rating exercises where raters make ratings indepen-
from the typical process. dently, share their results and rationale for why ratings were made
followed by another round of ratings, and discussion until ratings
are consistent among judges. This requires some guidance by re-
Conducting Impactful Observational Research searchers to ensure that one judge is not dominating the discus-
One of the primary reasons qualitative methods have not been sion but also to ensure that judges are sharing their rationale for
embraced by all researchers is the types of bias and subjectivity their ratings. The latter is essential for creating shared mental
associated with such approaches. That is, many have argued that a models among judges—each judge must understand what the
researcher’s personal bias plays too great of a role in gathering, other is thinking during the rating process. Nearing the end of
synthesizing, and analyzing qualitative data 共Hammersly and training and prior to conducting data analysis, it is still essential
Gromm 1997兲. It should also be noted that, in contrast, some to assess reliability among judges via the application of statistical
researchers have countered this argument, suggesting that this techniques. Approaches vary, but typical data analyses include rwg
bias is evident in all types of research and that qualitative studies 共James et al. 1984兲 ICC1 共Bliese 2000兲, and more basic forms
are unfairly criticized. We concede that this may in fact be true, such as percentage of agreement. The reader is invited to view
but contend that studies are significantly improved to the extent Bliese 共2000兲 for a thoughtful discussion on inter-rater assessment
that bias is reduced and some degree of rigor is employed. For- as well as Bernardin and Buckley 共1981兲 and Woehr 共1994兲 for a
tunately, the reduction of bias is conversely related to two critical more specific discussion on types of rater training.
components of research: reliability and validity. We continue Validity. If reliability is viewed as consistency of a given phe-
then, with a discussion on how to maximize both and, in turn, nomenon, validity occurs when a phenomenon operates, in actu-
minimize investigator bias. ality, as we propose it does. It is important to note that validity is
Reliability. With regard to the social sciences, reliability refers not an all or nothing distinction—rather, it must be viewed as a
to the consistency of a given phenomenon or observation. For matter of degree 共Messick 1995兲. Validity has been conceptual-
example, measures of intelligence are created in such a way that ized in a variety of ways, although for the present discussion it is
if we give an individual an IQ test at Month 1 and then again at useful to break it into two basic forms: test validity and what
Month 6 we would expect roughly the same score each time. might broadly be termed design validity 共Cook and Campbell
Similarly, if a scale is intended to measure job satisfaction and 1979兲. Test validity refers to scales used in measuring psychologi-
each item on the scale is scored the same way by each employee cal phenomena. For example, a measure of intelligence is said to
共with reasonable variation among participants兲 that scale is said to be, at least partially, valid if it is related to other established tests
be reliable—each item produces a score similar to the other items. of intelligence. More central to the present discussion, observa-
Central to our discussion, reliability is often viewed as inversely tional research is valid if it is free from design confounds or
related to random error. That is, scales and measures that are biases introduced by the researchers—that is, it may be viewed as
reliable are said to have less error than those which are unreliable. having design validity. Referred to as internal validity, Campbell
Thus, in understanding how to analyze data we seek to minimize and Stanley 共1963兲 note that there are at least eight threats to
error variance or use approaches that result in reliable data. It is internal validity—a list expanded by four in later works by Cook
important to keep in mind, however, that reliability is not equiva- and Campbell 共1979兲. The reader is invited to view these manu-
lent to validity but rather is a necessary precursor to it. scripts to gain a more complete understanding of bias in research.
Reliability and its assessment come in many forms. One of the Of these threats, the most impactful is confounding or system-
simplest is test-retest reliability where scales are given at multiple atic covariation among events or manipulations that precludes
times and assessed for consistency between time points. Others causal interpretation on the part of the researcher. Consider, for
include internal consistency and split-half reliability where scale example, a researcher that is observing a group of students de-
items are compared to one another with the goal of determining signing a new cell phone. Let us imagine that the researcher ob-
consistency among items purported to measure the same con- serves the group every Friday. Let us also imagine that the design
struct. The type of reliability most applicable to observational team uses a new technology to help them in their design efforts on
research is inter-rater reliability. That is, assessing the extent to Fridays. In this scenario, it is impossible to determine if a team’s
which judges make the same assessment about a given phenom- success is due to the use of new technologies or due to being
enon or event. For example, if we ask three judges to rate the observed, as both occur simultaneously on Fridays. Thus it is
quality of a given product we would hope to find consistency essential that observations are conducted in such a way as to not
among their rating and if we did, this would suggest adequate systematically covary with events they are observing.
inter-rater reliability. Further, by finding adequate reliability, we In addition to internal validity it is also important to briefly
would be more confident that our ratings minimize the potential discuss external validity 共Cook and Campbell 1979兲 as it applies
for error. On the other hand, if the judges’ ratings were largely directly to the use of observational methods. More precisely, ob-
divergent, it would be unclear as to what we were, exactly, mea- servational research has been criticized for lacking generalizabilty
suring. In short, we would have less confidence in our data and, 共i.e., external validity兲 or the capacity to extend findings beyond
ultimately, our results. those events and participants observed in the study. Although this
How then, do we maximize reliability and therefore reduce criticism may be true for isolated case studies, by applying a
error in observational research? The simplest and most effective mixed-method approach to observational research a greater de-
answer is training. Raters must be exposed to the phenomenon gree of external validity may be obtained. For generalizability to
they were chosen to assess and gain experience in making judg- be maximized, it also is important to choose a sample of partici-
ments on that phenomenon. The end result is the formation of pants that are representative of the population to which we hope
shared mental models—or similar perspectives on rating the be- to generalize. In doing so, we can be more confident that our
haviors or events at hand 共Mathieu et al. 2000兲. This may be findings are applicable to the broader population and not just our
accomplished several ways; the first is providing them with clear select sample.

78 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Fig. 1. Image of the Studiocode software interface, showing the video in the upper left, the framework coding window in the upper right, and the
content analysis timeline along the bottom

Analyses and Results Computer Software for the Analysis

of Audio-Video Recordings
The type of analyses employed in observational research varies as
widely as the research questions driving them. For example, re-
searchers interested in comparing the mean performance of teams The use of audio-visual recording devices allow for the capture of
using technology versus those who do not will likely employ an an activity under study. Once the activity is recorded, greater
ANOVA based analysis model 共Tabachnick and Fidell 2001兲. On rigor or exactness in the content analysis can be achieved. The use
the other hand, researchers interested in assessing the relationship of video creates the ability for the coder to return to periods of
between frequency of technology use and performance are likely activity. Absent the use of audio-video recordings for coding, ob-
to use correlation or regression based models 共Aiken et al. 2003兲. servers typically identify activities at set intervals during direct
Going further, as research questions become more complex re- observation. A recent example of this type of observational cod-
searchers may employ the use of structural equation models or a ing was performed by Gopinath who used 30 s intervals to iden-
analytic tool gaining popularity due to its ability to appropriately tify types of conversation used in regular construction job
handle nested data structures, hierarchical linear modeling progress meetings to demonstrate inefficiencies in communication
共Raudenbush and Bryk 2002兲. 共Gopinath and Messner 2004兲. This remains necessary at times
The driving force behind analytic decisions must be the re- due to limitations in recording activities. The use of audio-video
search question at hand. Factored into the decision, however, are recordings offers the ability to code entire durations of a discus-
more pragmatic issues such as sample size and nature of the study sion topic or activity. Also, by using recordings there is the op-
hypotheses. For example, if the research is exploratory, the re- portunity to clarify or refine the coding framework by direct
searchers should note their results and conclusions as such 共Tukey comparison of the coded outcomes between multiple observers. In
1977兲. On the other hand, if the researchers are testing explicit the setting of a live coding, the results would only contain the
hypotheses, more rigorous standards of analyses are often em- quantitative data. Using audio-video recordings for coding allows
ployed. Finally, it is important to blend both quantitative and for the exact time of each occurrence to be recorded and when
qualitative elements when presenting results from mixed-method discrepancies occur between coders, the occurrence can be re-
observational research. For example, in their study of leadership viewed, allowing for the removal of errors, clearer communica-
Mumford and colleagues 共Mumford 2006兲 often conducted dis- tion by using challenging examples to train coders, and also the
criminant function analyses on their coded results—a wholly ability to revise the coding framework in cases of ambiguity.
quantitative endeavor. In explaining these results, however, the The introduction of specialized computer software has allowed
researchers also provided quoted material from the leader biogra- for greater efficiency in coding and analyzing audio-video data.
phies to provide concrete examples as well as general context for Several of these software applications allow users to create cus-
their findings. Although this approach is time-consuming, the tom frameworks for coding categories, as well as labeling func-
benefits of a more complete research picture are well worth the tions to correlate data directly, along with shortcuts to make the
mixed model effort. coding process more user-friendly 共Gyorke 2006兲 共Fig. 1兲. Along

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 79

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
with the efficiencies in coding the content, the applications typi-
cally have added functionality for searching the results, creating
matrices to organize the results, exporting selected video for iden-
tified categories, and in some cases even incorporate statistical
packages for analysis of the resulting quantitative data. The major
challenge of performing content analysis is removing bias from
the results inherent in the process from the coders seeing what is
expected. The use of recorded sessions allows for verification of
the coding 共Scott and DeSanctis 1992兲 and the newer software
applications improve the efficiencies and exactness of this pro-
cess, resulting in minimization of the bias through a more thor-
ough and rigorous process. Fig. 2. Photo of a student group in the Immersive Construction
共ICon兲 Lab using the VCS

Illustrative Example of Using Observational Time

Studies in Construction Research four member teams. When comparing the outcomes of the use of
the VCS application for the education exercise, there was an ob-
Time studies have been used in construction research, though servation that the shift to the new user interface with slightly
only to a small extent of the potential. Several research studies different functionality including a slightly improved user interface
investigating worker productivity has used time studies to track for entering activity and sequence data, led to student groups
the time spent performing particular tasks in an effort to identify spending a much shorter time period working on the project, and
field challenges, differing conditions, or to identify means for also yielded a decrease in the average project grade by 3%, which
improving worker efficiency 共Gillbreth 1912; Taylor 1903兲. was not statistically significant. This led to the fundamental ques-
Though the nature of capturing quantitative data are seemingly in tion regarding “why” the amount of time varied significantly for
line with the proposed methodology, the use of time studies often the groups along with quality differences in the final grade. Each
does not rely on the qualitative data available but instead uses group was video-taped during their exercise and this data pro-
standard time and efficiency measures as a basis of comparison. vided an opportunity to investigate through observation whether
Thus, these studies are only partially benefiting from the available the interface and functionality were mitigating factors in the shift.
opportunities. More recently, researchers studying the use of In the first offering of the educational module in 2006, two
three-dimensional 共3D兲 and four-dimensional 共4D兲 software have different software interfaces were compared to identify changes in
used observational studies to demonstrate improved efficiencies the process used to develop their sequence of construction activi-
of communication. Liston used observational studies to quantify ties. The students were asked to develop a short interval produc-
the time spent in “less productive” conversations when changing tion schedule for the structural sequence of the MGM Grand
from standard two-dimensional project documentation to a 3D renovation project in Las Vegas. For the study there were 10
computer-aided drafting 共CAD兲 model and 4D CAD simulation of participating groups in the course, which was an upper level con-
the construction process 共Liston et al. 2001兲. The study was one struction project controls class in the Architectural Engineering
of the first within construction to use the qualitative observations curriculum at Penn State. All student groups were developed
about the seemingly greater efficiency of communication when through random assignment, and they consisted of three or four
using such technology and support it with the quantitative data members. The study was designed where one set of 5 groups used
developed through coding communication using time studies of the newly developed VCS, and another set of 5 groups used a
project meetings. In similar efforts, Wang and colleagues devel- commercial scheduling and 4D CAD application for developing
oped an educational module for improving sequencing knowledge their construction sequence. In the analysis of the groups using
through the use of 3D CAD and 4D software 共Wang et al. 2007兲. the two different software packages, videos were taken of each of
Wang performed an experiment and then coded student groups the groups working for a continuous period within the Immersive
using a new 4D simulation module to demonstrate shifts in time Construction Lab at Penn State. Each group was given up to 4 h,
spent during different stages sequencing design during the devel- with all groups finishing in less time. In Fig. 2 a group is shown
opment process. using the first version of the VCS in Penn State’s Immersive
To clearly present the use of observational studies with the use Construction Lab. In the initial results the study showed a change
of content analysis for evaluation, a case study was performed. in the time spent in different stages of the sequencing process,
The use of this case study allows exploration of issues on how to with less time spent trying to understand the problem and more
implement this methodology, from defining what to measure, to time performing analysis and creating the sequence.
developing the coding framework and reliability measures, by For the 10 videos recorded during the group meetings, the
addressing the methodology steps point by point throughout the average duration of the videos in the first offering was 3 h, while
case study example. This study is an outgrowth of the work per- in the second offering it was 50 min The set up of this particular
formed by Wang at Penn State evaluating the use of a virtual study is simpler than most because the data, in this case videos for
construction simulator 共VCS兲 using an education module for con- the groups using the different software packages, was already
struction sequencing. In an ongoing research project, a software collected. Since the collection of the data was set up as an explor-
change was made in the development of successive versions of atory study for the use of the software, there is little room to
the VCS interface. In the 2006 course offering, the students used control for the variables but they still need to be clarified for the
an initial version of the VCS created in the Deep Creator game purposes of identifying limitations of the analysis and use of the
engine, while in 2007 the student used a new interface developed outcomes. All of the class members were either in the final year of
in the Irrlicht open source rendering engine. In both offerings the undergraduate work in Architectural Engineering or graduate stu-
course had 10 groups of students randomly assigned to three or dents. This gave the population as homogeneous a range of par-

80 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
ticipants as possible given the experience and knowledge about model and not how they actually used the software interface.
the scheduling exercise they were undertaking. The participants Since the visual information in the model was identical between
were all 22–24 years of age and in their final year of the Archi- the two study groups, the information in the outcomes would not
tectural Engineering program at Penn State, over 90% of each be related to the hypotheses or allow for conclusions of value
group was Caucasian, and 70% of each group was male. about the interfaces. The reason behind this seemingly tangential
discussion about the case study is not the details of the framework
used, but the designing of the framework and why was one ap-
Research Question
proach chosen over another. Similar to design intent explanations
Having identified the dynamics of the exploratory study, a ques- architects and engineers use to explain facility systems at early
tion to investigate using observational methods is necessary to design stages, the intent of a coding framework should be clear
move forward with identifying a framework for analyzing the and should consider the type of results the analysis will return.
data. Potential differences were identified along two main areas of
interest. First, since the change between the two groups in the
study was believed to be mainly a more user-friendly interface, Step 2. Distinguishing between Categories
the interaction between the student groups and the software inter- With the main and subcategories for the framework identified, the
face was a suitable starting place for trying to identify differences definitions for the subcategories were developed to clearly com-
in how they were used. The intention in studying this was to see municate the differences in interaction. The definitions should be
if there are clear differences in the time or manner of use of the simple, concise, and clear to minimize ambiguity. For the soft-
software.
ware interaction, the three subcategories were developed and de-
The other potential area of difference between the two groups
fined as follows:
is in the dynamics of the groups themselves. This area of research
1. Authoring content: using the software interface to create or
could potentially be a very challenging area to study, and the
document information or content related to the task at hand,
intention with this exploratory study was to see if there is some
e.g., typing in activity durations or selecting geometry in the
evidence to show that the interface is the main reason for the
model to create groups;
change in the results. The interest in studying this area is to iden-
2. Navigating the model: using the software interface to move
tify if there are noticeable differences in the group dynamics that
through the model or information contained therein either for
could be a greater factor than the software interface.
general exploration of the information or to locate specific
We employed the guidelines suggested by Holsti 共1969兲 for
information, e.g., rotating the model to view it from the top
using content analysis. Although these guidelines do not fully
or side; and
eliminate potential biases, they do minimize the possibility that
3. Analyzing outcomes: using the software interface to manipu-
the findings will reflect the analyst’s subjective bias rather than
late the data to synthesize new information/results or to cre-
the actual content.
ate new ways of viewing the data. Examples include
performing calculations to get results or developing an ani-
Step 1. Identify the Categories mation, e.g., reviewing a 4D simulation.
For the discussion among the team members, the category was
Within the use of the VCS interface, two main categories were
the actual team members, rather than separate categories of dis-
chosen to compare the two groups of participants, reflected in the
cussion. The members were generically labeled 1–4, starting
core research questions: 共1兲 interaction with the software and 共2兲
from the left side of the video and continuing to the right and
discussion between the team members. For software interaction,
clockwise, depending on how the team members orient them-
the subcategories were broken into: authoring content, navigating
selves. The breakdown of the discussion in this case is more a
the model, and analyzing outcomes. For team member discussion,
question of what discussion constitutes discussion worth coding
the subcategories were tracking the time each team member
for analysis.
spends discussing the task. Along with tracking the team member
discussion, the lab facilitator was tracked as well. Since the study
used software which was new to the student groups, there was a Step 3. Identifying the Content Unit
facilitator available to answer questions or help if there was dif-
ficulty using the interface. This was coded like the team discus- The content unit can vary with the intent of the coding, and the
sion, by tracking any time the facilitator spoke with the team. manner of breaking it down impacts the type of analyses which
Although these categories may appear simple, they were dis- can be performed at the completion. For the discussion content of
cussed in detail to identify which actions by the students could at the groups, the focus was not on the number of contributions an
the end of the process indicate the differences in value of the two individual made but was on the amount of time each team mem-
interfaces. ber spoke. Under this scenario the definition only needs to cover
Another approach was considered for the case study analysis what meets the minimum standard for the speaker to be coded as
wherein the coding would track the number of specific references part of the discussion. To consider a verbal statement as a contri-
to the model. As such, it would be necessary to code references to bution to the discussion by one of the group members it needed to
the model, as well as determine how to code physical gestures, meet the following two criteria:
verbal references, and when both are used together. The question 1. The contribution, either a question or statement, needs to
which ruled out the use of this approach was what information constitute a whole statement and complete thought. Started
will we have when complete, and will we be able to use that but not completed statements and simple one word com-
information to draw conclusions? In this case, the results of the ments, such as yes or why, were not coded as contributing
content analysis would produce frequency information about the time.
discussion references to the model, but the information would 2. The contribution needs to be related to the topic at hand. The
only convey how often they used the visual information in the participants in the study are working for an extended period

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 81

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 of time and it is expected that tangents to the conversation using a 95% standard which had been predetermined as the level
will take place. expected for the framework, as shown in Table 1.

Step 4. Clear Reasoning behind the Inferences Rigorous Coding versus Data Sampling
The reasoning in the background of the coding process constitutes One of the challenges which was identified and reasons why the
the underlying backbone of the research outcomes. It was neces- recent developments of technology has help invigorated observa-
sary in the development of the definitions and content unit to pilot tional studies is the use of data sampling to represent the true
the coding process using the videos from the study. The piloting events which take place in an observational setting. In live obser-
process allowed for clarification and examples to be used to help vational studies, typically, the observers will note activities as
train multiple coders and also for iterations and clearer reasoning regular intervals, similar to the way a time lapse video shows
behind the framework definitions to be developed. The reasoning steps in activity without showing all of the activity between
and clarifications developed during this process was, and usually frames. However, as with any study there is efficiency in the
should be, incorporated into the category explanations to demon- process if a representative sample can be used to represent a
strate the intent of the categories. larger population. Using video enabled software to code allows
The interaction with the software interface was coded to iden- for very exact time to be measured for an activity. However, there
tify if the design or the user friendliness of the interfaces affected is an intense labor effort involved in generating the quantitative
the educational outcomes. By tracking the different manners of its data from the observational video. The challenge then is to iden-
use and their times, we could determine if the two software inter- tify when a representative sample can be used, and when coding
faces were used in a similar manner or in a different manner, and of an entire study is worthwhile.
also were able to identify if that was the factor which affected the For the undertaken study, there were two areas of interest
project outcomes. raised and each had a correlating category within the coding
framework. It was decided for the study, to undertake a thorough
and complete coding of the software interaction because the soft-
Ensuring Reliability
ware is the known difference between the two participant popu-
There were two issues identified as central to ensuring impactful lations. When considering the use of the software, if there is a
research. The first was validity and the elimination of covariation difference in how the groups utilize the software then a sampling
in the collected research data, which is ensured through the design approach would be significantly more likely to miss significant
of the research task. In the case study, the research is using data data about how the groups used the software. From the prelimi-
from previous studies and though covariation is minimized nary studies performed by Wang, it was shown that using the
through the use of similar pools of participants, there is the still software shifted the process for undertaking sequencing. If a simi-
some small amount of error created from the use of different lar shift occurs from the first to the second version of the soft-
offerings of the course and dynamics of the participant pool ware, it would be challenging to choose truly representative
which cannot be controlled. The second issue raised was the re- samples of the groups’ activities.
liability of the content analysis for ensuring reproducibility of the On the other hand a sampling approach was taken for the
results, often referred to as inter-rater reliability. The method used group discussions. The interest in the group discussions was more
identifies the level of coding errors among the various coders as exploratory and rigorous coding of the issue would be time inten-
defined in Eq. 共1兲 sive with the possibility of no valuable data. For this reason ran-
dom sampling of the discussion was coded to see if the data
Level of coding error suggested substantial differences and value in further analysis or
more detailed coding. For the discussion, the interest is not the
Time in agreement
= 共1兲 process of use as it is with the software interface, but purely in the
共Time in agreement + Time in disagreement兲 dynamics of the group. The group dynamics are unlikely to
The act of coding a video for this level of framework took an change to such an extent that random samples of the discussion
individual coder approximately three to four times the duration of would be largely different than considering the whole duration of
the video being coded which made testing the reliability of the activity. To sample the discussions it was decided to code five
framework for the entire data set unrealistic for an exploratory
study until there is some evidence to support the hypotheses. All
of the chosen coders had experience with the sequencing activity, Table 1. Coder Reliability Levels for a Sample Portion of Video Obser-
with 4D modeling techniques, and were trained in the coding and vation
use of the framework. Determining an appropriate level of inter- Coder 1 Coder 2
rater reliability should be very thoughtful since “defining an ac-
ceptable level of reliability is one of the many problems in Total time Total time Reliability
content analysis for which there is no single solution” 共Holsti Category 共min:s兲 共min:s兲 level
1969兲. The reliability standards need to be answered within the Navigating the model 2:11 2:42 96.6%
context of a given research problem 共Krippendorff 1980兲. Since Authoring content 1:00 1:31 95.7%
the data set was relatively small and the framework simple, two Analyzing outcome 0:00 0:00 100.0%
coders were able to perform all of the video coding, and these Person 1 0:47 0:50 99.7%
coders were then used to determine the level of reliability. For Person 2 2:04 1:44 97.8%
more challenging frameworks and larger data sets the reliability
Person 3 2:59 3:15 98.3%
would need to account for all of the coders and may utilize one of
Person 4 6:05 6:38 96.4%
the other reliability tests suggested previously. The agreement
Facilitator 0:00 0:00 100.0%
level between the coders exceeded the standard of reliability

82 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 28.82 共65.2%兲
11.82 共26.7%兲
intervals of 10 min each. This provides a significant amount of

3.59 共8.1%兲
Average
time for groups to hold discussion and allows for lulls in conver-
sation relating to tasks such as calculation or use of the software.

44.23
It is important to note that there is additional error incorporated
into the results for the discussion coding that is not present in the
software coding because of this choice.

20.15 共63.8%兲
9.67 共30.6%兲
1.75 共5.5%兲
Group 6
Results

31.57
VCS 2
When reviewing the videos taken during a study, content analysis
is a valuable tool for deriving the quantitative measures, but the

7.21 共22.0%兲
25.3 共77.3%兲

0.21 共0.6%兲
Group 5
first step is to review the video and observe the content and group
activity. The quantitative measures follow to demonstrate how

32.72
often an action takes place or how much time is spent performing
a given activity. For the case study there were two areas of inter-
est noted and these will be addressed first.

41.02 共60.0%兲
18.58 共27.2%兲
8.81 共12.9%兲
It was clearly evident from observing the videos that the sec-

Table 2. Time Spent Using the Software by Each Group, Shown in Minutes, and as Percentage of the Total Interaction Time with the Software

Group 4
ond version of the software was easier to use, more intuitive, and
user friendly. For the second version of the software, none of the

68.41
groups spent more than 20 min creating their sequence, whereas
the groups using the original version of the software all spent over
50 min creating their sequences. Also, the facilitator spent more

73.70 共81.8%兲
11.87 共13.2%兲

4.51 共5.0%兲
time assisting the teams using the original version of the software,

Average
while the facilitator helping with the second version rarely had to
assist with the software, if at all.

90.08
However, when watching the manner of use of the software,
the overall process of interacting with the software was very simi-

14.71 共19.7%兲
52.61 共70.5%兲
lar. The software was oriented around creating a 4D model of the

7.33 共9.8%兲
Group 3
construction sequence. All of the groups generated their sequence,
created the schedule using the software, and then reviewed the 4D

74.65
sequence. All of the groups went through a similar process in this
regard, and after reviewing the 4D simulation, none of the groups
VCS 1

went back to their schedule and made significant changes. In all

13.02 共17.2%兲
59.57 共78.7%兲
3.14 共4.1%兲

of the cases there were adjustments made to the schedule, typi-

Group 2

cally to perform minor adjustments when the team missed a com-

ponent in the sequence or an element was out of order. The
75.73

average time spent analyzing the simulation for the first version
of the software was 4.5 min while for the newer version it was 3.6
min. When reviewing the timing of the 4D analysis by the groups,
108.93 共90.9%兲
7.87 共6.6%兲

3.06 共2.6%兲

all of the groups performed the review within the last 15 min of
Group 1

their time in the lab 共Table 2兲.

119.86

There was a clear difference in the amount of discussion held

among the groups depending on which version of the VCS inter-
face was used. For the groups using the first interface, each team
member spoke during the sampled video for an average of more
minutes 共time as %兲
minutes 共time as %兲
minutes 共time as %兲

than 11 min, while for the second interface it was only just over 6
min. These results indicate that groups using the first version of
the software spent approximately twice as much time discussing
the problem than with the newer version of the software. Also, as
minutes

shown in Table 3, the time any individual spent speaking varied

from as little as 5 to over 60% of the time. This indicates that
in
in
in
in

there was no typical time one team member spent speaking, and
Time
Time
Time
Time

the variation was consistent between the two versions of the soft-
ware interface. Considering that the range of time speaking was
consistent, but there was a clear difference in the overall time
time with the software
Navigating the model

spent in discussion, there is some different dynamic between the

Analyzing outcome
Authoring content

two groups regarding the approach to the problem.

Total interaction

Looking at the same consideration from a different angle, the

first time the teams chose to author content for the simulation was
identified within each video 共Fig. 3兲. The groups using the first
version typically spent a greater amount of time in preparation
before moving to the use of the software and took more time to

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 83

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 3. Percentage of Total Conversation Time That Each Team Member Spoke during the Meeting
VCS 1a VCS 2b
Group 1 Group 2 Group 3 Group 4 Group 5 Group 6
共%兲 共%兲 共%兲 共%兲 共%兲 共%兲
Person 1 26.8 61.9 34.6 9.8 37.2 12.2
Person 2 12.2 22.2 50.3 14.8 37.2 30.6
Person 3 55.8 15.9 15.1 14.8 24.8 57.2
c c c
Person 4 5.2 60.6 0.7
a
VCS 1 calculated from a sample of the meeting.
b
VCS 2 calculated from entire meeting.
c
The group only contained three members.

prepare before using the software. The simulation using the soft- Discussion
ware was one task within the assignment, and the teams using the
original version seem to have planned to perform the whole class Though the case study example is a simple study using an explor-
assignment during the time they were in the lab, and spent time atory question, the potential implications of this mixed methodol-
performing calculations and holding discussions for how the con- ogy study within construction research is quite powerful.
struction progresses before considering how to detail the erection Examples of current mixed-method studies are readily available
sequence. The groups using the second version of the interface for a multitude of varying research questions. For example, in a
moved directly into creating the content for the simulation. The series of studies designed to examine leadership, Mumford and
groups using the second version seemed to have had their prepa- colleagues 共Mumford 2006; Mumford and Van Doorn 2001;
ratory work done in advance of their time in the lab, and used the Strange and Mumford 2005兲 content analyzed biographies of over
time to focus only on the sequencing aspect. 200 historic leaders. By applying strict coding methods and using
Along with the results which allowed the hypothesis to be well-trained coders, the researchers were able to apply traditional
addressed, there was another potential indicator noticed during
statistical techniques to largely qualitative data. The results of
the observation. As noted above, the facilitator spent a noticeably
their research stood in sharp contrast to current ways of thinking
greater amount of time assisting with the first interface. When
about leadership. What is underscored by these studies is that by
watching these discussions, it was noted that in order to utilize the
applying these mixed methods, researchers may observe phenom-
first version of the interface, there were certain assumptions made
ena and trends that would not be available using either qualitative
in the creation of the software which affected the manner of data
共e.g., single case study兲 or quantitative 共e.g., survey兲 methods
input. The software was created to help students find the “best”
alone. Turning to more traditional observational approaches, Am-
way to sequence the construction, so the interface was designed to
abile and colleagues 共Ruscio and Amabile 1999; Ruscio et al.
complement better strategies. When the facilitator was interacting
1998兲 have examined the creative process in several studies.
with the student groups, there was often dialog about how to
group geometry or create schedule activities to best work with the Using videotaped accounts and think-aloud protocols, the re-
software interface. The discussions, however, often induced some searchers once again applied traditional coding techniques to the
changes regarding the breakdown of the construction sequencing. study of innovation in a natural setting. By using well-trained
These discussions may have helped the teams using the first in- raters, the researchers were again able to apply traditional statis-
terface to better schedule their sequences, which the teams using tical techniques in their study. What is particularly notable about
the second interface were not able to receive. The results in this the work by Amabile and colleagues is how well they were able
regard were not conclusive, because the content of the discussions to study creativity—a complex phenomenon that is, at a mini-
between the facilitator and the teams were not the focus of the mum, a challenge to investigate. More precisely, by applying both
analysis, but this dialog could be reviewed in more detail to qualitative and quantitative methods the researchers were able to
evaluate the potential impact. shed light on a topic that is in desperate need of greater under-
standing. Although countless other examples exist, these should
suffice to make our basic point: observational research—
particularly mixed-method observational research—has a number
of unique advantages that make the challenges associated with the
method worthwhile. The implications for the unique interactions
involved in the design and construction process offer an opportu-
nity to utilize these advantages. The emergent question, then, is
how to conduct this type of research in such a way that maxi-
mizes the researchers’ capacity to both understand and convey
their findings to others. The integration of the qualitative and
quantitative data sets allows for the researcher to draw inferences
from the qualitative video, and review the quantitative data for the
level of impact, or correlation between variables.
While the case study was only exploratory in nature, the in-
tention was to demonstrate the process and how the qualitative
Fig. 3. Figure shows the time elapsed in minutes when the groups and quantitative results can be used to complement each other.
first authored content for the simulation The proper design of the study is essential for the researcher to be

84 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
able to draw valuable data from the video and to be able to cor- combining qualitative and quantitative data, but cannot overcome
relate the coded content with the research questions or hypoth- the bias and covariation introduced in the original design of any
eses. It is strongly suggested that any researchers interested in study.
pursuing this type of work spend time testing and piloting the Along with the challenges and limitations of the coding, the
method, both in terms of the framework used and how to draw challenges inherent in observation as a methodology are always
relationships between the qualitative and quantitative aspects of difficult to foresee. The example from the case study regarding
the methodology. the facilitator assisting with the use of the interface shows that the
observer can inadvertently alter the interactions and outcomes of
the research participants. This limitation and challenge of the
methodology can never be wholly removed, but using technology
Conclusions
to take the person observing out of the setting or finding means to
make the study more rigorous through larger samples and rotating
The construction industry is dynamic in nature with constantly facilitators could minimize the error created by these variables.
changing teams, team members, settings, and methods. The use of One benefit of using observational research methods is that there
observational studies within construction can create quantitative is a better opportunity for identifying these type of experimental
data about the processes and interactions used, while being able to design issues since the researcher can review the specific activi-
fit that data within a qualitative context can prove very valuable in ties that occurred during the experiment. In other more quantita-
a range of research topics. tively focused studies, this type of subtle interaction which can
The example observational study presented in the paper illus- influence the data collected, may easily go undetected.
trates a technique for developing rigorous data collection and
analysis techniques for observational studies using content analy- Potential Uses for Observational Studies
sis and time study data from the coding of audio-video record-
ings. The case study demonstrated the process for developing a There are many potential uses for observational studies, and this
structured coding framework, incorporated rigor and reliability section is by no means exhaustive. When we wish to gain a better
into the coding process, and presented some examples of how the understanding from human interaction or behaviors, it is impor-
observed content could be represented in the quantitative data tant to observe and study behaviors. Construction has many such
developed during the content analysis. While the study was ex- interactions, both in industry and academia. Studying the team
ploratory in nature, the ability of the tool to demonstrate quanti- dynamics of group interactions can provide valuable insight into
tatively what is visually observed creates great potential for how people lead and make decisions. Further studies into leader-
ship, team dynamics, and decision making within construction are
employing more qualitative research techniques in a more rigor-
all important areas that could benefit from more observational
ous manner.
studies. The use of observational studies will create opportunities
to track field construction using observation to identify trends or
Challenges and Limitations techniques to be further studied, and allow more quantitative
analyses of the trends.
“Hence efforts to analyze motives, values, attitudes, and Another important area where many observational studies are
other interesting but elusive variables necessarily require performed is within human computer interaction. The ability to
imaginative and often painstaking efforts to validate infer- study and refine technology so that it is more easily used can
ences drawn from component data. Quantification of improve the productivity of those who use the computer applica-
documentary materials may yield important and interest- tions. Computers and software are rapidly growing in popularity
ing data about many aspects of human experience and in construction professions and on job sites. Finding ways to
behavior, but the temptation to count things for the sake maximize the value through improved efficiency of these pro-
of counting, unless resisted, is almost certain to yield pre- cesses is ongoing, but tracking how it can be used in various
cise findings which are meaningless, trivial, or both” forms on construction projects, similar to the case study pre-
共Holsti 1969兲. sented, could identify methods of better using computers for com-
munication and construction process planning.
The use of content analysis for observational studies enables
In addition to group dynamics, observational studies can also
the creation of vast amounts of quantitative data which can sup-
be beneficial for studying jobsite activities, with a particular focus
port research efforts. The process of generating the data are slow
on productivity and safety. It is critical that we continue to gain a
and painstaking, even using the available software for such ef-
better understanding of the actual performance, priorities and de-
forts. Thus it is essential to focus the coding only on the content
cision models of our workforce, and best methods for investigat-
which will contribute to the research question at hand. Should a
ing these factors through observational research methods is
subcategory need to be refined or a new category added, the con- important. The site of construction activity is ever changing and
tent can easily be recoded, but the time spent generating data that through observational studies it is possible to quantitatively docu-
will never be used is lost forever. Piloting the overall process and ment these activities, while still considering them within the con-
coding framework on a single video or small set of videos is text of their projects and sites. The potential for this methodology
strongly suggested to quickly refine the framework from the con- to be applied within construction research is extensive and thus
tent which does not contribute to the overall research efforts. far has been underutilized.
Observing and coding human interaction and communication
is complex and even a trained analyst with keen insight may find Acknowledgments
it difficult to make maximum use of their data unless they use
systematic methods. As with all research, proper design of the We would like to thank the student participants for agreeing to be
study and thorough attention to detail is essential for the collec- observed and recorded during the study presented in this paper.
tion of data. The methodology presented offers new means for We also thank the National Science Foundation for their support

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 85

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
of this research through Grant Nos. 0348457 and 0342861. Any Univ., University Park, Pa., 具http://ets.tlt.psu.edu/whitepapers/
opinions, findings, conclusions, or recommendations are those of Studiocode_Whitepaper.pdf典 共Aug. 5, 2008兲.
the writers and do not reflect those of the National Science Foun- Hammersly, M., and Gromm, R. 共1997兲. “Bias in social research.” Socio-
dation or the project participants. logical Research Online, 2共1兲, 具http://www.socresonline.org.uk/
socresonline/2/1/2.html典 共July 29, 2008兲.
Holsti, O. R. 共1969兲. Content analysis for the social sciences and humani-
ties, Addison-Wesley, Reading, Mass.
References James, L. R., Demaree, R. G., and Wolf, G. 共1984兲. “Estimating within-
group interater reliability with and without response bias.” J. Appl.
Aiken, L. S., West, S. G., and Pitts, S. C. 共2003兲. “Multiple regression Psychol., 69, 85–98.
analysis.” Comprehensive handbook of psychology. Volume 2. Re- Krippendorff, K. 共1980兲. Content analysis: An introduction to its meth-
search methods in psychology, J. A. Schinka and W. F. Velicer, eds., odology, Sage, Thousand Oaks, Calif.
Wiley, New York. Liston, K., Fischer, M., and Winograd, T. 共2001兲. “Focused sharing of
Atkinson, P., Coffey, A., and Delamont, S. 共1999兲. “Ethnography: Post, information for multi-disciplinary decision making by project teams.”
past, and present.” J. Contemp. Ethnogr., 28, 460–471. J. Inf. Tech. Constr., 6, 69–82.
Bauer, M. W., Atkinson, P., and Gaskell, G. 共2000兲. Handbook of ethnog- Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., and Cannon-
Bowers, J. A. 共2000兲. “The influence of shared mental models on
raphy, Sage, Thousand Oaks, Calif.
team process and performance.” J. Appl. Psychol., 85, 273–283.
Bernardin, H. J., and Buckley, M. R. 共1981兲. “Strategies in rater train-
Messick, S. 共1995兲. “Validity of psychological assessment.” Am. Psy-
ing.” Acad. Manage. Rev., 6, 205–212.
chol., 50, 741–749.
Biello, D. 共2006兲. “Fact or fiction?: Archimedes coined the term “Eu-
reka!” in the bath.” Scientific American, 具http://www.sciam.com/ Mintzberg, H. 共1973兲. The nature of managerial work, Harper & Row,
article.cfm?id⫽fact-or-fiction-archimede典 共July 24, 2008兲. New York.
Bliese, P. D. 共2000兲. “Within-group agreement, non-independence, and Mumford, M. D. 共2006兲. Pathways to outstanding leadership: A com-
parative analysis of charismatic, ideological and pragmatic leaders,
reliability: Implications for data aggregation and analysis.” Multilevel
Erlbaum, Mahwah, N.J.
theory, research, and methods in organizations, K. J. Klein and S. W. Mumford, M. D., and Van Doorn, J. R. 共2001兲. “The leadership of prag-
J. Kozlowski, eds., Jossey-Bass, San Francisco, 439–381. matism: Reconsidering Franklin in the age of charisma.” Leadership
Buchholz, B., Paquet, V., Wellman, H., and Forde, M. 共2003兲. “Quantifi- Q., 12, 279–309.
cation of ergonomic hazards for ironworkers performing concrete re- Plano Clark, V. L., Creswell, J. W., O’Neil Green, D., and Shope, R. J.
inforcement tasks during heavy highway construction.” Am. Ind. Hyg.
共2008兲. “Mixing quantitative and qualitative approaches.” Handbook
Assoc. J., 64共2兲, 243–250.
of emergent methods, S. N. Hesse-Biber and P. Leavy, eds., Guilford,
Campbell, D. T., and Stanley, J. C. 共1963兲. “Experimental and quasi-
New York.
experimental designs for research on teaching.” Handbook of research Raudenbush, S. W., and Bryk, A. S. 共2002兲. Hierarchical linear models:
on teaching, N. L. Gauge, ed., Rand McNally, Chicago. Applications and data analysis methods, Sage, New York.
Conduitt, J. 共2006兲. “Conduitt’s account of Newton’s life at Cambridge.” Ruscio, A. M., and Amabile, T. M. 共1999兲. “Effects of instructional style
Newtonproject, King’s College, Cambridge, Keynes Ms. 130.4, on problem-solving creativity.” Creat. Res. J., 12, 251–266.
具http://www.newtonproject.sussex.ac.uk/texts/viewtext.php?id⫽THE Ruscio, J., Whitney, D. M., and Amabile, T. M. 共1998兲. “Looking inside
M00167&mode⫽normalized典 共Apr. 27, 2008兲. the fishbowl of creativity: Verbal and behavioral predictors of creative
Cook, T. D., and Campbell, D. T. 共1979兲. Quasi-experimentation: Design performance.” Creat. Res. J., 11, 243–263.
and analysis for field settings, Rand McNally, Chicago. Scott, M., and DeSanctis, P. G. 共1992兲. “Microlevel structuration in
Darwin, C. 共1859兲. On the origin of species by means of natural selec- computer-supported group decision making.” Hum. Commun. Res.,
tion, or the preservation of favoured races in the struggle for life, 19共1兲, 5–49.
1st Ed., John Murray, London, 具http://darwin-online.org.uk/ Strange, J. M., and Mumford, M. D. 共2005兲. “The origins of vision:
content/frameset?itemID⫽F373&viewtype⫽text&pageseq⫽1典 共May Effects of reflection, models and analysis.” Leadership Q., 16, 121–
27, 2008兲. 148.
Denzin, N. K., and Lincoln, Y. S. 共1994兲. Handbook of qualitative re- Tabachnick, B. G., and Fidell, L. 共2001兲. Using multivariate statistics,
search, Sage, Thousand Oaks, Calif. Allyn and Bacon, Needham Heights, Mass.
Dov, N. 共2005兲. “Developing an observation model in the conventional Taylor, F. W. 共1903兲. Shop management, Harper & Row, New York.
and simulation lab lessons: Observations on learning and instruction Tukey, J. W. 共1977兲. Exploratory data analysis, Addison-Wesly, Reading,
processes.” Proc., 2005 ASEE/IEEE Frontiers in Education Conf., Mass.
IEEE, Indianapolis. Wang, L., Messner, J. I., and Leicht, R. 共2007兲. “Assessment of 4D mod-
Gill, J., and Johnson, P. 共2002兲. Research methods for managers, Sage, eling for schedule visualization in construction engineering educa-
Thousand Oaks, Calif. tion.” Proc., 24th Int. Conf. on Information Technology in
Gillbreth, F. B. 共1912兲. Primer of scientific management, Van Nostrand Construction, Univ. of Maribor, Maribor, Slovenia.
Reinhold, New York. Wax, R. H. 共1971兲. Doing field work: Warnings and advice, Univ. of
Gopinath, R., and Messner, J. I. 共2004兲. “Applying immersive virtual Chicago, Chicago.
facility prototyping in AECO.” Proc., CONVR 2004: 4th Conf. of Woehr, D. 共1994兲. “Understanding frame-of-reference training: The im-
Construction Applications of Virtual Reality, Dept. of Information pact of training on the recall of performance information.” J. Appl.
Sciences and Technologies of ISCTE, Lisbon, Portugal, 79–86. Psychol., 79, 525–534.
Gyorke, A. 共2006兲. Studiocode white paper: A video coding and analysis Yin, R. K. 共2003兲. Case study research: Design and methods, Sage, Thou-
tool with applications for education and research, Pennsylvania State sand Oaks, Calif.

86 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Action Research as a Proactive Research Method
for Construction Engineering and Management
Salman Azhar1; Irtishad Ahmad2; and Maung K. Sein3

Abstract: Academic research in applied disciplines such as construction engineering and management 共CEM兲 has the dual mission of
simultaneously contributing to the solution of practical problems and creating theoretical and conceptual knowledge. To do so, appropriate
research approaches are needed. However, extant literature in the field has paid little attention to this issue and research methods used have
been almost entirely either quantitative surveys or case studies. In this paper, action research 共AR兲 is proposed as an answer to this
knowledge gap. AR aims at building and testing theory within the context of solving an immediate practical problem in a real setting. The
paper describes the underlying philosophy and application procedure of AR and highlights its strengths and weaknesses. Then, the
applicability of the method to CEM is illustrated through a case study of improving access to information to support planning and decision
making in a construction owner organization through designing and implementing a data warehouse. The findings indicate that AR is a
reliable, structured, and rigorous research approach that is very useful for conducting applied research in construction and enabling
academia to influence and improve construction industry practices. It can also effectively help to improve collaboration between academic
researchers and industry practitioners in research and development projects.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000081
CE Database subject headings: Research; Information management; Organizations; Decision support systems; Databases; Construc-
tion management.
Author keywords: Information management; Organizations; Decision support systems; Databases; Classification description; Organi-
zational issues; Information technologies.

Introduction research on real-life problems which are vital for the construction
industry in addition to fundamental or basic research. Many of the
Academic research has often been criticized for focusing on the- researchable problems in construction tend to be applied in nature
oretical and conceptual issues and neglecting the needs of the and frequently involve modifications of practices and procedures
industry. This criticism has been particularly directed at applied from other disciplines. Consequently, discovery of “overriding
disciplines such as information systems 共Benbasat and Zmud theories” is rare in CEM as compared to pure engineering and
1999兲, management 共Kelemen and Bansal 2002兲, as well as con- natural sciences 共Hauck and Chen 1998兲. Hence, applied research
struction engineering and management 共CEM兲 共Betts and Wood- could be more beneficial for the construction industry and may
Harper 1994兲. Thus, academic research loses relevance and has lead to better management practices, more effective field proce-
little or no impact on practice. One reason for this unfortunate dures, and improved levels of productivity. However, there are
state is that research methods used by academics in CEM, namely some inherent difficulties associated with applied research such as
surveys and case studies, have mostly studied phenomena that has a lack of scientific rigor, the inability to replicate the procedures,
already occurred, i.e., it has focused on existing reality. CEM, by the challenge of application of results to a wider population, and
epistemology and axiology, is a “proactive” field, in that each the lack of dissemination due to concerns about propriety infor-
construction project is an intervention into what exists and thus mation 共Hauck and Chen 1998兲. In essence then, applied research,
creates new reality. in its current form, does not prioritize abstraction and extraction
It follows then, that CEM research needs to employ proactive of conceptual knowledge.
research approaches. There is a need to conduct applied or field What is clearly needed in CEM is a research approach that
combines the objectives of both applied and basic research by
1
Assistant Professor, McWhorter School of Building Science, College contributing toward solution of practical problems and creation of
of Architecture, Design and Construction, Auburn Univ., Auburn, new theoretical knowledge at the same time. An approach that
AL 36849 共corresponding author兲. E-mail: [email protected] fulfills these criteria is action research 共AR兲 where the researcher
2
Professor, Chairperson, Dept. of Construction Management, Florida reviews the existing situation 共problem domain兲, identifies the
International Univ., Miami, FL 33199. E-mail: [email protected] problem共s兲, gets involved in introducing some changes to im-
3
Professor, Dept. of Information Systems, Univ. of Agder 共UiA兲, prove the situation, evaluates the effect of those changes, and
Kristiansand, Norway. E-mail: [email protected] reflects on the process and the outcome to generate new knowl-
Note. This manuscript was submitted on July 24, 2008; approved on
edge 共Baskerville 1997, 1999; Hult and Lennung 1980; Naoum
April 1, 2009; published online on June 17, 2009. Discussion period open
until June 1, 2010; separate discussions must be submitted for individual 2001兲. It is an established research approach that has been in use
papers. This paper is part of the Journal of Construction Engineering in other applied disciplines such as information systems 共e.g.,
and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, ISSN Lindgren et al. 2004兲, management 共Susman and Evered 1978兲,
0733-9364/2010/1-87–98/$25.00. education 共McTaggart 1991兲, health care 共Meyer 2003兲, and

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 87

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
development studies 共Buchy and Ahmed 2007兲 to name a few. AR within the context of solving an immediate practical problem in a
is unique in the way it associates research and practice through real setting. It combines theory and practice, researchers and prac-
change and reflection 共Rezgui 2007兲. It produces highly reliable titioners, and intervention and reflection. AR is not a specific
research results, because it is grounded in practical action, aimed method of research; rather it is an approach. Zuber-Skerritt 共1996兲
at solving a realistic problem situation while carefully informing defined three types of AR: technical which aims to improve ef-
theory 共Baskerville 1999兲. AR is fundamentally different from the fectiveness or efficiency of practice or provide professional
traditional research approaches in CEM, namely, survey and case development by researcher as “outside expert”; practical
study. With these approaches, the researcher tends not to affect or which aims to improve or transform practitioners’ understanding
interface with that which is being studied 共Naoum 2001兲. AR by through active participation and cooperation with the researcher;
contrast involves active participation by the researcher in the pro- and emancipatory which aims to transform organizational sys-
cess under study, in order to identify, promote, and evaluate prob- tems through collaboration with the researcher acting as “process
lems and potential solutions. moderator.”
In this paper, we examine the potential applicability of AR in All types of AR share a common core of characteristics. Its
CEM. While a few reported studies have used this approach epistemological paradigm can be described as pragmatism in that
共Cushman 2001; Hauck and Chen 1998; Barker et al. 2004; “truth” is premised upon utility. Hult and Lennung 共1980兲 lists the
Rezgui 2007兲, there has been little attempt to elaborate on the following three characteristics of AR which distinguishes it from
applicability of the approach in CEM. It is also found that more other research approaches
researchers in the Europe and Asia are familiar with this approach • AR aims at an increased understanding of an immediate prob-
as compared to the researchers in the United States 共Azhar 2007兲. lem situation, with emphasis on the complex and multifaceted
A search of past issues of the Journal of Construction Engineer- nature of organizations.
ing and Management 共JCEM兲 revealed that only one study has • AR simultaneously assists in practical problem solving and
used AR 共Barker et al. 2004兲. Possible reasons behind this trend expands scientific knowledge. This goal extends into two im-
might be the relatively “young” age of such programs in the portant process characteristics: First, highly interpretive as-
United States and less focus on applied research in the past. We sumptions are made about the observations; second, the
attempt to address this gap by systematically examining AR and researcher intervenes in the problem setting.
setting guidelines for conducting research using this approach. • AR is performed collaboratively and enhances the competen-
The rest of the paper is organized as follows. First, a descrip- cies of both researchers and practitioners. It links theory and
tion of AR highlighting its strengths and weaknesses, its underly- practice to generate a solution.
ing philosophy and application procedure is provided. In
describing the process, the paper follows the precepts of canoni- Steps in Action Research Approach
cal AR articulated by, among others, Baskerville 共1999兲 in infor-
mation systems, a discipline that is very similar to CEM in its AR has a five-phase cyclical process. Before the cycle is com-
orientation. Then to demonstrate its use in construction, a case menced, it first requires the establishment of a client-system in-
study of improving the use of information systems to support frastructure or research environment 共Susman and Evered 1978兲.
planning and decision making in a construction owner organiza- This infrastructure is the specifications and agreement that pro-
tion through designing and implementing a data warehouse 共a vides the protocol under which the researchers and practitioners
new database management technology兲 is presented. The paper would conduct the research work. Considerations found within
concludes with a discussion and reflection on how AR can con- the agreement may include the boundaries of the research
tribute to more collaborative efforts between construction re- domain, the roles of researchers and practitioners, and may
searchers and practitioners who are interested to conduct research also recognize the latitude of the researchers to disseminate the
to solve industry-based problems. learning that is gained in the research. A key aspect of the client-
system infrastructure is the collaborative nature of the undertak-
ing. The researchers work closely with practitioners who are
Action Research Approach located within the client-system. These individuals provide the
subject system knowledge and insight necessary to understand the
The term AR was coined by Kurt Lewin who developed the anomalies being studied 共Clark 1972兲.
method in the wake of social changes that happened after World Once the client-system infrastructure is established, the five-
War II. He articulated the method as “a way of generating knowl- phased cycle of AR commences. As shown in Fig. 1, the phases
edge about a social system while, at the same time, attempting to are:
change it” 共Lewin 1946兲. Later scholars, especially Argyris and
his associates 关e.g., Argyris and Schon 共1978兲兴 further developed
the method. A comprehensive definition is provided by Hult and Phase 1: Diagnosing
Lennung 共1980兲. Diagnosing corresponds to the identification of the primary re-
“AR simultaneously assists in practical problem-solving search problem共s兲. It involves self-interpretation of the complex
and expands scientific knowledge, as well as enhances the research problem, not through reduction and simplification, but
competencies of the respective actors, being performed rather in a holistic fashion. The diagnosis leads to the develop-
collaboratively in an immediate situation using data ment of certain theoretical assumptions 共i.e., a working hypoth-
feedback in a cyclical process aiming at an increased esis兲 about the nature of the problem domain 共Baskerville 1999兲.
understanding of change processes in social systems
and undertaken within a mutually acceptable ethical Phase 2: Action Planning
framework.”
Action planning establishes the target for change and the ap-
In essence then, AR aims at building and/or testing theory proach to change. In this phase, organizational actions are speci-

88 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 1999兲. While the first two types of knowledge contribute mainly
to the practitioner audience, the third type contributes to the aca-
demic audience
5. 1. • The new practical knowledge gained by the organization dur-
Specifying Diagnosing ing the research. It is also called “double-loop learning.”
Learning • The additional knowledge gained when the change is unsuc-
cessful. This may provide foundations for diagnosing in prepa-
ration for further AR interventions.
• The scientific knowledge gained through reflection on the suc-
2. cess or failure of the action taken. This provides important
Action knowledge to the scientific community about the theoretical
Planning
4.
framework which was the basis of the action taken.
Evaluating The AR cycle can continue, whether the action proved suc-
cessful or not, to develop further knowledge about the organiza-
tion and the validity of relevant theoretical frameworks. As a
3.
result of such studies, the organization learns more about its na-
Action
Taking ture and environment while the researchers refine and further de-
velop the theoretical elements of the action thus leading to the
Client-System Environment continuous evolution of the theoretical framework used 共Argyris
and Schön 1978兲.
Fig. 1. AR cycle 关adapted from Baskerville 共1999兲兴

Distinguishing Action Research from Consulting

AR processes and typical organizational consulting processes

fied that should relieve or improve the primary problem共s兲. The contain substantial similarities. This leads to the criticism that is
discovery of the planned actions is guided by the theoretical often raised that AR is really consulting in disguise 关see Avison
framework developed in phase 1, which indicates both desired et al. 共1999兲 and McKay and Marshall 共2001兲 for a good discus-
future state and the changes that would achieve such a state sion on this兴. In seeking to answer to this criticism, McKay and
共Baskerville 1999兲. Marshall 共2001兲 propose that AR should be conceptualized as
consisting of two interdependent and interlocking cycles, one for
Phase 3: Action Taking “problem solving interest” and the other for “research interest.”
While the first can be seen as similar to consulting, it is the latter
Action taking implements the planned action共s兲 from Phase 2. cycle that distinguishes AR as a research method.
The researchers and practitioners collaboratively intervene into In a similar vein, other researchers have listed five key dif-
the client 共practitioner’s兲 organization, causing certain changes to ferences between AR and consultancy 共Kubr 1986; Lippitt and
be made. Several forms of intervention strategy can be adopted. Lippit 1978兲. These are
For example, the intervention might be directive, in which the • Motivation: AR is motivated by its scientific prospects, per-
research “directs” the change, or nondirective, in which the haps epitomized in scientific publications. Consulting is moti-
change is sought indirectly. The process can draw its steps from vated by commercial benefits, including profits and additional
social psychology, e.g., engagement, unfreezing, learning and re- stocks of proprietary knowledge about solutions to organiza-
framing 共Baskerville 1999兲. tional problems.
• Commitment: AR makes a commitment to the research com-
munity for the production of scientific knowledge, as well as
Phase 4: Evaluating
to the client. In a consulting situation, the commitment is to
After the actions are completed, the collaborative researchers and the client alone.
practitioners evaluate the outcomes. Evaluation includes deter- • Approach: Collaboration is essential in AR because of its id-
mining whether the theoretical effects of the action were realized, iographic assumptions. Consulting typically values its “outsid-
and whether these effects relieved the problems. Where the er’s” unbiased viewpoint in providing an objective perspective
change was successful, the evaluation must critically question on the organizational problems.
whether the action undertaken was the sole cause of success. • Foundation for recommendations: In AR, this foundation is a
Where the change was unsuccessful, some framework for the next theoretical framework. Consultants are expected to suggest so-
iteration of the AR cycle 共including adjusting the hypotheses兲 lutions that, in their experience, proved successful in similar
should be established 共Baskerville 1999兲. situations.
• Essence of the organizational understanding: In AR, organi-
zational understanding is founded on practical success from
Phase 5: Specifying Learning
iterative experimental changes in the organization. Typical
While the phase of specifying learning is formally undertaken consultation teams develop an understanding through their in-
last, it is usually an ongoing process. At every phase of the re- dependent critical analysis of the problem situation.
search, continuous reflection takes place which leads to under- In summary, consultants are usually paid to dictate experi-
standing which in turn leads to abstraction of new knowledge. enced reliable solutions based on their independent review.
The knowledge gained in AR 共whether the action was successful Should, in a particular construction sector, the consultants lack
or unsuccessful兲 can be classified into three types 共Baskerville prior relevant experience, it is highly unlikely that they will pro-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 89

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
vide suitable guidelines to the construction clients. Action re- Problem
Identification Evaluation of Information Systems in
Diagnosing
searchers on the other hand act out of scientific interest to help the Construction Owner Organizations

organization itself to learn by formulating a series of experimental Solution

solutions based on an evolving, untested theory 共Baskerville Development Multi-Perspective Enterprise Modeling
of Client Organization
1997兲. AR thus has the potential to influence and improve indus-
try practice through innovative approaches in small manageable Action Functional Model and Reference Evaluation &
Architecture of Data Warehousing
Planning Learning
chunks. Feedback System

Organizational Restructuring Model

Strengths and Weaknesses of Action Research

Testing
Action and Prototype Data Warehousing System
Taking Validation
AR has both strengths and weaknesses. Hales and Chakravorty Implementation, Testing and Validation

共2006兲 highlighted two major strengths of this approach. First, it Action Action
Research Phase Research Roadmap Research Phase
provides a rich explanation of “how” and “why” phenomena
共problem under investigation兲 occur, which sometimes cannot be
Fig. 2. Research methodology framework
explained through statistical or regression models. Second, re-
search problem共s兲 are studied in a natural setting which would be
expensive, difficult, and/or impossible to replicate in a laboratory
organizations do not effectively use these data for planning and
experiment.
decision-making due to two reasons. First, the information sys-
A weakness of AR is that it fundamentally assumes that an
tems in construction organizations are designed to support day-
espoused theory should adequately specify action, which is rarely
to-day construction operations. The data stored in these systems
the case. A second weakness is that the conclusions from a single
are often nonvalidated, nonintegrated, and are available in a for-
study may have limited generalizability 共Hales and Chakravorty
mat that makes it difficult for decision makers to use in order to
2006兲. However, such real world studies are still valuable for
make timely decisions. Second, the organizational structure is
refining theory and suggesting directions for further investigations
often not compatible with the information systems thereby result-
共Stake 2000兲. In addition, the research may not be a top priority
ing in higher operational costs and lower productivity. These two
for the client and it may affect the execution of the research. An
issues were investigated in this research. The scope of the re-
opposite problem may also arise attributable mainly to the prac-
search was limited to construction owner organizations that are
ticality of the results. It is sometimes difficult to identify the
continuously involved in multiple construction projects such as
“stopping point” for the research and the client may prefer 共or
departments of transportation, county/city and state governments,
sometimes insist兲 to continue the research beyond its initial scope.
transit agencies, port authorities, etc.
This problem can be addressed by clearly and explicitly defining
the research scope, the roles and responsibilities of the research-
ers and practitioners, and the endpoint of the research, in the
research agreement 共Avison et al. 1999; Stringer 1996兲. Description of the Research Process

In accordance with the tenets of AR, the research was conducted

Action Research in Construction: Case Study in five interrelated phases as shown in Fig. 2.

The applicability of AR approach in CEM is demonstrated Phase 1: Problem Diagnosis

through a case study of improving the accessibility and availabil-
ity of information for helping senior managers in a construction The primary research problem was assessed through literature
owner organization. The action taken involved designing and review and an initial hypothesis was developed as follows, “Con-
implementing a data warehouse. A data warehouse is an analytical struction owner organizations do not use information systems for
database created by integrating data from multiple heterogeneous planning and decision making effectively.” To validate the initial
operational databases and other information sources. The primary research hypothesis, a questionnaire survey was conducted on
purpose of a data warehouse is to provide easy access to espe- construction owner organizations in the United States. Of the 550
cially prepared data that can be used by the senior management of questionnaires sent, 163 valid responses were received; represent-
an organization for planning and decision-making 共Ahmad and ing a total response rate of approximately 30%. The results indi-
Azhar 2004兲. The concept of data warehousing is simple, techni- cated that 40% of surveyed organizations employ a formal
cally appealing but its implementation in construction organiza- information system to store and manage project data. While 15%
tions poses enormous organizational, financial, and technical of the organizations use their existing information systems to
challenges. Using AR approach, a framework is developed to ad- make everyday decisions, only 11% use them for short and long
dress these challenges which is illustrated in this case study. It is term planning. The reasons found were low decision-support ca-
beyond the scope of this paper to thoroughly describe the project. pabilities of existing information systems, poor data quality, dif-
Interested readers can find complete details in Azhar 共2005兲 and ficult data access and high cost of operations. Ninety-one percent
Azhar and Ahmad 共2007兲. of surveyed organizations indicated that they require a new and
enhanced information system to support planning and decision-
making 共Azhar and Ahmad 2007兲.
Research Problem The results of the questionnaire survey validated the initial
research hypothesis and provided relevant data to explicitly de-
Construction organizations typically deal with large volumes of fine the research objectives and scope. The next task was to
project data containing valuable information. Through literature plan actions based on theoretical premises to solve the identified
review and via a questionnaire survey, it was assessed that these problems.

90 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 spectives such as functional, organizational, informational,
decision-making etc. These models were developed to understand
Extraction the organization’s business operations, flow of information
within, into and outside the organization, identification of deci-
Transactions
Data
Queries sion nodes and their hierarchy, and data requirements for these
W
Cleaning decisions. The information drawn from the enterprise modeling
Data Marts was used to recognize functional, informational, and organiza-
Reports
tional requirements for the proposed data warehouse.
Integration
Flat Files
Metadata
Development of Functional Model and Reference
Architecture of the Proposed Data Warehouse
Summarization Data Based on the user requirements identified via enterprise modeling,
Mining
External Data a functional 共or logical兲 model of the data warehouse and its
corresponding reference architecture were developed as shown in
Sources Processing Storage Analysis Fig. 4. The functional model guided the focus group how the
system would work while the reference architecture indicated its
Fig. 3. Conceptual schema of a data warehouse physical and technical implementation using different software
and hardware tools. Using this reference architecture, a prototype
system was developed.
Phase 2: Action Planning
AR holds that if the same problem situation is present in a num- Organizational Restructuring
ber of organizations of similar nature, i.e., the specific situation During the course of the research, it was recognized that the suc-
represents a class of problems, then one organization may be cessful implementation of the proposed data warehouse would
chosen as a collaborator for intervention, i.e., data collection, require appropriate changes in the organizational structure of the
model/framework development, testing and validation. The re- client organization to maximize productivity and ensure maxi-
search results then could be transferred to other organizations mum cost savings. The data warehouse functional model and the
with suitable modifications 共Reason and Bradbury 2001兲. In this prototype system were shown to the focus group to explain its
case, the class of problem was “ineffective use of information functionalities and possible role in the future decision-support
systems for planning and decision making.” operations. After the demonstration, the focus group gave its
One local public construction owner organization in the state feedback on how the implementation of data warehousing sys-
of Florida, hereafter termed as client organization was chosen as a tem would affect the existing organizational structure. Based on
collaborator for this research study. The client organization ac- this input, several organizational redesign schemes were devel-
tively participated in the initial questionnaire survey. The reason oped. After a series of discussions, the focus group approved
for this selection was their staff’s motivation concerning this re- a final organizational restructuring model for implementation.
search, firm commitment of executive management and grant of This organizational restructuring is discussed further in the next
access to research-related data. The client organization remained section.
involved in this research via a “focus group” consisting of orga-
nization’s executives from different departments/divisions. This
Phase 3: Action Taking
group provided their feedback at different stages of research
through questionnaire surveys, individual and group discussions, In this phase, the prototype data warehousing system was imple-
and via brain-storming sessions. In consultation with the focus mented in the client organization to assess its effectiveness. Train-
group, the researchers decided to test the data warehousing tech- ing workshops were conducted to educate the members of
nology as a promising solution to solve the problems diagnosed in the focus group about this new system. The members of the fo-
the first phase. cus group tested the performance of the system and verified its
A data warehouse 共Fig. 3兲 is a dedicated database system cre- accuracy.
ated by combining data from multiple databases for purposes of After testing the prototype system, the focus group proposed
analysis 共Kimball and Ross 2002兲. It organizes the collected data that the organizational restructuring should be carried out after
for consistency and easy interpretation, keeps “old” data for his- one year of full-scale system implementation and in two phases.
torical analysis, and makes access to, and use of data a simple In the first phase, the construction division and project control
task so that users can do it themselves without great technical division of the client organization would be downsized while in
proficiency in data handling. Its purpose is to provide easy access the second phase, they would be merged to form a new division.
to especially arranged data that can be used with decision support These organizational changes would help to decrease the hierar-
applications, such as management reporting, queries, data mining, chy in the organizational structure and result in substantial cost
and executive information systems 共Ahmad and Azhar 2005; savings.
Kimball and Ross 2002兲. Implementing an organization-wide in- The existing organizational structure of the construction and
formation system such as a data warehouse invariably involves project control divisions is shown in Fig. 5. Initially, there were
organizational change. The three tasks accomplished in the action approximately 40 employees working in these two divisions. Dur-
planning phase were enterprise modeling, data warehouse system ing the group discussions, it was determined that most of the
development and restructuring of organizational setup. computational work was being performed by the section manag-
ers. They collected project data from the office and field staff;
Enterprise Modeling organized and analyzed the data; prepared reports; and sent these
The purpose of enterprise modeling was to capture the current or reports to respective mangers or division chiefs for necessary
“as-is” state of the organization and model it from various per- decision-making. After implementation and appropriate changes

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 91

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Staging Area
Data Sources Data Marts Integration into OLAP End User
Extraction, Validation,
Transactional Data Marts Enterprise Data Analysis Applications
Transformation,
Systems Warehouse Data Mining Reports, Charts
Integration

Projects
Information

Project
Meta
Design
Project Productivity Data
Performance Information
Scheduling
Data Validation, Correction and
Source Data Files (Temporary Store)

Target Data Files (Temporary Store) Reports

Cost and Contractor Schedule Queries
Estimating Performance Management
Integration

DSS/EIS
Enterprise
Data Mining
Wide
Project Data
Progress
Cost Financial Warehouse
Management Management
Accounts &
Payroll

Claims Safety
Contracts
Management Management

Safety &
Quality

Claims

MS Access, MS Power MS Excel,

Software Tools MS Access MS Access
Excel, VBA OLAP Xcelsius

IT Infrastructure Windows NT Server, CISCO 1720 Routers, CISCO 2950 Switches, Raid Drive

Fig. 4. Functional model and reference architecture of the proposed data warehousing system

in business processes, these roles would be carried out by the data and as a result, unnecessary time and efforts were wasted in co-
warehousing system and the services of section managers in the ordination and decision-making. The various processes could be
existing form may no longer be required. Hence, it was decided to streamlined if both divisions were merged to work under a single
eliminate these positions after one year of data warehousing management. Based on this, the second phase of organizational
implementation. The new organizational structure is illustrated in restructuring was planned as shown in Fig. 7. It was proposed to
Fig. 6. The significant features of this organizational structure are merge construction division and project control division to form a
reduced organizational hierarchy and downsizing of management new “construction planning and management division.” This new
staff by 23%. It was also decided that the data warehousing sys- division would carry out all operations related to construction
tem would be maintained by the existing staff of the Information planning, management, and control. As the division would work
Technology 共IT兲 division 共not shown in Figs. 5 and 6兲. under a single division chief, the decision-making would be
It was further decided that this new organizational structure quicker and more effective. It was further expected that this
should persist for a period of 2 years 共3 years in total after in- change would increase coordination between the employees and
cluding 1 year of training兲. During this time, it was expected that result in improved productivity. The number of employees would
the division chiefs and managers would have developed necessary now be reduced to 26 thereby indicating a 35% reduction in
expertise to use data warehousing system for their daily short- workforce compared to the existing one.
term and long-term business needs. At the same time, the data
warehouse would have been populated with complete historical
Phase 4: Evaluating
and current project data. Only after meeting these milestones,
would the second phase of organizational restructuring be carried Improvements in the client organization after implementing the
out. data warehouse were evaluated using various performance mea-
During Phase 1, it was assessed that different functional areas sures. Table 1 shows the focus group opinions before and after
of project management had been split between the two divisions data warehouse implementation. The results are statistically sig-

92 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Executive Director

Deputy Director Planning

and Development

Chief Quality Control Asst. Director Construction

and Engineering

Chief Design Chief Chief Project

& Engineering Construction Control

Senior Engineer Manager Manager

Planning Cost & Scheduling Contract Services

Engineer - 2
Planning Section Manager Section Manager
Scheduling Contract Administration

Manager - 3
Construction Staff (2) Staff (3)
Mgmt. Services

(Section) Manager - 2 Section Manager Section Manager

Facilities Cost Control Documents Control

Staff (4) Staff (1) Staff (2)

Construction
Managers -3 (3) Section Manager Section Manager
Busway Segment II Progress Reporting Claims Administration

R.E. & CE &

I Staff (7) Staff (1) Staff (2)

Construction
Manager - 2
Capital Projects

Construction Division (20) Project Control Division (20)

Fig. 5. Existing organizational structure of the client organization

nificant 共p-value in all cases is less than 0.05兲 indicating that savings in direct salary costs would be approximately $1.4 mil-
significant improvements were recorded after the implementation lion and $8.6 million, respectively. It is important to note that data
of the data warehousing system. Table 2 compares the costs in- warehousing would also result in substantial savings in indirect
curred for data warehousing implementation with the resulting costs such as information management and processing costs, re-
direct cost savings which would be achieved as a result of port preparation costs, and other miscellaneous operational costs.
changes in the organizational structure. However, these costs are not included here because the focus of
As is evident from Table 2, a period of approximately 3 1/2–4 this comparison is to show the benefits of organizational restruc-
years would be required to pay off the data warehousing costs by turing on direct costs savings.
means of savings in direct salary costs. After 5 and 10 years, net To further validate the research results, a presentation was

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 93

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Executive Director

Deputy Director Planning

and Development

Chief Quality Control Asst. Director Construction

and Engineering

Chief Design Chief Chief Project

& Engineering Construction Control

Manager
Cost Control
Senior Engineer
Planning

Staff (1)
Engineer
Planning

Manager Schduling
& Progress Reporting
Manager - 3
Construction
Mgmt. Services

Staff (2)

Staff (4)

Manager
Contract Services

Construction
Managers -3 (3)
Busway Segment II Staff (3)

Staff (7)
Manager
Claims Administration

Construction
Manager - 2
Capital Projects Staff (2)

Construction Division (18) Project Control Division (13)

Fig. 6. Proposed organizational structure after one year of data warehousing implementation

made to twenty construction owner organizations who earlier took Learning for Practice (Double Loop Learning)
part in the problem-diagnostic survey. The presentation was on The project has demonstrated how to design and implement in-
possible implementation of the proposed data warehouse in their formation systems for planning and decision-making using enter-
organizations. Their feedback was used to develop a generalized
prise modeling, data warehouse architecture, and organizational
data warehousing design and implementation model for the con-
restructuring in a specific construction owner organization. The
struction owner organizations as shown in Fig. 8.
client organization can use this knowledge to design new infor-
mation systems or enhance existing systems. This could lead to
Phase 5: Specifying Learning process improvement. This is the type of learning that results
The learning from this research project can be grouped into two from the “problem solving interest” cycle as articulated by
categories McKay and Marshall 共2001兲.

94 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Executive Director

Deputy Director Planning

and Development

Chief Design and Chief Construction

Chief Quality Control
Engineering Planning & Management

Manager Manager
Cost Planning, Planning, Scheduling
Estimating & Control & Progress Monitoring

Staff (2) Staff (2)

Manager Manager
Contracts Services Construction
& Claims Administration Management Services

Staff (3) Staff (6)

Construction Manager Senior Engineering

Field Projects Planning

Staff (6)

Proposed Construction Planning and Management Division (26)

Fig. 7. Proposed organizational structure after three years of data warehousing implementation

Learning for Research amount and type of information required in a given environment
The primary transferable knowledge contribution of the research and the organization’s information processing capability. In con-
is a five-phase framework for planning, designing, building, and struction organizations, the organizational structure is often con-
implementing a data warehousing system in construction owner sidered as rigid and usually not redesigned after a new or an
organizations 共see Fig. 8兲. This learning results from the “research enhanced information system is implemented. This research
interest” cycle 共McKay and Marshall 2001兲. Since the nature showed that substantial cost savings could be realized if the or-
and sequence of construction operations in most construction ganizational structure and information systems are compatible
owner organizations are same, and the decision-making require- with each other.
ments are not much different, this framework and its associated
models can be used by other organizations with little modifica-
tions. These organizations do not have to “reinvent the wheel.”
This effort will save a significant amount of time and resources Discussion
for these organizations.
The research increased the understanding of the interplay be- The objective of the paper was to forward the case for using AR
tween technology and organization. It was found that the organi- in CEM research. It was argued that few CEM researchers have
zational design and the design of work processes is shaped by the used this approach. Even the few published research studies that

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 95

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 1. Comparison in Performance before and after Implementation of Data Warehousing System
Mean of responsesa
Sum of Significant
Without data With data squares error
Comparison criteria warehouse warehouse 共X2兲 共p兲
Ease in data access 2.83 4.33 8.42 0.01
Data quality 3.12 4.67 6.97 0.03
Data integration 2.45 4.56 7.01 0.02
Organizational productivity improvements 1.98 4.67 9.14 0.01
Quality of reports 2.78 4.37 8.12 0.02
Support for every day decisions 2.46 4.67 10.12 0.00
Support for short and long term planning 1.78 4.83 6.98 0.01
a
A Likert scale of 1–5 was used to assess the respondents’ opinions where 1 represents “highly dissatisfied” and 5 represents “highly satisfied.”

reported using AR did not provide a detailed description of the

Table 2. Comparison of Data Warehousing Implementation Cost and Net approach, nor did they provide guidelines for applying it. This
Associated Direct Cost Savings paper attempts to fill this knowledge gap. In doing so, it is ac-
Initial and Savings due to knowledged that these studies have demonstrated the feasibility
Time after data operational organizational Net of using AR in CEM.
warehousing costs restructuring savings For example, Hauck and Chen 共1998兲 used AR to develop a
implementation 共$兲 共$兲 共$兲
centralized schedule control system for a large homebuilding
3 years 1,231,639 795,917 ⫺435,722 company in the United States based on centralized critical path
5 years 1,691,832 3,092,461 1,400,629 scheduling method. Cushman 共2001兲 employed AR in a two-year
10 years 2,465,429 11,090,792 8,625,363 joint industry/academic project funded by the British government
involving five construction companies and two universities. The

Responsible Data Warehousing Planning, Design and Implementation Milestones Tools/Techniques

Department
Executive Identifying Data Warehousing Need
Management Questionnaire Surveys
Evaluation of existing Organization Vision Focus Group Opinions
Information System(s)
IT Department

Executive Modeling of Construction Owner Organization CIMOSA Framework

Management - IDEF0 models
Functional Model Information Model Organizational - OO/UML model
Structure Model
Construction - Decision-Info Matrix
Divisions Data and Information Decision-Support Data Flow Diagrams
Decision Model Flow Models Requirements Matrices Information Flow Model
IT Department Decisions-Info Flow Mapping

Construction Development of Data Warehousing System

Divisions Data Modeling Bottom-up Approach
Data Profiling Data ETL Schemes Schemas STAR schema
IT Department Three-tier Architecture
Reference Architecture System Development End-user Applications OLAP Tools
Consultants

Development of IT Infrastructure for DWH Implementation

IT Department C2C Communication
Selection of Software Network Reconfiguration “Data-Centric”
and Hardware Components

Development of Organizational Restructuring Plan Focus Group Discussions

Executive
Management Redesign Business OR
Processes Downsize Departments
Dynamic Simulation
Construction Modeling
Divisions Merge Departments (System Dynamics)

Fig. 8. Framework for data warehouse design and implementation in construction owner organizations

96 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 3. Canonical Action Research Criteria 关Adapted from Davison et al. 共2004兲兴
Criterion Description
Principle of researcher-client agreement 共RCA兲 The RCA provides the basis for mutual commitment and role expectations
Principle of cyclical process model 共CMP兲 The CPM consists of the stages diagnosing, action planning, action taking,
evaluating, and specifying learning
The principle of theory Theory must play a central role in AR
The principle of change through action Action and change are indivisible research elements related through
intervention focused on producing change
The principle of learning through reflection Considered reflection and learning allow a researcher to make both a
practical and theoretical contribution

purpose of the project was to build a high value construction Benefits of Using Action Research
environment 共B-Hive兲 by identifying and analyzing the business,
technical, human and organizational issues associated with adding AR provides an answer to the criticism that traditionally, aca-
values to construction projects through teams cooperating within demic researchers and the construction industry practitioners do
temporary multiple organizations. Rezgui 共2007兲 investigated the not collaborate closely in most construction research projects.
effectiveness of information and communication technology There is a perception among the construction practitioners that
共ICT兲-enabled virtual collaborative teams in construction projects academic research is more focused on subjects and issues which
using AR. Each of these studies were conducted through collabo- are not crucial for the construction industry. The practitioners also
ration with industry partners, were successful in solving the prac- claim that the academic research results are sometimes inappli-
tical problem and generated academic knowledge in the form of cable and impractical for use in real-life construction projects.
systems, models and frameworks. The researchers on the other hand argue that the construction
It is evident from these examples and the case reported in this industry practitioners often do not entertain innovative research
paper that AR provides a structured approach to conduct research ideas which require a major change in the industry practices and
while maintaining a high level of academic rigor and permitting procedures. This situation dictates the need to enhance the
the application of results to a wider audience. It is highly suitable researcher-practitioner collaboration to conduct research on prob-
for conducting research in construction especially in multidisci- lems which are vital for the construction industry and to find out
plinary research that simultaneously involves technological, orga- their adoptable solutions. AR provides a platform to achieve this
nizational, and behavioral aspects. Examples of such areas are objective.
designing and testing new equipment, systems or business pro- One specific way of achieving research-practice collaboration
cesses, and information systems design and implementation. It is is through students. AR could be very attractive for students with
also a useful method where multiple parties are involved such as professional experience who have identified a problem during
partnering, alliancing, contractual arrangements, and virtual the course of their work and wish to investigate and propose a
teams/organizations. change to improve the situation. It also provides an opportunity to
AR allows researchers to perform systematic investigations “connect” students with the problems they will be asked to solve
to seek solution of problems experienced by practitioners or as managers of construction companies. After graduation, these
to examine the effectiveness of proposed frameworks. For ex- students will not only be familiar with the problems and needs of
ample, based on theoretical concepts, Ahmad and Sein 共1997兲 the construction industry but also have the skills to solve them.
developed a framework to implement Total Quality Management In this way, AR can play a pivotal role in bringing academia
in construction project teams. The only way to validate their and the construction industry closer. The case described in this
framework is through AR. In short, AR is particularly relevant in paper is an excellent illustration. It is based on the doctoral dis-
solving industry problems that require innovative and untried so- sertation research of one of the writers who was advised by an-
lutions. Such solutions are inherently risky and thus go against the other writer 共The third writer helped with the methodological
grain of the construction industry that assiduously seeks to avoid aspects兲.
risk.
Using AR in CEM can be challenging. One particular chal-
lenge is the establishment of evaluation criteria. The attainment of Concluding Remarks
research objectives will be evaluated by these criteria to deter-
mine whether they have been accomplished. It may be useful A key objective of the paper was to provide guidelines for con-
to adopt a set of outside standards 共or even results obtained in ducting research using this approach. The process of canonical
previous published research兲 as the evaluation criteria in addi- AR described in an earlier section was followed in the case study
tion to the list of measures developed by the researcher or described in this paper. In that sense, this in itself was an AR
even the practitioner. By adopting these external standards of project. Based on reflection on the experience, it can be recom-
evaluation, the effectiveness and efficacy of the action taken can mended that the steps and phases of canonical AR be followed
have higher credibility with the wider audience. The rigor of with little modification other than to adapt to the context of the
an AR study can be mapped against the criteria laid down by individual research projects.
Davison et al. 共2004兲 and shown in Table 3. The academic re- AR is not without its problems for researchers. The research
viewers can use these criteria to decide about the validity of the collaborative framework could potentially diminish the research-
research results. er’s ability to control the process and the outcomes of the re-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 97

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
search. The lack of control makes it difficult to apply AR as an Benbasat, I., and Zmud, R. W. 共1999兲. “Empirical research in information
instrument in an orchestrated research program. Another potential systems: The practice of relevance.” MIS Q., 23共1兲, 3–16.
problem can arise from the need for researchers and practitioners Betts, M., and Wood-Harper, T. 共1994兲. “Re-engineering construction: A
new management research agenda.” Constr. Manage. Econom., 12共6兲,
to share a mutually acceptable ethical framework. Successful AR
551–556.
is unlikely where there is conflict between researchers and prac- Buchy, M., and Ahmed, S. 共2007兲. “Social learning, academics and
titioners or among practitioners themselves. For example, prob- NGOs: Can the collaborative formula work?” Action Res., 5共4兲, 358–
lems may well arise if the research could lead to people being 377.
fired 共Avison et al. 1999兲. Such result can be in conflict with the Clark, P. 共1972兲. Action research and organizational change, Harper &
researcher’s principles but be acceptable to practitioners or vice Row, New York.
versa. Cushman, M. 共2001兲. “Action research in the UK construction industry-
Despite its challenges, AR provides a rewarding experience The B-hive project.” Proc., IFIP WG 8.2 Working Conf. on Realign-
for researchers who want to work closely with the practitioner ing Research and Practice in Information Systems Development: The
community and in general for academia to influence and affect Social and Organizational Perspective, Boise State Univ., Boise, Id.
practice while creating new knowledge. It can be combined Davison, R. M., Martinsons, M. G., and Kock, N. 共2004兲. “Principles of
with other research methods to generate new theory and/or to canonical action research.” J. Inf. Syst., 14共1兲, 65–86.
reinforce or contradict existing theory. Similarly it can benefit Hales, D. N., and Chakravorty, S. S. 共2006兲. “Implementation of Dem-
ing’s style of quality management: An action research study in a plas-
the industry by evolving improved management practices, more
tics company.” Int. J. Prod. Econ., 103共1兲, 131–148.
effective field procedures, and development of new products, ma- Hauck, A., and Chen, G. 共1998兲. “Using action research as a viable al-
terials, and tools which could result in high tangible and intan- ternative for graduate theses and dissertations in construction manage-
gible benefits. ment.” Int. J. Constr. Ed. Res., 3共2兲, 79–91.
Hult, M., and Lennung, S. A. 共1980兲. “Towards a definition of action
research: A note and bibliography.” J. Manage. Stud. (Oxford), 5共2兲,
241–250.
References
Kelemen, M. L., and Bansal, P. 共2002兲. “The conventions of management
research and their relevance to management practice.” Brit. J. Man-
Ahmad, I., and Azhar, S. 共2004兲. “Data warehousing in construction: age., 13共1兲, 97–108.
From conception to application.” Data warehousing: Design and de- Kimball, R., and Ross, M. 共2002兲. The data warehouse toolkit: The com-
velopment perspectives, S. J. Krishna, ed., ICFAI Books, India. plete guide to dimensional modeling, 2nd Ed., Wiley, New York.
Ahmad, I., and Azhar, S. 共2005兲. “Implementing data warehousing in the Kubr, M. 共1986兲. Management consulting: A guide to the profession, 2nd
construction industry: Opportunities and challenges.” Proc., 3rd Int. Ed., International Labor Office, Geneva.
Conf. on Construction in the 21st Century (CITC-III), National Tech- Lewin, K. 共1946兲. “Action research and minority problems.” J. Soc. Is-
nical Univ. of Athens 共NTUA兲, Athens, Greece, 863–871. sues, 2共4兲, 34–46.
Ahmad, I., and Sein, M. K. 共1997兲. “Building construction project teams Lindgren, R., Henfridsson, O., and Schultze, U. 共2004兲. “Design prin-
for TQM: A factor-element impact model.” Constr. Manage. Econom., ciples for competence management systems: A synthesis of an action
15共5兲, 457–467. research study.” MIS Q., 28共3兲, 435–472.
Argyris, C., and Schön, D. 共1978兲. Organizational learning: A theory of Lippitt, G., and Lippit, R. 共1978兲. The consulting process in action, Uni-
action perspective, Addison-Wesley, Reading, Mass. versity Associates, San Diego.
Avison, D., Lau, F., Myers, M., and Nielsen, P. A. 共1999兲. “Action re- McKay, J., and Marshall, P. 共2001兲. “The dual imperatives of action
search.” Commun. ACM, 42共1兲, 94–97. research.” Inf. Technol. People, 14共1兲, 46–59.
Azhar, S. 共2005兲. “Information systems to support decision-making in McTaggart, R. 共1991兲. “Principles for participatory action research.”
construction owner organizations: A data warehousing approach.” Adult Educ. Q., 41共3兲, 168–187.
Ph.D. dissertation, Florida International Univ., Miami. Meyer, J. 共2003兲. “Questioning design and method: Exploring the value
Azhar, S. 共2007兲. “Improving collaboration between researchers and of action research in relation to R&D in primary care.” Prim. Health
practitioners in construction research projects using action research Care Res. Dev., 4共2兲, 99–108.
technique.” Proc. 43rd ASC National Annual Conf. 共CD-ROM兲, As- Naoum, S. G. 共2001兲. Dissertation research and writing for construction
sociated Schools of Construction 共ASC兲, Flagstaff, Ariz. students, Butterworth-Heinemann, Stoneham, Mass.
Azhar, S., and Ahmad, I. 共2007兲. “A case study of data warehousing Reason, P., and Bradbury, H. 共2001兲. Handbook of action research: Par-
implementation in a public construction owner organization.” Con- ticipative inquiry and practice, Sage Publications, London.
struction data warehouse, 2nd Ed., S-H. Song, ed., Seoul National Rezgui, Y. 共2007兲. “Exploring virtual team-working effectiveness in the
Univ., Seoul, Korea, 15–31. construction sector.” Interact. Comput., 19共1兲, 96–112.
Barker, R., Childerhouse, P., Naim, M., Masat, J., and Wilson, D. 共2004兲. Stake, R. E. 共2000兲. Hand book of qualitative research, 2nd Ed., Sage
“Potential of total cycle time compression in construction: Focus on Publications, London.
program development and design.” J. Constr. Eng. Manage., 130共2兲, Stringer, E. T. 共1996兲. Action research: A handbook for practitioners,
177–187. Sage Publications, London.
Baskerville, R. 共1997兲. “Distinguishing action research from participative Susman, G., and Evered, R. 共1978兲. “An assessment of the scientific
case studies.” J. Sys. Inf. Technol., 1共1兲, 25–45. merits of action research.” Adm. Sci. Q., 23共4兲, 582–603.
Baskerville, R. 共1999兲. “Investigating information systems with action Zuber-Skerritt, O. 共1996兲. New directions in action research, Falmer
research.” Commun. AIS, 2共19兲, 7–17. Press, London.

98 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Qualitative Research: Application of the Delphi Method
to CEM Research
Matthew R. Hallowell1 and John A. Gambatese2

Abstract: Construction engineering and management 共CEM兲 researchers often rely on alternative research techniques when traditional
methods fail. For example, surveys, interviews, and group-brainstorming techniques may not be appropriate for research that involves
confounding factors and requires access to sensitive data. In such an environment, the Delphi technique allows researchers to obtain
highly reliable data from certified experts through the use of strategically designed surveys. At present, the Delphi method has not seen
widespread use in CEM research. This is likely due to variation among studies that implement Delphi in CEM research and ambiguity in
literature that provides guidance for the specific parameters associated with the method. Using the guidance in this paper, the reader may:
共1兲 understand the merits, appropriate application, and appropriate procedure of the traditional Delphi process; 共2兲 identify and qualify
potential expert panelists according to objective guidelines; 共3兲 select the appropriate parameters of the study such as the number of
panelists, number of rounds, type of feedback, and measure of consensus; 共4兲 identify potential biases that may negatively impact the
quality of the results; and 共5兲 appropriately structure the surveys and conduct the process in such a way that bias is minimized or
eliminated.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000137
CE Database subject headings: Construction management; Research; Methodology; Risk management.
Author keywords: Construction management; Research methods; Delphi; Risk management.

Introduction predefined guidelines and are asked to participate in two or more

rounds of structured surveys. After each round, the facilitator pro-
The dynamic and transient nature of construction projects makes vides an anonymous summary of the experts’ input from the pre-
construction engineering and management 共CEM兲 research par- vious survey as a part of the subsequent survey. In each
ticularly challenging. For example, experimental research on subsequent round, participants are encouraged to review the
safety, risk management, innovation, and technology forecasting anonymous opinion of the other panelists and consider revising
is often unrealistic due to the sensitivity and complexity of the their previous response. The goal during this process is to de-
topics. To study such subjects, researchers typically rely on sur- crease the variability of the responses and achieve group consen-
vey and group-brainstorming techniques to collect subjective sus about the correct value. Finally, the process is concluded after
data. The inherent structure of these studies may involve substan- a predefined criterion 共e.g., number of rounds or the achievement
tial bias that researchers must recognize and minimize. Therefore, of consensus兲 is met and a statistical aggregation of the responses
a structured research method that offers researchers the opportu- in the final round determines the results.
nity to control bias and ensure qualification of the respondents is This research method differs from traditional simple survey
desirable. The Delphi technique, originally developed by the methods in that the respondents are certified as experts according
Rand Corporation to study the impact of technology on warfare, to predefined guidelines before the survey process begins, and
allows researchers to maintain significant control over bias in a consensus is achieved through the use of controlled and anony-
well-structured academically rigorous process using the judgment mous feedback provided by the facilitator during multiple rounds.
of qualified experts. This research technique also allows the expert panelists to anony-
The Delphi method is a systematic and interactive research mously interact and allows the facilitator to exhibit strong control
technique for obtaining the judgment of a panel of independent over the interactions among panelists.
experts on a specific topic. Individuals are selected according to In contemporary research, the Delphi method is particularly
useful when objective data are unattainable, there is a lack of
1
Assistant Professor, Civil, Environmental, and Architectural Engi- empirical evidence, experimental research is unrealistic or unethi-
neering, Univ. of Colorado, Boulder, CO 80309-0428 共corresponding cal, or when the heterogeneity of the participants must be pre-
author兲. E-mail: [email protected] served to assure validity of the results. Despite its application to
2
Associate Professor, School of Civil and Construction Engineering, CEM research, this well-defined and highly successful research
Oregon State Univ., 220 Owen Hall, Corvallis, OR 97331. E-mail: technique has not seen widespread use. Several factors may con-
[email protected] tribute to the limited use of the Delphi method. For example,
Note. This manuscript was submitted on July 21, 2008; approved on
some critics claim that many Delphi studies result in low-quality
March 27, 2009; published online on August 22, 2009. Discussion period
open until June 1, 2010; separate discussions must be submitted for indi- findings limited by the facilitator’s survey instruments, poor
vidual papers. This paper is part of the Journal of Construction Engi- choice of experts, lack of effort to reduce bias, unreliable analy-
neering and Management, Vol. 136, No. 1, January 1, 2010. ©ASCE, ses, and limited feedback during the study 共Gupta and Clarke
ISSN 0733-9364/2010/1-99–107/$25.00. 1996兲. Furthermore, the significant variation in the Delphi process

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 99

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 1. Characteristics of Delphi Studies in CEM Research
Number Number Measure
Study Panelists qualifications of rounds of panelist Feedback of consensus
Arditi and Gunaydin 共1999兲 Specific prequalified 3 14 Mean Standard deviation
del Caño and de la Cruz 共2002兲 Specific, not prequalified 1 20 None indicated None indicated
de la Cruz et al. 共2006兲 Specific not prequalified 1 20 None indicated None indicated
Gunhan and Arditi 共2005a兲 Specific, prequalified 2 12 Mean Standard deviation
Gunhan and Arditi 共2005b兲 Specific, prequalified 2 12 Mean Standard deviation
Hyun et al. 共2008兲 Specific, prequalified 3a 7 None indicated None indicated
Robinson 共1991兲 Not specific 3 26 Mean Standard deviation
a
Involved three rounds of independent and unique surveys with no apparent feedback between rounds.

implemented in CEM studies makes the method confusing and Gunaydin 1999; del Caño and de la Cruz 2002; de la Cruz et al.
unclear. 2006; Gunhan and Arditi 2005a,b; Hyun et al. 2008; Robinson
The objective of this paper is to provide CEM researchers with 1991兲. The method was used in a variety of capacities and in-
a standard methodology for implementing the Delphi method in volved varying degrees of rigor. Three of the seven papers studied
rigorous studies intended for publication in journals such as the construction risk, two studied impact factors, one identified dif-
ASCE Journal of Construction Engineering and Management. ferences in perceptions of construction process quality, and one
Using a combination of available literature and the experiences of used the method to define pairwise comparisons for input into the
the writers, a detailed procedure for implementing the method is analytic hierarchy process 共AHP兲. All studies used the qualitative
introduced. Initially, the paper describes the traditional procedures Delphi method to obtain quantitative results. Specifically, re-
required to conduct the research methodology as described in searchers used the Delphi process to quantify risk, impact factors,
literature. In an effort to create a standard and adaptable method- or perception of process quality. While the application of the
ology with application to many types of CEM research, guide- Delphi method appeared to be consistent among construction-
lines and minimum requirements for effective implementation of related publications, there was significant variation on the specific
all phases of the Delphi process are presented. The writers pay characteristics of the research processes. Table 1 summarizes the
special attention to techniques that may be implemented to mini- various characteristics of the publications reviewed. This table
mize judgment-based bias due to the highly subjective nature of was created using the information provided in the publications.
the research method. To clarify the table, studies that used objective criteria to
The standard Delphi method presented in this paper has strong qualify expert panelists are denoted in the table as “specific,
potential for widespread application in CEM research. Character- prequalified.” Those that simply report the qualifications once the
istics of the industry such as dynamic work environments, tran- process is complete are denoted “specific, not prequalified,” and
sient nature, exposure to the elements, coordination of multiple studies that did not indicate any specific requirements are listed as
trades and engineering disciplines, multidisciplinary engineering, “not specific.”
and the workplace hazards often make traditional objective re- There is significant variation in the Delphi studies reviewed.
search infeasible. For example, an experimental research study For example, the requirements for expert qualification, the appro-
that tested the viability of a safety intervention on active construc- priate methods for data collection, analysis and transmission of
tion sites would be unethical as it may increase the risk of injury. controlled and anonymous feedback, the sufficient number of
For such a study, Delphi could serve an alternative methodology rounds of surveys to complete the process, and appropriate mea-
for validating the effectiveness of the intervention without expos- sures of consensus are inconsistent among publications. Four of
ing workers to increased safety risk. Delphi is also preferred to the seven publications indicate that the mean was used as feed-
subjective research methodologies such as traditional surveys or back and standard deviation was used to describe consensus while
focus groups because of the exceptionally high quality of the three did not indicate that any feedback existed between rounds
participants, ability to minimize judgment-based bias, and ease of and no measure of consensus was discussed. Similarly, the
implementation in an increasingly global industry. number of rounds ranged from one to three and in one study,
Hyun et al. 共2008兲, surveys with a completely different focus
were administered in each of the three rounds. In a creative ad-
Literature Review aptation of the Delphi method, Arditi and Gunaydin 共1999兲 used
two distinct panels in an effort to compare the perception of pro-
Since a significant portion of this paper is devoted to the careful cess quality between entry-level and long-term practitioners. In
collection, analysis, and summation of a large body of literature, this study, one of the expert panels consisted of entry-level pro-
this section will be devoted to identifying and discussing fessionals with little experience as these individuals are experts in
construction-related Delphi studies published in peer-reviewed entry-level perception. While there was variation in the applica-
journals and alternative opinion-based research methods. tion of the Delphi method, five of the seven studies incorporate all
of the basic elements. However, significant errors in the other two
Delphi studies were identified.
Delphi in Construction Engineering and Management
In a careful review of the publications by del Caño and de la
Research
Cruz 共del Caño and de la Cruz 2002; de la Cruz et al. 2006兲, the
In total, the writers identified seven studies published in peer- writers appear to have confused the Delphi method with inter-
reviewed journals that used Delphi as the primary or secondary views with highly qualified respondents. In these two publica-
research method to study construction-related topics 共Arditi and tions, there is no discussion of specifics that characterize the

100 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Delphi method such as multiple rounds, feedback, and achieve- other modern method of virtual meetings where experts can com-
ment of consensus. Due to the misconception of some researchers municate with one another in real time. In this method, the panel
regarding the appropriate procedures required, the variability members are not anonymous. The main pitfalls of interacting
among construction-related applications, and the lack of specific groups are bias effects due to potential dominance of one group
guidance in literature, a clear description of the Delphi method member over another and the financial and logistical difficulties
that provides researchers with specific guidance for the appropri- associated with gathering experts in one physical location.
ate use of the Delphi method to achieve high-quality results is
clearly warranted. Nominal Group Technique
There is a great deal of literature outside of ASCE journals that The nominal group technique 共NGT兲 is also referred to as
discusses the Delphi method. Unfortunately, most publications “estimate-talk-estimate” or “brainstorming NGT.” The NGT pro-
simply review the steps required to complete a research study, or cedure uses the same process as Delphi except that feedback is
survey researchers who have used the method to determine the delivered through face-to-face meetings and discussions between
range of applications. There is little focus on the specific details rounds. This method has been proven effective in expediting the
of the research methodology that ultimately defines the level of data collection procedure but often results in significantly more
rigor, the scope of inference, and the reliability and defendability biased results and conformity 共Erffmeyer and Lane 1984兲. Similar
of the results. Because of the nature of the Delphi method, it is to interacting groups, this method is difficult to conduct because it
possible to apply the technique to many fields of study and incor- requires the collection of experts in one geographic location.
porate a diverse group of participants. However, the variability of The three alternative techniques to Delphi may be appropriate
minimum requirements for expert panelists, type of feedback, when standards cannot be met for Delphi, the experts are already
number of rounds, structure of surveys, and measures of consen- located in one physical location, or when time is severely limited.
sus often leads to significant methodological diversity. This al- However, Rowe and Wright 共1999兲 found that literature supports
lows a study to be poorly constructed leading to biased results. Delphi as the preferred method in a review of peer-reviewed stud-
ies that implement expert-based studies.
Alternatives to the Delphi Method
A key decision in any research project is the careful selection of Structure of the Delphi Method
an appropriate research methodology. Delphi has been found to
be the preferred research method when the problem does not lend To orient readers who are unfamiliar with the Delphi process and
itself to precise analytical techniques, questions to be answered to provide a context for the remainder of the paper, the typical
by intuitive judgment supersede questions to be answered by con- events associated with a complete, traditional Delphi study must
crete measurement, and disagreement exists among experts to the be discussed and graphically illustrated. It is vitally important that
extent that a refereed communication process is desired 共Pill the basic steps are understood and incorporated and that any de-
1971; Linstone and Turoff 1975兲. viation from the standard process is recognized, discussed, and
The Delphi method is not appropriate for all studies. Research- justified in any resulting publication. Most Delphi studies pub-
ers are always encouraged to explore all reasonable and objective lished in ASCE journals do not recognize or justify deviations in
options for the collection of such data before considering a quali- the methodology, making the publications unnecessarily vague.
tative method such as Delphi. For example, objective and empiri- The typical order of events of the Delphi process is summa-
cal data that can be measured or collected in a controlled rized in Fig. 1. This flowchart represents the order of events and
scientific method are always preferred to subjective opinions. The illustrates the role of multiple rounds. This general structure is
Delphi method is also inappropriate when the general recom- suggested for all Delphi studies applied to CEM research.
mended structure cannot be followed, the method is applied for a During design of a recent Delphi study, the writers found that
purpose other than achieving the consensus of a group of experts, guidance for the basic steps of the Delphi procedure illustrated by
the ideals of the process are violated, objective data are available, Fig. 1 was abundant 共Hallowell 2008兲. However, the specific de-
experts are unavailable or unwilling to participate, and when the sign of the Delphi process was limited, outdated, and difficult to
facilitator does not have the appropriate experience, education, or obtain. In particular, there was no literature that addressed con-
time to lead the study or interpret results 共Veltri 1985兲. When the trols that may be implemented to reduce the effects of biased
Delphi process is inappropriate there are three alternative meth- responses.
ods of obtaining expert opinion that may be considered: staticized The following section of the paper is devoted to the creation of
groups, interacting groups, and the nominal group technique. specific guidelines for the rigorous implementation of the Delphi
Each of these methods is described next. method. For reference, the writers highlight specifics associated
with a recent study conducted by Hallowell 共2008兲 that attempted
Staticized Groups to quantify safety and health risks. The study successfully used
The staticized groups research method is identical to the Delphi the Delphi method to obtain probability and severity values, on
method with the exclusion of feedback or iteration. That is, the 1–10 scales, from expert panelists. The specific Delphi process
method represents the aggregate responses of certified experts used for this study is discussed in detail and is summarized in
from initial questioning. Therefore, there is no interaction be- Fig 1. The writers believe that the methods described lead to
tween panel members. Some writers such as Erffmeyer and Lane highly reliable results and are applicable to many types of CEM
共1984兲 prefer this methodology because panel members are less research.
likely to conform about an incorrect value.
Expertise Requirements
Interacting Groups
This method, otherwise known as “focus groups,” involves col- In the Delphi process, the most important facet of a panel member
lecting experts in one physical location, via teleconference, or is their level of expertise. Nevertheless, the characteristics re-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 101

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Identify Select experts Validate expert
Identify potential based on status and inform
research
experts predefined criteria panelists of study
question
req’ts

Develop Transmit Collect and

Evaluate
questionnaire questionnaire to analyze round
consensus
using methods to expert panel responses
Start minimize bias
Round
n+1 Target consensus HAS
been achieved
Develop feedback
Target consensus has NOT been achieved
for panelists of
subsequent round

Report results

Fig. 1. Suggested Delphi procedure

quired to define an individual as an “expert” are equivocal. As in the following: demonstration of knowledge which members of
nearly all studies, a major objective is to obtain an unbiased rep- recognized professions and society at large judge as being of ex-
resentative sample. Therefore, the method of selecting expert pert quality, exhibition of expertise by willingly submitting for
panel members should be strategic and unbiased as well. Two critical examination, various publications related to the discipline
studies were identified that provide explicit guidance for qualify- involved, participation in professionally related forums, confer-
ing individuals as experts. Unfortunately, the requirements dis- ences, and workshops with colleagues interested in advancing the
cussed in these publications are dissimilar and vague 共Rogers and related profession. While these guidelines are flexible and are
Lopez 2002; Veltri 1985兲. likely to qualify many available participants, the quality of the
Rogers and Lopez 共2002兲 required that all expert panel mem- results may be compromised. Because of the lack of complete
bers meet at least two of the following requirements 共within the guidance, the writers created two sets of specific expertise re-
field of study under examination兲: authorship, conference pre- quirements for use in a recent study based on literature and char-
senter, member or chair of committee, employed in practice with acteristics of well-known and widely recognized experts
5 years of experience, and employed as a faculty member at an 共Hallowell 2008兲.
institute of higher learning. One will note that the authorship and When a diverse group of highly qualified and well-rounded
conference presenter requirements are especially vague and open experts is desired for rigorous CEM research, the writers suggest
to interpretation. In a similar study, Veltri 共1985兲 incorporated that panelists meet at least four specific requirements listed in
more flexible guidelines requiring that panel member meet one of Table 2. Requiring that panelists meet at least four requirements

Table 2. Guidelines for the Rigorous Implementation of the Delphi Research Method
Characteristic Minimum requirement
Identifying potential experts Membership in a nationally recognized committee in the focus area of the
research 共e.g., ASCE Site Safety Committee兲
Primary writer of publications in ASCE journals
Known participation in similar expert-based studies
Qualifying panelists as experts Experts must satisfy at least four of the following criteria in the topics related
to the research:
• Primary or secondary writer of at least three peer-reviewed journal articles
• Invited to present at a conference
• Member or chair of a nationally recognized committee
• At least 5 years of professional experience in the construction industry
• Faculty member at an accredited institution of higher learning
• Writer or editor of a book or book chapter on the topic of construction
safety and health, or risk management
• Advanced degree in the field of civil engineering, CEM, or other related
fields 共minimum of a BS兲
• Professional registration such as Professional Engineer 共PE兲, Licensed
Architect 共AIA兲, Certified Safety Professional 共CSP兲, Associated Risk
Manager 共ARM兲
Number of panelists 8–12
Number of rounds 3
Feedback for each round
Round 1 Data from preliminary research or archived data 共if available兲
Round 2 Median response from Round 1
Round 3 Median response from Round 2 and reasons for outlying responses
Absolute deviation 共with consensus indicated by a value ⬍1 / 10 of the range
Measuring consensus of possible values for quantitative studies兲

102 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 3. Flexible Point System for the Qualification of Expert Panelists writers created two independent panels of 12 and 15 members
Points each 共Hallowell 2008兲. This number was easily managed by a
Achievement or experience 共each兲 full-time facilitator.
Professional registration 3
Year of professional experience 1 Number of Rounds
Conference presentation 0.5 The purpose of multiple rounds is twofold. The main objective is
Member of a committee 1 to reach consensus by reducing variance in responses. The second
Chair of a committee 3 purpose is to improve precision. Both of these objectives are
Peer-reviewed journal article 2 achieved through the use of controlled feedback and iteration. It is
Faculty member at an accredited university 3 assumed, and supported by literature, that convergence to a col-
Writer/editor of a book 4 lective opinion and precision 共i.e., “closeness” to actual state兲 are
Writer of a book chapter 2 improved as a result of each round. However, literature provides
Advanced degrees: very little guidance for the acceptable number of iterations.
BS 4 A summary of peer-reviewed Delphi studies indicates that the
MS 2 number of rounds ranged from two to six 共Dalkey et al. 1970;
Ph.D. 4 Gupta and Clarke 1996; Linstone and Turoff 1975; Pill 1971兲.
Over one-half of these studies found acceptable convergence after
three or fewer iterations. In fact, Dalkey et al. 共1970兲 suggests
in the field under examination encourages a healthy balance of that the Delphi results are most accurate after round two and
academic and professional experience and ensures that panelists become less accurate as a result of additional rounds. However,
have distinguished themselves as experts on the topic. the use of at least three rounds allows the facilitator to obtain
The requirements listed in Table 2 are likely to lead to a well- reasons for outlying responses as a part of the second round and
qualified and diverse panel with a wide range of experience. In to report these reasons as feedback in round three. By reviewing
some studies, however, researchers may wish to have a more the reasons for outlying responses in the third and final round,
narrow focus requiring the qualification of specific types of indi- panelists are more likely to consider all options and reach con-
viduals. For example, a study may not require academic experi- sensus about the correct value rather than conforming to an incor-
ence if such experience is irrelevant or unlikely to add value. In rect opinion. Studies that include only two rounds are incapable
such a study, having panelists with academic accolades and pub- of adequately identifying outlying viewpoints, obtaining justifica-
lishing experience may be inappropriate. In this case, a relative tion, and sharing this information with other panelists.
point system that allows one to select specific expert qualities
may be more appropriate. The writers offer a relative point sys- Feedback Process
tem in Table 3 for qualifying experts that may be more flexible for
some studies. This point system is based on the relative time As indicated earlier, the feedback process is the mechanism for
commitment required to successfully complete each of the informing panel members of the opinions of their anonymous
achievements or experiences and is based on the best judgment of counterparts. Without iterating and providing this controlled feed-
the writers and practices of professional licensing agencies. One back, the process could not be called, “Delphi.” The most com-
should note that this point system may need to be adapted to mon feedback provided in subsequent rounds is simple statistical
meet the specific goals of a research project. In order to meet a summaries such as median, mean, or quartile ranges. Some stud-
minimum level of qualification using the point system shown in ies provide additional information such as the arguments from the
Table 3, it is suggested that panelists score at least one point in panel members whose opinions are outside the interquartile range.
four different achievement or experience categories and a mini- Including anonymous justification for outlying observations en-
mum of 11 total points in order to qualify for participation. sures that all opinions are considered.
Best 共1974兲 found that Delphi groups that were given reasons
as part of the feedback in addition to median and range of esti-
Number of Panel Members mates were significantly more accurate than Delphi groups that
The impact of the number of panelists on the accuracy and effec- were provided with only the latter. Despite conflicting evidence in
tiveness of the method has been studied by Brockhoff 共1975兲 and social psychology regarding the influence of various feedback
Boje and Murnighan 共1982兲. Neither study found a significant methods on accuracy, conformity, change in opinion, and consen-
correlation between the number of panel members and effective- sus, Delphi studies that included reasons and simple statistical
ness. A summary in Rowe and Wright 共1999兲 indicates that the summaries lead to more accurate results 共Rowe and Wright 1999兲.
size of a Delphi panel has ranged in peer-reviewed studies from a One should note that none of the seven construction-related
low of three members to a high of 80. Since most studies incor- publications appear to have used reporting reasons as a form of
porate between eight and 16 panelists, a minimum of eight is feedback.
suggested. The specific number of panelists should be dictated by
the characteristics of the study such as the number of available
Measuring Consensus
experts, the desired geographic representation, and the capability
of the facilitator. The selected number of panel members should One of the more difficult aspects of the Delphi process is the
also take into consideration that some panelists may decide to appropriate method of measuring consensus. While it is common
drop out of the study due to other commitments or disinterest. A to use variance as a measure of consensus, guidance that de-
sufficient number of panelists should be selected at the start of the scribes the level of variance that represents consensus is not avail-
process to ensure a qualified panel at the end of the study assum- able in literature. The writers believe that such guidance is not
ing some will not complete each round. In a recent study the provided because the data collected for nearly every study is

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 103

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
unique. Therefore, it would be inappropriate to indicate a certain however, be evaluated by researchers on a case-by-case basis to
level of variance that represents adequate consensus for all ensure the quality of the study.
studies.
In the Delphi study completed by Hallowell 共2008兲 the respon-
dents’ consensus was considered to have been achieved when Judgment-Based Bias in Risk Studies
the absolute deviation was within one unit on a 1–10 scale 共i.e., Judgment is a skill that is used in decision making when disput-
+ / −5% deviation about the median兲. The absolute deviation was able factual information is absent and is the primary mechanism
used as a measure of consensus in lieu of the standard deviation for data collection in any Delphi study. One or more of the fol-
because it measures variability in response about the median lowing three classes of judgment are used to reason and eventu-
rather than the mean. For this study, the median values repre- ally make choices:
sented the results of this study because the median is less likely to • Diagnostic: diagnostic judgment involves using intuition, vi-
be influenced by biased results. sualization, organization and structuring of evidence, and the
A summary of the suggested guidelines for the Delphi process understanding of relationships to reach a conclusion.
is provided in Table 2. These guidelines are suggested by the • Inductive: inductive reasoning requires the synthesis of evi-
writers to ensure a high level of rigor in any CEM study. How- dence and information from a variety of sources. Inducing
ever, one may be required to augment or modify the specific requires use of an individual’s awareness of signs and evi-
requirements and parameters in Table 3 to meet the specific needs dence to draw conclusions. The ability to draw correct conclu-
of a particular study. One should also note that the guidelines sions using inductive reasoning is directly related to an
offered in Table 2 are for the standard application of the Delphi individual’s experience, observations, and ability to recognize
method to any type of study. These guidelines do not include evidence.
specific controls to minimize judgment-based bias during the pro- • Interpretive: interpretive reasoning involves the recognition of
cess, an important aspect of any rigorous study. This topic is patterns, spatial relationships, correlations and causal relation-
discussed as it relates to risk quantification in the subsequent ships. Individuals who can effectively reason through interpre-
section of this paper. tation must be able to critically review, evaluate and develop
In addition to following the suggested guidelines, researchers and context for a particular scenario.
should take care when developing the Delphi survey to ensure In a Delphi study it is assumed that identified experts are
that the length is reasonable to ensure an adequate response rate. uniquely capable of providing proficient judgment using one or
Also, pilot tests of the surveys are suggested to ensure that the more of the three reasoning methods identified earlier due to their
level of detail is appropriate for the study, the role of the expert extraordinary experience or intelligence. One should note, how-
panelists is well defined, and survey instructions are easy to fol- ever, that various sources of bias may exist despite the panelists’
low. status as certified experts. Any panelist is likely to be susceptible
to one or more of the eight major forms of judgment-based bias
during the Delphi process to some degree. Although there are
Methods to Minimize Bias hundreds of types of biases, the writers selected eight major forms
of bias because of their potential ability to negatively impact the
Rigorous and defendable research studies must hold the minimi- quality of the results of a Delphi study. These factors are not
zation of bias paramount. Bias in judgment is important to con- necessarily controlled when following traditional guidelines for
sider because cognitive shortcuts that distort the true nature of Delphi implementation. Special attention is paid to biases that
opinion or observation may lead to inaccuracies in judgment may affect risk perceptions.
共Heath and Tindale 1994兲. Unlike objective methods where bias is
mainly introduced by the researcher, the success of the Delphi Collective Unconscious
process depends on the unbiased judgment of experts. Despite the Simply, the theory of collective unconscious, otherwise known as
fact that bias can be detrimental to Delphi studies, no literature the “bandwagon effect,” states that decision makers tend to join a
reviewed specifically addresses bias during the Delphi process popular trend. In other words, individuals are likely to uncon-
with only one exception. Anonymity, one of the elements that sciously feel pressure to conform to the common or standard be-
characterize the Delphi process, is incorporated to minimize one liefs within a particular group. According to Durkheim 共1982兲,
form of judgment-based bias: dominance. Other forms of bias individual beliefs are limitless unless constrained or directed by
such as primacy, recency, and the Von Restorff effect may also social forces such as peer pressure or dominance. The bandwagon
adversely affect the study if not recognized and controlled. effect occurs when social forces compel an individual to conform.
This section of the paper is devoted to the identification and The collective unconscious must be considered in a Delphi study
discussion of eight judgment-based biases that may adversely af- because bias occurs when a decision maker conforms to popular
fect a Delphi study that aims to quantify risks in the construction belief without examining the merits of the position.
industry. The writers chose to highlight risk studies because major
forms of bias have been identified and controlled in a recent Contrast Effect
Delphi study completed by Hallowell 共2008兲 and three of the The contrast effect occurs when the perception of a given subject
eight construction-related Delphi studies identified in literature is enhanced or diminished by the value of the immediately pre-
specifically address risk quantification. Furthermore, identifica- ceding subject. Bjarnason and Jonsson 共2005兲 contend that an
tion of all types of cognitive bias that may affect all types of CEM individual’s evaluation of a criterion may be directly influenced
studies is outside the scope of this paper because literature in the by a previous exposure of substantially higher or lower value. In
field of social psychology has identified hundreds of cognitive theory, the contrast effect can cause significant bias, especially
biases. It is the opinion of the writers that implementing the con- when Delphi panel members are to rate risks or identify relative
trols suggested for the eight biases discussed in this paper is suf- differences among multiple factors. In such a case, a significant
ficient for the purpose of most CEM research. This should, contrast bias may exist when panelists are asked to rate back-to-

104 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
back factors of substantially different values. Since most Delphi tively difficult to control for, one method of controlling for this
studies in construction research involve rating risks or factors of bias is the removal of panelists who have recently experienced
some kind, the Delphi questionnaire must be structured to mini- events related to the study.
mize contrast effects.
Primacy Effect
Neglect of Probability The primacy effect is a relatively subtle form of cognitive bias.
There are many cases where individuals underestimate the role of This effect results from the unconscious assignment of impor-
probability in the subjective quantification of risk. This bias in- tance to initial questions, observations, or other stimuli. The
volves the disregard of likelihood when making a decision under theory states that individuals are inherently more concerned with
uncertainty. For example, Rottenstreich and Hsee 共2001兲 found initial stimuli. That is, relatively speaking, the first stimulus will
that while the average individual was willing to pay $7 to avoid be considered more important than the final observation. In terms
a 1% chance of a painful electric shock, the same individuals of risk quantification, an individual is more likely to assign im-
were only willing to pay $10 to avoid a 99% chance of the same portance to a risk scenario at the beginning of a Delphi survey
shock. In this case, the subjects have devalued the concept of than at the end of the survey.
probability in this scenario. Any study that involves risk quanti-
fication or ratings of likelihood may be susceptible to the neglect Dominance
of probability bias. Controls are especially important because re- Dominance occurs when one, usually very vocal or intimidating
searchers suggest that the neglect of probability is relatively com- group member, exhibits great control over the ratings of the other
mon 共Martin 2006兲. members. This common source of bias is typically controlled
through anonymity and equal weighting of responses. These
Von Restorff Effect means for limiting the bias ensure that each panel member
submits their true opinion without the influence of a dominant
The Von Restorff Effect was first introduced to the field of psy-
member.
chology when subjects were found to recognize and remember
The eight biases discussed represent the salient biases for most
relatively extreme events more often and more accurately than
construction-related studies. Depending on the nature of the re-
less extreme events 共Restorff 1933兲. Simply, individuals are more
search, the facilitator may need to review social psychology lit-
likely to remember events associated with severe outcomes
erature to identify other potential biases. Once biases are
thereby distorting the perception of probability.
identified, it is essential that the facilitator take all reasonable
This phenomenon is likely to bias risk perceptions because
steps available to control and minimize their potential effects.
more extreme events are likely to be recalled. It is especially
The following section defines some methods for designing a
important to consider this bias when soliciting risk perceptions Delphi study to minimize and avoid the eight biases discussed
because individuals are more likely to overestimate probability earlier.
values when an especially high magnitude is involved 共Krimsky
and Golding 1992兲. This effectively creates an artificially inflated
risk score for potential events associated with a higher level of
severity. Minimizing the Effects of Biases
In the field of statistics, a prominent method of bias reduction is
Myside Bias
the use of randomization. Randomization is a control by which a
According to Perkins 共1989兲, myside bias occurs when an indi- researcher ensures that every subset of the greater population has
vidual generates arguments only on one side of an issue. Perkins an equal chance of being selected. Randomization may be
provided a demonstration of this bias by asking the study partici- achieved through the use of workbooks with random number
pants to list the thoughts that occur to them when considering tables, using the last four digits in telephone directories, or by
controversial subjects. The majority of the participants recorded using a pseudorandom number generator such as MS Excel. For
thoughts that pertained to only one side of the controversy. Ac- construction risk studies, randomization can be used to reduce
cording to 共Baron 2003兲, participants can be, “easily prompted for bias associated with the contrast, Von Restroff, and primacy ef-
additional arguments on the other side, although prompting for fects. The remaining five biases may be controlled by the strategic
further arguments on their favored side is less effective. So the design of the survey and feedback mechanisms. Table 4 provides
failure to think of arguments on the other side is typically not the the reader with a summary of controls that may be implemented.
result of not knowing them.” A short description of each of the six controls listed in Table 4 is
The persistence of irrational belief is generally a result of one’s provided next. These controls were successfully implemented on
personal opinion and has little basis in pure fact. This phenom- a recent study 共Hallowell 2008兲.
enon also exists when uncompromising individuals do not seek 1. Randomize questions in the survey: the order of questions in
objective viewpoints. Myside bias is especially important to con- the Delphi surveys may be randomized for each Delphi panel
sider in Delphi studies because the chief objective is to reach member. In addition, a new randomized order for each pan-
consensus among the experts. Therefore, controls that ensure the elist should be created for each round. Random numbers may
consideration of multiple viewpoints are essential. be generated and assigned to each question using a random
number generator and the relative ranking 共e.g., highest to
Recency Effect lowest兲 may be used to determine the order of the questions.
The recency effect occurs when subjects are more likely to arti- 2. Include reasons in controlled feedback: as indicated, Delphi
ficially inflate risk ratings because similar incidents have recently groups that were given feedback that included reasons for
occurred in their personal lives. That is to say, recent events are specific panelist responses in addition to median and range of
given inappropriate levels of salience in relation to others. The estimates were significantly more accurate than Delphi
effect of recency is relatively common. While recency is rela- groups that were provided with only the latter. Expert panel-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 105

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 4. Controls for Bias in the Delphi Process affected by recency bias, the results may be omitted using
Bias Control/countermeasure statistical justification.
It is important for one to understand the limits of the controls
Collective Include reasons in the controlled feedback suggested. Some of the controls, such as randomization of ques-
unconscious to the Delphi panel for each round
tion order theoretically eliminate contrast and primacy biases
Contrast effect Randomize the order of questions for each while others such as reporting medians only partially mitigate
panel member and for each round, and
recency bias. Table 5 summarizes the impacts of the controls on
report final results as a median
the various biases. In this table, the complete removal of a
Neglect Require that the probability ratings and
judgment-based bias is denoted “*x*“ while the partial removal
of probability severity ratings for each risk are recorded
independently of a bias is denoted “x.” In two recent risk studies, the writers
Von Restorff Include reasons in controlled feedback and
implemented all of the controls listed in Table 5 and were confi-
effect conduct multiple rounds of surveys dent that bias was successfully minimized. Also, one should note
Myside bias Include reasons in the controlled feedback and
that the controls developed for Table 5 may also minimize other
report final risk ratings as a median forms of bias not previously mentioned in this paper.
Recency effect Remove individuals who have experienced
recent events, remove outlying observations,
conduct multiple rounds, and report results
as a median Conclusions
Primacy effect Randomize the order of questions for each
panel member The challenging nature of construction management research re-
Dominance Ensure anonymity of expert panelists quires the use of alternative research techniques in studies where
traditional methods are not applicable or effective. Many topics of
CEM research, such as safety and health, risk management, fore-
ists should be asked to provide a very brief justification for casting, and innovation, are complex, involve many confounding
their ratings during the second round. This justification factors, or are extremely sensitive precluding the use of tradi-
should be summarized and reported as part of the controlled tional research methodologies. Despite its potential application,
feedback in the third round. the Delphi method has seen minimal use in CEM research. In
3. Conduct multiple rounds of surveys and maintain anonymity: fact, a review of literature indicated that only eight major studies
iteration is an essential component of any Delphi study. The implemented Delphi as the primary or secondary research
primary role of multiple rounds is to achieve a high degree of method. Also, a significant variation in methodological approach
consensus among panel members. The potential reduction in was found to exist in this small body of literature. Additionally,
bias is rarely discussed. Iteration involves the redistribution these publications ranged significantly in quality. The writers be-
of the Delphi survey accompanied with controlled feedback. lieve that this variation is a result of the inconsistencies and am-
Maintaining anonymity is essential for avoiding the influence biguity of literature that provides guidance for the method.
of dominant panel members. In an effort to consolidate a relatively large body of literature,
4. Require independent probability and severity ratings: sur- the writers have created a standard but flexible method that may
veys should be structured such that panel members will be be applied to most types of CEM research. Using the guidance in
required to consider probability and severity values sepa- this paper, the reader may: understand the merits, appropriate ap-
rately for each risk scenario. plication, and appropriate procedure of the traditional Delphi pro-
5. Report medians: results should be reported in terms of the cess; identify and qualify potential expert panelists according to
median rather than the mean because the median response is objective guidelines; select the appropriate parameters of the
less likely to be affected by biased responses. study such as the number of panelists, number of rounds, type of
6. Remove members who experienced recent events: in the in- feedback, and measure of consensus; identify potential biases that
troductory survey, Delphi members should be asked a series may negatively impact the quality of the results; and appropri-
of questions related to their recent experience with construc- ately structure the surveys and conduct the process in such a way
tion risks. Respondents who indicate recent exposure should that bias is minimized or eliminated. By following the guidelines
be closely monitored during the Delphi process or removed suggested in this paper, high-quality, minimally biased results are
from the study completely. If the individual appears to be expected.

Table 5. Bias Reduction Resulting from Controls Implemented during Delphi

Bias
Neglect
Collective of Von Myside
Control unconscious Contrast probability Restorff bias Recency Primacy Dominance
Randomize question order *x * *x*
Include reasons in feedback x x x
Iteration and anonymity x * x*
Separate probability and severity ratings * x*
Report medians x x x x x
Remove members who experienced recent events * x*

106 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
References Delphi technique: A bibliography 共1975–1994兲.” Technol. Forecast.
Soc. Change, 53共2兲, 185–211.
Arditi, D., and Gunaydin, H. M. 共1999兲. “Perceptions of process quality Hallowell, M. 共2008兲. “A formal model of construction safety and health
in building projects.” J. Manage. Eng., 15共2兲, 43–53. risk management.” Ph.D. dissertation, Oregon State Univ., Corvallis,
Ore.
Baron, J. 共2003兲. Myside bias in thinking about abortion, University of
Heath, L., and Tindale, R. S. 共1994兲. Applications of heuristics and biases
Pennsylvania Press, Philadelphia.
Best, R. J. 共1974兲. “An experiment in Delphi estimation in marketing to social issues, Plenum, New York.
decision making.” J. Mark. Res., 11共1兲, 448–452. Hyun, C., Cho, K., Koo, K., Hong, T., and Moon, H. 共2008兲. “Effect of
Bjarnason, T., and Jonsson, S. H. 共2005兲. “Contrast effects in perceived delivery methods on design performance in multifamily housing
risk of substance use.” Subst Use Misuse, 40共1兲, 1733–1748. projects.” J. Constr. Eng. Manage., 134共7兲, 468–482.
Boje, D. M., and Murnighan, J. K. 共1982兲. “Group confidence pressures Krimsky, S., and Golding, D. 共1992兲. Social theories of risk, Praeger,
decisions.” Manage. Sci., 28共1兲, 1187–1196. London.
Brockhoff, K. 共1975兲. “The performance of forecasting groups in com- Linstone, H. A., and Turoff, M. 共1975兲. The Delphi method: Techniques
puter dialogue and face-to-face discussion.” The Delphi method: and applications, Addison-Wesley, Reading, Mass.
Techniques and applications, Addison-Wesley, Reading, Mass., 291– Martin, W. 共2006兲. Theories of judgment, University Press, New York.
321. Perkins, D. N. 共1989兲. “Reasoning as it is and could be: An empirical
Dalkey, N., Brown, B., and Cochran, S. 共1970兲. “Use of self-ratings to perspective.” Thinking across Cultures: 3rd International Conf. on
improve group estimates.” Technological Forecasting, 1共3兲, 283–291. Thinking, Erlbaum, Hillsdale, N.J.
de la Cruz, M., del Caño, A., and de la Cruz, E. 共2006兲. “Downside risks Pill, J. 共1971兲. “The Delphi method: Substance, context, a critique and
in construction projects developed by the civil service: The case of annotated bibliography.” Socio-Econ. Plan. Sci., 5共1兲, 55–71.
Spain.” J. Constr. Eng. Manage., 132共8兲, 844–852. Restorff, H. V. 共1933兲. “Über die Wirkung von Bereichsbildungen im
del Caño, A., and de la Cruz, M. 共2002兲. “Integrated methodology for Spurenfeld.” Psychol. Forsch., 18, 299–334.
project risk management.” J. Constr. Eng. Manage., 128共6兲, 473–485. Robinson, J. 共1991兲. “Delphi methodology for economic impact assess-
Durkheim, E. 共1982兲. The rules of the sociological method, Free Press, ment.” J. Transp. Eng., 117共3兲, 335–348.
New York. Rogers, M. R., and Lopez, E. C. 共2002兲. “Identifying critical crosscultural
Erffmeyer, R. C., and Lane, I. M. 共1984兲. “Quality and acceptance of an school psychology competencies.” J. Soc. Psychol., 40共2兲, 115–141.
evaluative task: The effects of four group decision-making formats.” Rottenstreich, Y., and Hsee, C. K. 共2001兲. “Money, kisses, and electric
Group Organiz. Stud., 9共4兲, 509–529. shocks: On the affective psychology of risk.” Psychol. Sci., 12, 185–
Gunhan, S., and Arditi, D. 共2005a兲. “Factors affecting international con- 190.
struction.” J. Constr. Eng. Manage., 31共3兲, 273–282. Rowe, G., and Wright, G. 共1999兲. “The Delphi technique as a forecasting
Gunhan, S., and Arditi, D. 共2005b兲. “International expansion decision tool: Issue and analysis.” Int. J. Forecast., 15, 353–375.
for construction companies.” J. Constr. Eng. Manage., 131共8兲, 928– Veltri, A. T. 共1985兲. “Expected use of management principles for safety
937. function management.” Ph.D. dissertation, West Virginia Univ.,
Gupta, U. G., and Clarke, R. E. 共1996兲. “Theory and applications of the Morgantown, W.Va.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 107

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Mixed Method Research: Fundamental Issues of Design,
Validity, and Reliability in Construction Research
Deborah A. Abowitz1 and T. Michael Toole2

Abstract: The fact that people play key roles in nearly all aspects of construction suggests that effective construction research requires
proper application of social science research methods. This is particularly true for researchers studying topics that involve human actions
or behavior in construction processes, such as leadership, innovation, and planning. In social science research, no single method of data
collection 共survey, experiment, participant observation, or unobtrusive research兲 is ideal. Each method has inherent strengths and weak-
nesses. Careful attention to the methodological ABCs of the design process, as discussed here, can enhance the validity and reliability of
a given study. Combining quantitative and qualitative approaches in research design and data collection, however, should be considered
whenever possible. Such mixed-methods research is more expensive than a single method approach, in terms of time, money, and energy,
but improves the validity and reliability of the resulting data and strengthens causal inferences by providing the opportunity to observe
data convergence or divergence in hypothesis testing.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000026
CE Database subject headings: Research; Methodology; Measurement; Data analysis; Construction management.
Author keywords: Research methods; Mixed methods; Social science; Concept measurement; Data analysis.

Introduction pant observation, or interviews and archival data兲, allows us to

balance the strengths and weaknesses of each approach and, if
Construction is essentially a “social” process. In effect, construc- our theoretical premises are correct, converge on a common pat-
tion can be considered to be the application by people of technol- tern or result. Combining multiple methods in this way, a form of
ogy developed by people to achieve goals established by people triangulation takes place however, within a larger methodological
involving the erection or retrofitting of infrastructure and build- context. Using multiple or mixed methods “affects not only mea-
ings. The fact that people play key roles in nearly all aspects of surement but all stages of research” 共Brewer and Hunter 1989,
the construction process suggests that in order to understand the p. 21兲. To utilize a mixed method approach properly, the key is to
human or social factors, effective construction research requires incorporate sound methodological principles at each stage of the
the proper application of social science research methods. Even design process.
when the concepts in our research 共such as alternative delivery The discussion here is intended for construction researchers
methods兲 do not ostensibly involve individual human traits or whose particular methods intersect with those of social scientists
behavior per se, social factors can complicate the research process 共that is, those who use surveys, ethnographic research, and the
and jeopardize the results. Whether these factors derive from spe- like兲, whether as new doctoral students in the field or as construc-
cific actions of, or decisions by, researchers in the design stage or tion faculty performing research in social science areas of con-
in the field, and/or from individual informants in organizational struction for the first time. This paper represents a contribution to
settings, the social nature of construction research, and the impli- the literature not because it presents innovative construction re-
cations of it as such, need to be carefully considered. search, but because it synthesizes the substantive and method-
The focus of this paper is on some of the hazards we face in ological expertise of a construction engineer with that of a social
the application of social science methods to construction research science methodologist to highlight and summarize critical social
and on the utility of applying a mixed or multimethod approach to research problems in construction research and the utility of a
enhance the reliability and validity of our results. Using multiple mixed rather than singular methodological approach.
methods, in particular mixing qualitative and quantitative tech- The importance of addressing social research problems in con-
niques, on a given research project 共such as surveys and partici- struction and the need for mixed research methods in particular
can be supported by examining the papers published in the five
1 issues of the Journal of Construction Engineering and Manage-
Professor, Sociology, Bucknell Univ., Lewisburg, PA 17837. E-mail:
[email protected] ment 共JCEM兲 between February and August 2008. Analyses of
2
Associate Professor, Civil and Environmental Engineering, the entire manuscripts 共not just the abstracts兲 indicated that social
Bucknell Univ., Lewisburg, PA 17837 共corresponding author兲. E-mail: science constructs 共defined shortly兲 were either the focus of the
[email protected] paper or were key parts of the background discussion for 22 of
Note. This manuscript was submitted on July 29, 2008; approved on
the 43 papers published, yet none of the papers used multiple
December 29, 2008; published online on March 27, 2009. Discussion
period open until June 1, 2010; separate discussions must be submitted methods. Additional relevant findings from the writers’ analysis
for individual papers. This paper is part of the Journal of Construction of the five issues of the JCEM will be mentioned later in this
Engineering and Management, Vol. 136, No. 1, January 1, 2010. paper. Following a brief review of the existing literature on con-
©ASCE, ISSN 0733-9364/2010/1-108–116/$25.00. struction research methods, we review basic methodological prin-

108 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Table 1. Four Types of Validity in Social Science Research 共Cook and significance 共statistical conclusiveness validity兲, helps us under-
Campbell 1979兲 stand the broader problems of causal inference, generalizability,
Type of measurement reliability and validity, and statistical inference in
validity Key issue construction and other types of research. These problems in
research design, and the application of multiple or mixed method
Construct Do the indicators capture the expected relationships
validity among the concepts being researched?
approaches to address them, are discussed in the following
sections.
Statistical Are the relationships between hypothesized
conclusiveness independent and dependent variables statistically
validity significant?
Methodological ABCs
Internal Has the research truly demonstrated a causal link
validity between the hypothesized variables, or are there
Although the emphasis in the examples discussed here is on the
plausible alternative explanations for the statistical
association between the independent and dependent application of social science methods in construction research,
variables? these principles pertain whether we are studying social phenom-
ena or aspects of the physical world. The lessons highlighted
External Are the apparent relationships found within the
here draw from the first writer’s extensive experience teaching
validity sample’s experimental subjects generalizable to the
larger population assumed in the hypothesis?
social science research methods, conducting survey research
and quantitative analyses, and writing peer reviews for journals,
book publishers, and granting agencies both in the United States
ciples related to the application of a mixed methodological and abroad. They also draw from the second writer’s experiences
design. as a construction researcher, paper reviewer and JCEM specialty
editor.
Surveys, questionnaires, experiments, ethnographic observa-
Review of the Existing Literature on Construction
tion, and unobtrusive techniques are all valuable research tools,
Research Methods
but each yields somewhat different perspectives on a research
The literature on construction research methods is rather sparse, question. Each is subject to particular problems of validity and
which would seem to confirm the need for this special issue of reliability; each has distinct limits of generalizability. Whether
the JCEM. Although the book is not well known in the United using one method to generate and collect data, or combining
States, an excellent contribution is made by Fellows and Liu methods in a mixed-method approach, researchers need to be cog-
共2008兲, which focuses on the chronological processes that should nizant of the consequences of each of the methodological choices
be followed to ensure successful research. This book also focuses they make.
on the philosophical issues related to research methodologies.
Three 1997 papers also discussed research methods at the broad-
Proper Research Planning and Design Enables
est levels, debating the relative merits of theoretical versus em-
Successful Data Collection
pirical papers and qualitative versus statistical or quantitative
research 共Seymour et al. 1997; Raftery et al. 1997; Runeson As Babbie 共2008, p. 122兲 notes, “Research design involves a set
1997兲. Loosemore 共1999兲 identified cultural differences, espe- of decisions regarding what topic is to be studied among what
cially those involving communication, that make one category of population, with what research methods, for what purpose.”
research methods more appropriate than another. Walker 共1997兲 Proper research planning and design therefore involve several dis-
discussed his doctoral research process as a case study on the tinct issues. For example, if the topic is leadership in the construc-
challenges of obtaining data from thirty-three projects and ana- tion industry and its effect on firm performance, we have to
lyzing them using linear regression. Similarly, El-Diraby and determine which subjects to include from among firm leaders 共the
O’Connor 共2004兲 used the collection and analysis of bridge con- sampling technique and sample size兲 and the method共s兲 of data
struction data as a case study to summarize key methodological collection 共an experiment, survey, observational study, and so on兲.
issues taken from Cook and Campbell 共1979兲, which is one of the Each of these decisions affects the quality and quantity of data
major works on experimental design in the social sciences 关along collected.
with Campbell and Stanley 共1963兲兴 关for a thorough introductory With regard to the sampling plan, there are two initial ques-
level discussion of true experiments and the flaws associated with tions: the type of sample or sampling procedure used and the
various preexperimental and quasi-experimental designs, see sample size. Good science typically involves probability-based
Babbie 共2008兲 or Chambliss and Schutt 共2006兲兴. Four types of sampling 共an example of which is a “random” sample兲 to mini-
validity concern Cook and Campbell in particular. They are mize the chance of bias within the data; but probability based
briefly summarized in the Table 1. methods are rarely feasible in applied settings like ours and may
In both the social sciences and construction research, a clear not yield a reasonable response rate. This is why convenience
understanding of experimental design issues is critical 共Fellows samples 共and snowball sampling兲 are so common in construction
and Liu 2008兲. It is important not only to the quality 共validity兲 of research. Approaching individuals and/or firms that we feel are
any experimental data collected but for a deeper understanding likely to agree to participate enhances our response rates. Sub-
of basic research processes and problems. Careful study of ex- groups of interest in construction may include, for example, firms
perimental design texts, like Cook and Campbell 共1979兲 can working primarily in the commercial market versus industrial/
improve other research efforts—including surveys and ethno- process, infrastructure or residential markets, or firms operating
graphic observation—by drawing increased attention to the meth- primarily in one geographic region versus another region. Expe-
odological ABCs 共basic principles兲 of research design. Ultimately, rience has shown, however, that firms either will not agree to
understanding issues of internal and external validity in experi- participate in the study or will not provide the needed documents
ments, as well as measurement 共construct validity兲 and statistical unless they are truthfully told the research focus. Informed con-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 109

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
sent documents, necessary for human subjects research, must different meanings and measures to abstract social concepts like
therefore strike a careful balance—they need to contain enough these, they consist of “constructed” meanings, hence the use of
information about the research methods and risks for subjects to the term construct rather than concept by some researchers. In
make an informed choice about participating but do so without order to develop valid and reliable measures, whether as survey
prejudicing the data collected. questions, as indicators of experimental treatments, for behaviors
Research based on nonprobability sampling techniques, such observed in ethnographic study, or from archival sources, we
as that using convenience samples, can provide useful insights but therefore need clear and explicit conceptual definitions for each
it is limited with regard to the accuracy of estimates and its gen- construct 共variable兲 of interest 共Fellows and Liu 2008兲.
eralizability to larger populations 共Fellows and Liu 2008兲. With The conceptual level of analysis is among the key attributes
nonprobability based samples in particular, we cannot estimate to be specified. Are we looking at a micro or macrolevel
the size of the sampling error 共random error due to chance兲 nor characteristic—that is, does the trait or characteristic in question
determine what, if any, sample bias 共nonrandom or systematic pertain to an individual or to a group or organization as a whole?
errors兲 exists in the data. In probability-based sampling, the sta- Emotional intelligence, for example, is clearly an individual trait.
tistical power of the analysis increases as sample size increases Effective teamwork is a group construct just as organizational
共reducing sampling error and increasing confidence levels in a financial performance is a group construct 共in which the organi-
quantifiable way兲. By using convenience and other nonprobability zation constitutes the social group and the quality pertains to the
samples, however, the researcher gives up the ability to calculate group as the unit of analysis兲. Some constructs, however, such as
sampling error or meaningful confidence intervals. leadership, are often used to describe behaviors at more than one
The use of nonprobability samples appears to be common in level. For example, we say an individual exhibits good leadership
construction research based on the analysis of the February– skills over his or her subordinates, a team can exercise leadership
August issues of the JCEM. Nearly all of the 10 papers that pre- by suggesting improved procedures within a firm, and a firm can
sented findings based on surveys either acknowledged they used demonstrate leadership by introducing a series of new products in
convenience samples or did not address at all whether the sample a market. Although using the same term to describe micro and
was probability based. Given that none of the 18 papers that used macrobehaviors is common in construction research, this is prob-
one or more case studies to illustrate or empirically support the- lematic methodologically. In multilevel modeling, for example,
oretical frameworks or tools stated the source of the case studies, apparent causal relationships can be spurious effects of uncon-
it is reasonable to conclude that all of the case studies would be trolled and/or unmeasured multicollinearity between overlapping
considered convenience samples. micro and macrolevel measures.
To the extent that data derived from a convenience or snowball How we therefore define and specify each concept is critical
sample and data collected by an alternate means or method 共from for: first, constructing meaningful theoretical tests of our hypoth-
the application of a mixed method research design兲 converge on a eses; second, the design of valid and reliable 共adequate and accu-
common pattern or result, our ability to generalize from these rate兲 empirical measures in the operationalization stage; and third,
types of samples increases. Mixed method research designs can useful statistical analysis of the data. Furthermore, the negative
therefore help us overcome some of the inherent limitations of results of confounding the use and meaning of constructs, as dis-
any particular source/sample of data when 共or if兲 using methods cussed earlier, are compounded when applying a mixed method
with complementary strengths 共and weaknesses兲. approach, yielding results that not only may lack theoretical
meaning but which then cannot be empirically interpreted or com-
pared. Thus careful attention to conceptualization is critical—
Theoretical Concepts Must Be Explicitly Defined even more so when using multiple methods and measures.
before They Can Be Measured As previously noted, 22 of the 43 papers published in the
The meanings of most engineering terms are unambiguous. February–August 2008 issues of the JCEM included social con-
Stress, flow rate, and viscosity, for example, are well understood structs. Social constructs that were either the focus of the paper or
across the engineering community. Many construction manage- key terms underlying the background discussions in these papers
ment terms, on the other hand, like many concepts in the social included cultural and political impacts of a project, management
sciences, are somewhat imprecise, with meanings that can and commitment, employee training, employee and organizational
often do vary between researchers and research contexts. For ex- knowledge and reasoning, site safety, safety consciousness,
ample, leadership may be considered the ability to articulate an project management performance, employee dedication and atti-
organizational vision, to mobilize resources toward a set of goals, tude, employee work overload and stress, construction quality,
to motivate coworkers to perform at their best, to enact change, or personality attributes, collaboration and cooperation, and client
to establish systems that enable operational efficiency. Similarly, satisfaction. While the reader may feel that most construction
technological innovation may be thought of as creating new tech- researchers share a common understanding of these constructs on
nologies, adopting new technologies, transforming processes a broad level, the need for providing explicit definitions becomes
through a new type of system, or establishing a culture of em- more critical when one attempts to empirically investigate them,
bracing anything based on new technology. Effectively research- as discussed under the next principle.
ing many construction management research topics thus requires
researchers to define explicitly their theoretical concepts 共referred Theoretical Concepts Must Be Operationally Defined
to as “constructs” in the social sciences兲 at the beginning and to to Provide Valid and Reliable Measures
maintain a uniform definition throughout the research process.
In social science methods, the terms concept and construct, Most engineering terms have a standard method for measuring a
often used interchangeably, refer to the theoretical labels for traits value associated with them. Stress, strain, and viscosity, as ex-
or characteristics that exist analytically, at the abstract level, but amples, are measured using specific types of equipment and a
are not directly observable, such as job satisfaction, leadership, or range of procedures such that any engineer would agree the re-
innovation. To the extent that researchers may attach somewhat sulting values should be accurate. Many construction manage-

110 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
ment concepts are neither uniformly defined nor easily measured. sures developed by others and already reported in the literature
Consider the hypothesis that successful leaders ensure their firms gives us high levels of each. If such measures are not available,
are technologically innovative. How do we identify a leader in an then pretesting any new measures we create in a pilot study is
organization? Are all individuals holding certain titles 共president, essential. Finding out that a measure does not capture the concept
chief financial officer, water resources group manager兲 leaders? in question after the fact 共once the data are collected兲 leaves no
Do all leaders in an organization have one of these titles? If we room for adjustment or correction.
focus our study on individuals who are vice presidents or higher, In addition to concerns about measurement validity, careful
how should we measure the success of each leader? Should we operational definitions can reduce concerns about measurement
ask the individuals themselves if they are successful leaders? reliability. Reliability is based on the application of uniform mea-
Should we ask their bosses or subordinates? Should we seek ob- surement rules and the uniformity of measurement results over
jective archival metrics 共use existing organizational performance time. Having and applying operational definitions produces con-
data兲 that demonstrate their organizations, divisions or corporate sistent and stable results 共all else being equal兲. The same indicator
segments were recently successful, such as unit profits, growth in should produce the same results when other relevant factors re-
sales, productivity, or reported customer satisfaction? main unchanged. Problems of measurement reliability can arise
Operationalization, the process of choosing appropriate em- from inadvertent changes in the measuring instrument 共unplanned
pirical indicators, is critical to the success of a project. This pro- changes in question wording, for example兲, in the observer or
cess involves specifying the exact procedures to be used to mode of observation 共changing from personal observation to
measure the attributes or properties of each construct, defining video taping between subjects兲, or in the phenomenon itself 共as in
which specific observable indicators will be used to measure each spurious changes in reported answers due to fatigue on the part of
variable, and explaining how to interpret each indicator relative to subjects during a long interview兲. Social scientists generally agree
the theoretical property being measured. When complete, opera- that using archival indicators from existing sources of statistical
tional definitions provide other researchers with clear instructions data are more reliable 共consistent and stable兲 than self-reported
on how we measured each construct. Since there are multiple data because the latter are inherently subjective and may reflect
ways to operationalize social constructs in a survey questionnaire, changing individual biases and inaccuracies. Self-reported indica-
we need to specify the exact question or questions asked and tors may reflect individuals’ inflated opinions of their own abili-
make explicit how we are interpreting the answers in terms of the ties. Answers about other people or issues may be biased because
concept or construct in question. For example, to operationalize respondents fear that answering the question truthfully may lead
“client satisfaction,” we could include a global or overall question to trouble with their boss or coworkers or that they will disappoint
asking clients to rate their satisfaction with the construction firm’s the researchers. Archival data, too, is subject to biases and prob-
performance: “Overall, how would you rate your satisfaction with lems of reliability and validity. Often the specific measures we
the performance of this firm? Please choose a number from 1 to need do not match the data or indicators available and/or we do
10, where 1 = completely satisfied and 10= not at all satisfied.” In not have access to information on the possible measurement er-
this case, the lower the mean value among responses, the higher rors that pertain to the ones we do use. Changing organizational
the level of satisfaction. Alternatively, client satisfaction could be politics and practices, moreover, can affect the quality of such
operationalized by the percent of respondents who answered data particularly with regard to the comparability of measuring
“fairly” or “completely” satisfied in response to the question, “In change over time.
general, how satisfied have you been as a client with the work Unfortunately, identifying accessible and reliable archival in-
performed by this firm? Would you say you have been completely dicators is difficult for many constructs. In some cases, it is not a
satisfied, fairly satisfied, fairly dissatisfied, or completely dissat- lack of access to relevant corporate data but the nature of the
isfied?” As these examples show, providing the exact wording for constructs we are studying that provide the challenge. In fact,
survey questions 共or completely specifying the measures used some social scientists unapologetically identify some constructs
from archival data兲 and explaining how we interpret the answers as latent constructs, meaning they cannot be measured directly by
insures consistency on our parts as researchers and provides clar- one or more indicators 共Loehlin 1998兲. Structural equation mod-
ity to readers. Moreover, with careful operationalizations, we can eling is a relatively recent statistical tool that uses factor analysis
assess measurement validity and reliability and replication studies of multiple indicators to measure latent variables. Molenaar et al.
共another aspect of triangulation兲 become feasible. 共2000兲 and provide a good explanation of the use of structural
Measurement validity is the determination that an indicator equation modeling in construction research.
actually measures what it is supposed to measure. If an indicator Measurement validity and reliability are not our only concerns
is not valid, there will be systematic error or bias in the measure- in the operationalization process. As researchers, we also need to
ment process and the resulting data. There was a doctoral student, pay careful attention to indicators’ levels of measurement and the
for example, who spent many hundreds of hours collecting data degree to which they can be quantified. Interval and ratio level
from a large sample of construction firms for the student’s disser- data, for example, allow for the most complex mathematical ma-
tation. The hypotheses were important and interesting, pilot inter- nipulations. Since the level of measurement for each indicator
views seemed to indicate the hypotheses were accurate, and a depends on how it is operationalized, and the statistical tools that
reasonably large number of firms provided survey data. The re- can be used with each indicator depend on the level of measure-
search plan seemed sound other than concern voiced by the doc- ment, decisions made in this stage of research are critical for
toral committee about the quality of the indicators. It was the those that follow. More information is better than less, and if an
latter concern, however, that turned out to be critical after the fact. indicator is truly interval or ratio level, it should be collected in
The results were ultimately inconclusive statistically, apparently that form. If we are collecting data on employee seniority with the
because the indicators did not capture the underlying constructs. company, as measured in years of service, for example, it is better
Researchers need to review their indicators carefully in the plan- to ask a question that allows the respondent to specify how long
ning stages to ensure high levels of measurement validity. Where they have been with the company or firm in single years rather
possible and appropriate, using operational definitions and mea- than have them check off a box for a category that collapses the

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 111

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
data into larger groupings. If the critical threshold point for the sources to be innovative. In this case, rather than presume the
effect of seniority on perceptions of leadership falls in the middle direction of causality, the better approach is to collect data on
of a large preset category, we are likely to miss the significant both current and past innovative behavior and current and past
effect in the analysis. Smaller groupings can always be created financial performance. Although we are collecting all these mea-
during the analysis by temporarily recoding and collapsing a large sures simultaneously, using a quasi-longitudinal design, in which
amount of data into fewer categories when needed, but if the data data are collected on past and present values of all key variables,
are only collected in a less precise categorical form, we have no provides some evidence of the time order of change when direct
way to redress the problem after the fact. observation is not possible.
A mixed method approach, using two or more data collection The example earlier also illustrates the importance and diffi-
methods whose validity and reliability problems counterbalance culty of establishing nonspuriousness. To reject alternative expla-
each other, enables us to triangulate in on the “true” result. With- nations for any statistical relationship found between two
out careful operationalizations, however, the use of mixed meth- variables we need to be able to test and eliminate them. Because
ods 共or multiple measures within a single methodological we are rarely allowed to manipulate independent variables and/or
approach兲 only serves to create additional noise in the data rather physically control all extraneous factors when doing human sub-
than help illuminate the underlying patterns of interest. jects research, spurious relationships are a constant threat. For
Most of the 22 JCEM papers with social constructs that were example, if we were to measure the technological innovation lev-
analyzed did not identify the specific ways constructs were opera- els and financial performances of 100 construction firms and
tionalized. This may not be surprising given that manuscript found the two variables to be highly correlated, it may be that this
length limitations often preclude including survey instruments in association merely reflects the fact that both variables are related
the manuscript. The writers believe, however, that confirming the to the size of the firm, i.e., large firms are more innovative and
operationalization of key variables was appropriate is an impor- achieve higher profit margins, not that more innovative firms are
tant part of a peer-review process that is not possible given the more profitable.
typical manuscript peer-review process used by the JCEM and Nonspuriousness is the most difficult condition of causality to
other leading construction journals. establish. There is no way to actually prove that the relationship
between the independent and dependent variables is not spurious,
Hypothesized Causal Relationships Must Be Stated that is, that you can reject all other possible explanations. We can
Explicitly but Are Difficult to “Prove” use observation and logic to establish time order and observation
and statistical tools to establish and even quantify the degree of
Essentially all research should start with at least one meaningful statistical association or constant conjunction between variables.
research question, which typically leads to at least one hypothesis But how do we prove that there are no other likely explanations
investigated through empirical study. Most hypotheses are causal or hidden third factors? In fact, unless we can actually determine
in nature. Assuming there is a plausible theoretical explanation and test all possible alternative explanations, we can never truly
causally linking the two variables, three conditions must be met to prove the causal connection. We use a range of methodological
establish causality. First, we need to establish temporal prece- techniques and statistics to hold constant or control what poten-
dence, indicating that the hypothesized cause comes before the tially spurious factors we can 共that is why random assignment in
hypothesized effect. Second, we need constant conjunction, dem- experiments and random sampling for surveys are so important兲,
onstrating that changes in the independent variable are associated but the possibility always exists that there is some alternative
with changes in the dependent variable 共each time the value of the explanation that we did not consider 共a third factor we did not
hypothesized cause changes, there is a predictable change in the take into account兲. As a result, social scientists talk about con-
effect, which we often measure through the Pearson correlation firming or disconfirming, accepting or rejecting research hypoth-
coefficient or some other appropriate statistical measure of asso- eses, never “proving” them 共Judd et al. 1991兲.
ciation兲. Finally, we need to establish nonspuriousness, thus Some researchers acknowledge the difficulty of establishing
eliminating plausible alternative explanations for the statistical causality and as a consequence merely hypothesize that two vari-
relationships found 共Cook and Campbell 1979兲. ables are associated in some way. Such hypotheses are more tech-
Many construction researchers typically are able to establish a nically appropriate but make a rather limited contribution to the
plausible theoretical connection and demonstrate constant con- literature. Real contributions to our understanding come from at-
junction but do less well with temporal precedence and the prob- tempting to overcome both the methodological and theoretical
lem of spuriousness. We all know that correlation does not equal challenges that inevitably arise from compelling causal hypoth-
causality, yet many fail to acknowledge that regression only pre- eses. With careful attention to the conditions of causality and the
sumes causality; it does not prove it. The challenge to establishing use of appropriate controls and caveats, causal inferences can be
temporal precedence is that it is usually done by observation and drawn even if causality cannot be proven per se. Where appli-
could, in the strictest sense, require us to conduct controlled ex- cable, tests of statistical significance and confidence intervals can
periments involving individuals and companies, to observe what increase the strength of causal inferences. To that end, choosing
happens after we manipulate the value of the independent vari- appropriate statistical techniques in data analysis is critical. Be-
able. This is impractical in real-world settings. Instead, leadership yond this, mixed method studies can increase our confidence in
researchers typically measure both the independent and dependent causal inferences well beyond the results of one study. When
variables simultaneously and presume, by the use of logic, that different types of data, collected by different methods, converge
the former has influenced the latter. For example, we might mea- 共whether in support of the hypothesis or not兲, the result is likely
sure a firm’s innovation level and a firm’s financial performance not due to accidental correlations in a particular data sample,
at one point in time and presume that innovation has contributed however carefully collected.
to the firm’s financial performance. An astute colleague, however, As stated earlier, 18 of the 43 JCEM papers analyzed applied
might argue the causal relationship is reversed, that is, that a the theoretical framework or tool presented in the paper to one or
firm’s financial success provides them with the necessary re- more case studies. Unfortunately, perhaps due to manuscript

112 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
length limitations, the amounts of information about the case degrees of freedom兲 of the model is so low as to be essentially
studies were always so limited that the case studies could not meaningless. Meaningful research results need to explain suffi-
begin to serve as preliminary empirical validation of the theoret- cient variation for the significant relationships to matter. How
ical model. As such, the discussion of the case studies at most much explained variation is meaningful, however, varies across
provided support that the model could be applied to actual levels of analysis. With microlevel survey data in the social sci-
projects, but could not provide meaningful support that the model ences, for example, explaining more than 10% of the variance is
was accurate or reliable tests of the hypotheses as such. considered good and more than 20% is excellent. With mac-
rolevel data, such as organizational data commonly used in con-
struction research, the adjusted R2 tends to be much higher since
Appropriate Statistical Analysis Is Critical there is often some smoothing of the data when aggregate mea-
to Meaningful Results sures are constructed.
In construction research, linear regression is most often the sta- A growing statistical analysis trend in construction research is
tistical tool of choice. Two mistakes involving linear regression the use of factor analysis. A substantial portion of the papers
are common. The first mistake is to use regression even when the using factor analysis that the second writer has reviewed over the
data violate the fundamental assumptions underlying regression: past 5 years seemed to have used this technique inappropriately.
first, that dependent variables are continuous and approximate a Factor analysis is appropriate for identifying latent variables that
normal distribution; second, that independent variables approxi- underlie two or more of the specific variables measured. The ra-
mate a normal distribution or are categorical; and third, that the tional is that many social constructs 共such as leadership, innova-
relationships between dependent and independent variables are tion, and project management performance兲 cannot be adequately
linear in nature. Linear regression is a statistically robust proce- captured by only one measurement. One of several advanced sta-
dure, but experts suggest the need to perform a variety of tests tistical software packages can be used to identify the apparent
before and after performing multiple regression analysis to verify latent factors underlying the data based on relatively high load-
that the data or the model do not significantly violate the under- ings 共that is, statistical relationships兲 on the associated measured
lying assumptions 关see Belsley et al. 共1980兲 for examples兴. Con- variables. Unfortunately, it has been the second writer’s experi-
struction research data, especially those using Likert scales, are ence that some researchers have used the resulting factors without
often neither continuous nor normally distributed. When this is confirming that the factors identified by the software make intui-
the case or residual diagnostics indicate violations of regression tive sense. That is, regression analysis has been performed using
assumptions, researchers should consider supplementing their latent factors that are composed of measured variables that are
standard regression analysis with probabilistic regression, logisti- theoretically unrelated to one another.
cal regression or other appropriate statistical techniques. As anyone in the construction industry knows, you need the
The second mistake common among some construction re- right tool for the job. The statistical tools available for any given
searchers is to perform only bivariate analyses. Many relation- problem are the product of the various decisions made at each
ships between variables may be statistically significant but stage in the research process—from the type of sampling done, to
spurious, as discussed in the previous section. When two vari- the sample size, the operational definitions used, the levels of
ables are found to have a statistically significant association or measurement for each construct, to the causal conditions being
correlation 共i.e., the chance of a type I error being 5% or less兲, we tested. No methodological choice, even choosing the statistical
cannot be sure the relationship between them is nonspurious un- tools applicable to a given problem, is truly independent of
less we control for the confounding effects of antecedent vari- the others. Each has consequences; each has limitations and par-
ables. We can do this using partial correlation analyses 共assuming ticular sources of error 共problems of validity and reliability兲. The
we have interval or ratio level data兲 or some comparable nonpara- best way to address the inherent limitations of any singular meth-
metric statistical analyses 共if we have nominal and/or ordinal odological approach 共be it a survey, experiment, observation, or
level variables兲. As we noted earlier, the choice of multivariate archival research兲 or individual measure 共self-report, archival in-
statistical technique depends on the levels of measurement of the dicator, participant observation兲 is to apply more than one method
variables in the model 共which derive from the indicators chosen to the problem—to utilize a mixed methods research design.
in our operational definitions兲. In examining the relationship be-
tween firms’ levels of innovation and their financial performance,
we would want, for example, to look at the relationship between Mixed Method Research: No Single Method
firm innovation scores and financial performance while control- Is “Best”
ling for firm size and other relevant variables.
If the assumptions of the data hold, a relatively easy way to By using multiple methods to study the same problem, we can
eliminate plausible alternative explanations is to perform multi- detect recurrent patterns or consistent relationships among vari-
variate linear regression using a model that contains other poten- ables, results that are independent of one particular data source or
tially confounding variables 关for other appropriate multivariate type of measurement and its inherent weaknesses. Triangulation,
statistical techniques for professional research, see Vogt 共2007兲兴. simultaneously using multiple research methods or measures to
This way, we can estimate how much each independent variable test the same hypothesis or finding, is a valuable strategy in the
in the model contributes to each dependent variable, controlling research process, but more so when we mix methods that have
for all others. Moreover, we can compare the relative strength of different but complementary strengths and weaknesses 共Fellows
the relationship of each independent variable to the dependent and Liu 2008兲.
variable and estimate the contribution each makes to the overall In fact, triangulation is a term used in social science research
amount of variance explained. Some researchers focus solely on methods for decades. It was first used to describe convergent
the ␳ values to identify relationships that are statistically signifi- validation from multiple operationalism, where multiple tests of
cant, but ignore the fact that the adjusted R2 共the square of the the hypothesis are conducted using different operationalizations
Pearson product moment correlation coefficient adjusted for the or empirical indicators of the constructs in each test 共Campbell

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 113

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
and Fiske 1959兲. It then came to be used to refer to the applica- data. The results have limited generalizability and little control
tion of multiple methods of data collection 共mixed methods兲, the over the conditions of causality. Survey research often yields
use of multiple data-collection technologies per se, as well as the highly reliable measures and, if probability-based sampling is
use of multiple theories and multiple observers or researchers used, the results can be highly generalizable. But surveys are
共Denzin 1989兲. The use of multiple theories to generate and test reactive methods and are subject to error in self-reports 共deliber-
competing hypotheses and the use of multiple observers, while ate and not兲 while providing a limited ability 共via statistic con-
also forms of triangulation, are not applications of a mixed trols兲 to establish causality. Experimental designs provide the
method approach per se. As such, they are not discussed further strongest support for causal inferences. Lab experiments allow the
here 关see Brewer and Hunter 共1989兲 for detailed discussion of greatest “control” over subjects and the experimental conditions
these forms of triangulation兴. and therefore have the highest internal 共experimental兲 validity.
The simplest form of a mixed method approach is the triangu- Natural or field experiments, on the other hand, have lower inter-
lation of measures. This involves the use of multiple operational- nal validity but higher external validity 共the results are more
izations of the same construct in a study. It allows us to see more broadly generalizable to the real world application of the experi-
facets of the phenomenon in question. In surveys, this can involve mental treatments兲. Although unobtrusive measures are free from
the use of both closed-ended 共forced choice兲 and open-ended the problem of reactivity 共subjects are not altering their behavior
questions to measure a key construct 共behavior, trait, or attitude, in response to being studied兲, a problem that pertains in varying
for example兲. It can also involve the use of several distinctly degrees to different types of field work, surveys and experiments兲,
worded questions 共whether open or closed ended兲 for the same it is the least flexible method. Unobtrusive techniques are often
construct. To the extent that the results from different measures of less precise in the available measures and provide little if any
the same concept converge, our confidence in the results in- control over conditions of causality. As Brewer and Hunter noted:
creases. Different measures 共indicators兲 of a given construct are
“ . . . fieldwork’s realism is restricted by its small
not likely to reflect the same problems of measurement validity
scale . . . The ability of survey research to generalize is
since question wording and format affect how a subject interprets
limited by the ability to define theoretically relevant
its meaning as does the method of administration 共if the question
populations that can be readily sampled and questioned.
is asked by mail, by phone or face to face兲. If, however, the
The causal clarity of experimental research ends when
answers remain consistent 共data converge兲 no matter how many
there are confounding influences present that cannot be
different ways you ask the question, the result is reliable and you
eliminated by control and randomization. And nonreactive
are likely circling around something “valid” in the data.
studies’ freedom from reactive measurement effects can
The use of multiple methods of data collection, mixing quali-
never be taken for granted. Even in historical research, the
tative and quantitative approaches in a mixed method design, is a
possibility exists that the documents, statistics . . . have
broader form of triangulation—one that costs more but yields
been altered or edited in anticipation of scrutiny” 共p. 47兲.
greater utility. Mixing qualitative and quantitative methods allows
us to combine research styles whose strengths and weaknesses are The debate over the relative value of fundamentally different
counterbalanced. If the methods chosen only partially overlap in research approaches has persisted in the social sciences for many
style, a study using more than one method, applied either sequen- decades and has generally focused on several sets of related di-
tially or simultaneously, will provide richer, more comprehensive, chotomies: quantitative versus qualitative research, nomothetic
data 共Neuman 2000兲. versus ideographic explanations, and positivist versus phenom-
What social scientists call “fieldwork,” such as ethnography or enological approaches. To investigate a hypothesis involving
participant observation, allows us to study natural, often complex leadership in technological innovation, for example, should we
behavior in its natural setting, and to study a problem holistically use a survey to obtain a relatively large random sample of objec-
from the subjects’ point of view while observing change over tive data that can be subjected to rigorous statistical analysis, or
time. This approach, however, is confined to relatively small, lo- do a 3–6 month ethnographic study in which we shadow a few
calized groups, such as communities and organizations. We can- top managers in several firms? The survey approach, using a
not measure past behavior and must be very careful when we stratified sample, would allow us to report statistical relationships
generalize to large populations from such detailed case studies. with a high degree of confidence, yet we might omit key variables
Surveys, in contrast, allow us to collect a lot of information about in our survey or be unable to explain some of the relationships
large populations but only on topics that can be self-reported that appear in the data. The ethnographic approach, on the other
共whether verbally or in writing, face to face or via phone or com- hand, would provide us with the richly detailed observational data
puter兲. What surveys cannot do is allow us to study nonverbal and the opportunity to test highly nuanced explanations. Yet it
behavior or observe change over time directly. Experiments, opens us up to criticism that our writing merely reflects filtered
whether lab experiments or natural field experiments, provide the observations and subjective interpretations rather than “facts” and
strongest controls over the conditions of causality, but many if not that it is too limited in scope to be generalizable to the larger
most of the topics studied by social scientists, let alone those of population of firms or leaders.
interest to construction researchers, are generally not open to di- The mixed method approach is based on the premise that an
rect experimental manipulation either physically or ethically. Fi- effective body of research on a topic should include more than
nally, unobtrusive methods 共like the use of archival data兲 are one research approach. A typical practice in construction is to
considered nonreactive 共the responses are not affected by the distribute surveys to large samples via the postal service, e-mail
measurement process or by interaction with the researcher兲 and or a website, perform statistical analysis on the data collected, and
may be broadly applicable. But they are limited to the study of to supplement the survey with interviews before or after the sur-
those constructs and hypotheses for which data already exist vey. Another common method is to do a detailed case study of
and/or to those situations in which unobtrusive observation is one issue within a firm, or minicase studies with four to twelve
ethical. firms. While there have been a few “insider stories” published on
Field methods tend to yield highly valid but not highly reliable specific construction trades and major projects, these do not meet

114 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
the standards of ethnographic research. Social science research spreading costs among different parties and budgets, and
suggests that construction research would benefit greatly from throughout the research process” 共Brewer and Hunter
more ethnographic studies, especially in research areas still in the 1989, p. 96兲.
exploratory stage, which are then followed with larger systematic
Even with the additional costs, mixing methods with different
surveys. Although most construction researchers have neither the
strengths and weaknesses 共e.g., a survey with an experiment, a
interest nor capability to do long-term ethnographic studies, other survey with a field study, a survey with archival research, inter-
less time-consuming forms of observational or field methods views, and archival data兲 helps us in the process of causal infer-
could nonetheless lead to the development of better indicators, ence. If the data converge to support the hypotheses, we can have
survey instruments, and sampling designs. greater confidence that the results are not in fact spurious or mere
Aside from the use of field methods, a mixed method approach artifacts of one particular body of data. If the results of different
could incorporate unobtrusive research methods 共the use of exist- methods diverge, however, the lesson may be equally important
ing organizational or industry data, for example兲 to complement in that unexpected findings should push us as researchers to re-
self-reported survey data. At a minimum, the principle of triangu- examine our theoretical assumptions and improve our methods
lation implies that even with a single method of data collection, and measures. Whether a mixed method design results in data
multiple measures of key variables should be used. In this way, convergence or not, it is likely to provide more valid and reliable
the researcher is not reliant on single indicators that individually data and thereby allow us to have greater confidence in our
could yield invalid or unreliable information. Within a standard conclusions—whatever they might be.
survey questionnaire or interview schedule, including several dif-
ferent questions and versions of a question to measure the same
theoretical concept meets the spirit of a mixed or multimethod Conclusions
approach.
When doing mixed method research projects, however, the
This paper has reviewed several basic social science principles
researcher does face some new or additional challenges for cross- that construction researchers should follow, along with a mixed
validation of data to be possible. One problem that can arise in method approach to research design, in order to increase their
mixed method research is that the response rate varies markedly ability to draw credible and compelling conclusions from empiri-
among different approaches. Mail out/mail back surveys have cal research. The benefits of using a mixed method research de-
very low response rates, typically, in comparison to experiments sign are especially applicable to areas of construction research
or personal, face-to-face interviews. Furthermore, it is unlikely that align with the social sciences rather than with, for example,
that the subgroup of the population that does not respond to sur- operations research or information technology. Adhering to the
veys is the same as the subgroup who avoids field researchers, methodological ABCs helps researchers reduce threats to validity
who are likely omitted from official records, and/or fail to com- and reliability while mixed method studies simultaneously in-
plete an experiment. In fact, subjects typically missing from each crease the likelihood that the research will make a meaningful
methodological approach are likely to differ systematically. The contribution to the literature.
desire to counterbalance these particular sources of error and in- It should be clear to the reader that the issues raised in this
crease the overall generalizability of the study is one more reason paper regarding research methods are only the proverbial tip of
to utilize a mixed method design. the iceberg. This paper is intended to serve as an introduction to
Although the benefits of mixed method research designs in- issues in effective research methods in construction and to prin-
clude increased reliability and validity of the data and greater ciples of triangulation in social science methods in order to en-
confidence in tests of the hypotheses 共and the resulting conclu- courage researchers to more fully study the topic. Effective
sions兲, there are added costs for both the researchers and the research on topics involving human behavior in construction is
research subjects. Since the research costs, generally counted in hard work and requires expertise that is rarely provided in engi-
terms of time, money, and energy, are not identical for each ap- neering or construction management curricula. To understand the
proach, and the additional costs for additional methods of data full range of methodological challenges faced when doing re-
collection enter into the project at different stages for different search with and about human subjects, graduate students in con-
methods, there are tradeoffs between additional methods in terms struction should take at least one course on social science
of time and money of the researchers and the time and energy of research methods, preferably one that covers both qualitative and
subjects: quantitative approaches, and one course on multivariate statistical
analysis before initiating their research. Including social science
“ . . . while a field or nonreactive study plus a survey or research methods courses in a graduate curriculum in construction
an experiment is more expensive than either alone, the management is unusual and perhaps controversial. But somehow
costs involved are not necessarily borne simultaneously, integrating the knowledge derived from such courses is an impor-
nor are they doubled for all parties, nor do they entail tant step if the construction research community is to maintain a
double consumption of the same resources. Rather, field reputation for effective research.
and nonreactive studies most often chiefly consume re-
searchers’ time and energy in the search and analysis
phase, while surveys and experiments consume research- References
ers’ time and energy more heavily in the design and in-
strumentation phase, and require more money and more Babbie, E. 共2008兲. The basics of social research, Thomson Wadsworth,
of subjects’ time and energy in the recording of observa- Belmont, Calif.
tions . . . 关mixed method designs are兴 no cheaper in an Belsley, D. A., Kuh, E., and Welsch, R. E. 共1980兲. Regression diagnos-
absolute sense. However it does increase its feasibility by tics, Wiley, New York.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 115

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Brewer, J., and Hunter, A. 共1989兲. Multimethod research: A synthesis of Loosemore, M. 共1999兲. “International construction management research:
styles, Sage, Newbury Park, Calif. Cultural sensitivity in methodological design.” Constr. Manage.
Campbell, D. T., and Fiske, D. W. 共1959兲. “Convergent and discriminant Econom., 17共5兲, 553–561.
validation by the multitrait-multimethod matrix.” Psychol. Bull., Molenaar, K., Washington, S., and Diekmann, J. 共2000兲. “A structural
56共2兲, 81–105.
equation model of construction contract dispute potential.” J. Constr.
Campbell, D. T., and Stanley, J. 共1963兲. Experimental and quasi-
Eng. Manage., 126共4兲, 268–277.
experimental design for research, Rand McNally, Chicago.
Neuman, W. L. 共2000兲. Social research methods: Qualitative and quan-
Chambliss, D., and Schutt, R. K. 共2006兲. Making sense of the social
world: Methods of investigation, Pine Forge, Thousand Oaks, Calif. titative approaches, Allyn and Bacon, Needham Heights, Mass.
Cook, T. D., and Campbell, D. T. 共1979兲. Quasi-experimentation: Design Raftery, J., McGeorge, D., and Walter, M. 共1997兲. “Breaking up method-
& analysis issues for field settings, Houghton Mifflin, Boston. ological monopolies: A multi-paradigm approach to construction man-
Denzin, N. K. 共1989兲. The research act: A theoretical introduction to agement research.” Constr. Manage. Econom., 15共3兲, 291–297.
sociological methods, Prentice-Hall, Englewood Cliffs, N.J. Runeson, G. 共1997兲. “The role of theory in construction management
El-Diraby, T. E., and O’Connor, J. T. 共2004兲. “Lessons learned in design- research: Comment.” Constr. Manage. Econom., 15共3兲, 299–302.
ing research methodology in field-based construction research.” Seymour, D., Crook, D., and Rooke, J. 共1997兲. “The role of theory in
J. Profl. Issues Eng. Educ. Pract., 130共2兲, 109–114.
construction management: A call for debate.” Constr. Manage.
Fellows, R., and Liu, A. 共2008兲. Research methods for construction, 3rd
Econom., 15共1兲, 117–119.
Ed., Blackwell Science, Oxford, U.K.
Judd, C. M., Smith, E. R., and Kidder, L. H. 共1991兲. Research methods in Vogt, W. P. 共2007兲. Quantitative research methods for professionals, Pear-
social relations, 6th Ed., Holt, Rinehart, and Winston, New York. son Education, Boston.
Loehlin, J. C. 共1998兲. Latent variable models: An introduction to factor, Walker, D. H. T. 共1997兲. “Choosing an appropriate research methodol-
path, and structural analysis, Lawrence Erlbaum Associates, London. ogy.” Constr. Manage. Econom., 15共2兲, 149–159.

116 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Contextualist Research: Iterating between Methods While
Following an Empirically Grounded Approach
Stuart D. Green1; Chung-Chin Kao2; and Graeme D. Larsen3

Abstract: An alternative approach to research is described that has been developed through a succession of significant construction
management research projects. The approach follows the principles of iterative grounded theory, whereby researchers iterate between
alternative theoretical frameworks and emergent empirical data. Of particular importance is an orientation toward mixing methods,
thereby overcoming the existing tendency to dichotomize quantitative and qualitative approaches. The approach is positioned against the
existing contested literature on grounded theory, and the possibility of engaging with empirical data in a “theory free” manner is
discounted. Emphasis instead is given to the way in which researchers must be theoretically sensitive as a result of being steeped in
relevant literatures. Knowledge of existing literatures therefore shapes the initial research design; but emergent empirical findings cause
fresh theoretical perspectives to be mobilized. The advocated approach is further aligned with notions of knowledge coproduction and the
underlying principles of contextualist research. It is this unique combination of ideas which characterizes the paper’s contribution to the
research methodology literature within the field of construction management. Examples are provided and consideration is given to the
extent to which the emergent findings are generalizable beyond the specific context from which they are derived.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000027
CE Database subject headings: Research; Theories; Methodology; Construction management.
Author keywords: Contextualist research; Coproduction; Ground theory; Interpretive research; Research methodology.

Introduction utes to short-term problem-solving with the need for academic

research which contributes to generic understanding. It is con-
The purpose of this paper is to introduce the idea of contextualist tended that CM research too often gives scant attention to the
research into the domain of construction management 共CM兲 and recursive relationship between practice and context or to the im-
to relate it to emerging debates regarding the coproduction of portance of studying reality in flight 共cf. Pettigrew 2003兲. In light
knowledge. It is contended that these two concepts combine to of these criticisms an alternative methodology is advocated which
present a very different orientation toward research from that goes someway toward overcoming existing limitations. The advo-
which currently dominates within the international CM research cated approach is labeled iterative grounded theory and draws
community. The idea of coproduction is attracting increasing at- especially from the contribution of Orton 共1997兲. However, cau-
tention within the context of generic management research 共cf. tion is expressed regarding those who claim to enact grounded
theory solely as an inductive research process. The advocated
MacLean and MacIntosh 2002; Perkmann and Walsh 2007兲, but
approach is seen to align well with aspirations for knowledge
has not yet been articulated as a research approach within the
coproduction, and the research process is equated with a collec-
context of construction management. Initially, the principles of
tive process of sensemaking 共cf. Weick 1995兲. The paper is con-
coproduction research are reviewed together with the challenges
cluded by two examples of research which approximated toward
of meeting the needs of two different audiences. Reference is
the approach described. Particular attention is given to the meth-
made to the Mode 2 theory of knowledge production 共Gibbons et
ods applied and the extent to which knowledge coproduction was
al. 1994兲 which emphasizes the need for context-driven research.
achieved.
The aspiration is to balance the need for knowledge that contrib-

1
Professor, School of Construction Management and Engineering,
Univ. of Reading, Whiteknights, Reading, Berkshire RG6 6AW, U.K.
Coproduction Research
共corresponding author兲. E-mail: [email protected]
2
Research Fellow, School of Construction Management and Engineer- The issue of industry impact has always been of central impor-
ing, Univ. of Reading, Whiteknights, Reading, Berkshire RG6 6AW, U.K. tance within the context of CM research. Unfortunately, the de-
E-mail: [email protected] bate is too often characterized by a dichotomization of the roles
3
Lecturer, School of Construction Management and Engineering, played by academics and practitioners. Academics are assumed to
Univ. of Reading, Whiteknights, Reading, Berkshire RG6 6AW, U.K. be the generators of knowledge and practitioners are assumed to
E-mail: [email protected]
be grateful recipients. Of particular note is the way in which the
Note. This manuscript was submitted on September 1, 2008; approved
on January 21, 2009; published online on March 27, 2009. Discussion
ascribed role of practitioners tends to be limited to the application
period open until June 1, 2010; separate discussions must be submitted of the resultant knowledge in practice. The process of applying
for individual papers. This paper is part of the Journal of Construction “new knowledge” frequently attracts the label ‘innovation’. Not-
Engineering and Management, Vol. 136, No. 1, January 1, 2010. withstanding the longevity of such ideas, the underpinning linear
©ASCE, ISSN 0733-9364/2010/1-117–126/$25.00. model of research-innovation-impact has become discredited in

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 117

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
favor of other models which emphasize the nonlinear, iterative themselves and to be crucial constituent parts of any orientation
and multiagent character of the innovative process 共Perkmann and toward innovation. The challenge for academic researchers is
Walsh 2007兲. In consequence, attention has increasingly focused therefore less about the provision of instrumental outputs and
on “coproduction” research whereby practitioners and academics more about designing a research process which provides practi-
cooperate to develop new knowledge and technologies together. tioners with a meaningful and an appropriate means of engage-
More specifically, the term is used to signify a dynamic interac- ment. It should also be emphasized that this engagement should
tion between researchers and practitioners which unfolds over include the setting of the research objectives and the associated
time. The emphasis lies on the industrial benefits derived through scoping of the research. But it should also be stressed that these
participating in the research process. One of the central chal- are ongoing activities that evolve as the research progresses. This
lenges of coproduction research relates to the necessity to meet provides a deep-rooted challenge to established ideas about “fix-
the needs of two difference audiences: researchers and practitio- ing objectives in advance” and achieving “complete clarity before
ners. entering the field.”

Mode 2 Knowledge Production Continuity of Engagement

The concept of coproduction research builds on the Mode 2 In many respects, the CM research community has always been
theory of knowledge production, 共or rather its coproduction兲 put oriented toward coproduction research. We are not therefore
forth by Gibbons et al. 共1994兲. In essence, Gibbons et al. con- claiming that the concept is especially original, but what is new is
tended that a new form of knowledge production started to sur- that researchers have only recently started to focus attention on
face in the mid 20th century which they characterize as context the methodological challenges of enacting coproduction research.
driven, problem focused, and interdisciplinary. Mode 2 is seen to A further point of interest here is that the CM research community
involve multidisciplinary teams being brought together for short is increasingly cited as being an exemplar of coproduction re-
periods of time to work on specific problems in real world search. Indeed, embedded learned behaviors relating to coproduc-
contexts. This is distinguished from so-called traditional research tion research and its enactment within the CM research
共i.e., “Mode 1”兲, which Gibbons et al. saw as academic, investi- community are attracting increasing attention from mainstream
gator-initiated, and discipline-based knowledge production. The management researchers. Strong links with industry and a deeply
challenges of coproduction research should not be underesti- embedded understanding of the day-to-day challenges faced by
mated. The requirement to serve the needs of two different audi- the construction sector are essential prerequisites.
ences is especially challenging. Traditionalists of course will tend Of particular importance to coproduction research is an ongo-
to dismiss coproduction research as “mere consultancy.” And ing commitment to the feedback of emerging insights to industry
such criticisms are likely to be justified if the research only pro- partners on a variety of levels. Each request for an interview, or
duces outputs which are of immediate instrumental use to the access to data, should be combined with a commitment to feeding
participants. But the challenge is also to produce outputs which back the resultant findings and interpretations. In our experience,
contribute to the conceptual understanding of those outwith the social scientists and colleagues from within business schools are
immediate context from which the research was derived. Cer- frequently amazed by the depth and longevity of the relationships
tainly, we are not at all embarrassed if industry partners are able that CM researchers enjoy with industry. A useful metaphor to
to derive short-term benefit from our research. Indeed, we would describe the more established modus operandi within these other
see this as a positive benefit. But overall, we would emphasize the research communities is that of “smash-and-grab” research. Once
importance of acknowledging the need to deliver to two audi- they have the data, industry partners never hear from the research-
ences from the outset. Furthermore, we would concur with the ers again as they concentrate all their energies on the production
observation of MacLean and MacIntosh 共2002兲 that such projects of the journal publications upon which their careers depend. Such
are likely to be more rewarding than those which ignore the needs an approach is the antithesis of coproduction research.
of one audience in order to better satisfy the needs of the other.

Contextualist Research
Engaging Practitioners in the Research Process
In seeking to strike a balance between knowledge for instrumental The dominant paradigm of CM research has been frequently criti-
use and knowledge for conceptual use there is clearly a time cized for its tendency to extract issues outwith the context which
dimension involved 共Van Aken 2004兲. Collaborative research makes them meaningful 共Bresnen and Marshall 2001; Fernie et al.
projects frequently last several years, and the final outputs often 2006兲. Within organization studies there is a long tradition of
only emerge several years after the projects have finished. How- contextualist research which has received little recognition within
ever, there is an obvious tension in that the instrumental needs of the domain of CM. Of particular importance is to recognize that
practitioners are more pressing and they cannot be expected to contexts are invariably dynamic, and as such should be recog-
wait until the academic arguments are fully developed. However, nized as an active part of any analysis. Pettigrew 共2003兲 empha-
to focus too much on the outputs from research is to misunder- sized the importance of studying reality in flight and of locating
stand that the prime benefit of practitioners taking part in copro- present behavior in the context of its historical antecedents. Few
duction research lies in their participation in the research process. current CM researchers give significant attention to the flow of
They are more than capable of deriving their own knowledge for events over time, with the result that much of their work reduces
instrumental use, and whether or not such knowledge contributes to an “exercise in comparative statics.” Exemplars of this ap-
to a broader conceptual understanding is not their prime concern. proach within ASCE Journal of Construction Engineering and
Participation in the research process is seen to provide exposure Management include Thomas et al. 共1992兲, Cox et al. 共2003兲, and
to new ways of thinking and a basis for challenging ingrained Walsh et al. 共2005兲. In contrast, we would recommend that re-
practices. These are frequently seen to be sufficient “outputs” in searchers follow the approach of historians to “reconstruct past

118 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
contexts, processes, and decisions” in order to discover patterns, interpretive research, although the advocated approaches are char-
find underlying mechanisms and triggers, and combine inductive acterized by as many points of difference as they are similarities.
search with deductive reason 共Pettigrew 2003兲. One point of commonality they do share is the emphasis given to
Such an approach directly questions the relevance of narrowly achieving a contextualized understanding of the selected research
construed positivist research methodologies and points toward the setting. Elsewhere, Rooke 共1997兲 made the case for an ethno-
need to understand the way in which complex processes are en- graphic methodology whereby researchers immerse themselves in
acted and legitimized. The important point is that the context the research setting. However, it is notable that the role allocated
within which managerial decisions are made must be conceptual- to practitioners is once again predominantly passive. Knowledge
ized as an active part of the analysis. But context is not only is attributed to actors within the setting, but the methodological
shaping, it is also shaped by action 共Pettigrew 1997兲. Issues of task of the researchers is to access this knowledge while avoiding
consideration include unpacking how contexts have been shaped the imposition of meaning. Certainly there is little scope for the
over time, the relationship between language and action, and the active engagement of researchers and practitioners in ways which
way that human agency relates to the construction sector’s insti- would lend themselves toward the coproduction of new knowl-
tutional structures. Such a research approach would require a sig- edge.
nificant shift in theoretical orientation, together with a multitude A further research approach increasingly recognized within the
of research skills beyond those normally mobilized in the cause of CM community is that of grounded theory 共Glaser and Strauss
CM research. Of key importance would be the need to shift from 1967; Strauss and Corbin 1994兲. Grounded theory is widely re-
a “being ontology” toward a “becoming ontology” 共cf., Chia garded as a general methodology to build theory, which emerges
1995兲. From this perspective, it is argued that it is not possible systematically from continuous interplay between data collection
ever to represent or describe reality using static terminology. Re- and analysis. The broad tradition of grounded theory is readily
ality is essentially viewed as emergent, evanescent, and fleeting relevant to coproduction research, although the basic underpin-
共ibid.兲; reality never stops in order to actually be, but is in a ning ideas should be approached with a significant degree of cau-
continuous state of becoming. Such an ontological perspective tion 共cf., Suddaby 2006兲. In contrast to the way in which
thereby emphasizes embedded processes of flux and transforma- grounded theory is often presented within the CM literature 共e.g.,
tion, rather than seeking to isolate static characteristics that can be Dainty et al. 2000b; Fox and Skitmore 2007兲, we would discount
supposedly measured and subject to abstract statistical analysis. the possibility of engaging with empirical data in an entirely
However, we would warn against current tendencies to dichoto- theory free, or an inductive manner. Indeed, we would point to-
mize between different types of research 共cf., Runeson 1997; Sey- ward the need for researchers to be theoretically sensitive as a
mour et al. 1997兲. In our interpretation, contextualist research result of being steeped in relevant literatures. We would further
could easily include a combination of both qualitative and quan- emphasize the importance of an iterative research process
titative research methods. Prior to describing the advocated com- whereby emergent findings are interrogated against a succession
bination of methods in detail, it is appropriate to address the of theoretical models derived from the literature. In other words,
modes of engagement between researchers and practitioners that new knowledge is developed from a continuous interplay between
characterize different research methods. Coproduction implies an existing theories and empirical data. In this sense we are sympa-
unfolding and iterative engagement with empirical contexts which thetic to the idea of iterative grounded theory as advocated by
resonates with research approaches rooted in the established tra- Orton 共1997兲.
dition of naturalistic inquiry 共Lincoln and Guba 1985兲. The widespread misrepresentation of grounded theory within
the CM literature, coupled with its broader contestation, causes us
to be cautious about pledging too strong allegiance to “grounded
Iterative Grounded Theory theory” per se. Indeed, the approach described here sits equally
well within the generic category of interpretivism which, in es-
The majority of CM research tends to be conducted within the sence, seeks to achieve contextualized meaning 共Greene 1994兲.
hypothetico-deductive paradigm whereby theories and resultant Grounded theory is also too often seen to be synonymous with the
hypothesizes are tested through empirical research. One of the use qualitative analysis software packages such as NVivo 共cf.,
main practical advantages of this approach is that there is an Dainty et al. 2000a兲. In such cases the emphasis tends to rely on
initial clarity about what is to be investigated 共Easterby-Smith et the analysis of interview transcripts through detailed coding con-
al. 1991兲. This aids the efficient collection of information, and ventions. In our experience such software packages can indeed
also enables others to replicate the study 共ibid.兲. However, this help in managing large amounts of qualitative data; they are
mode of research usually offers limited engagement between re- therefore undoubtedly useful support tools. But we remain cau-
searchers and research subjects. Such engagement as does take tious of those who claim to enact an analysis process which is in
place usually occurs from a distance through the medium of ques- some way replicable. In our experience there is no substitute for
tionnaire surveys. Even when focus groups are subsequently con- multiple readings of interview transcripts together with an itera-
vened to discuss the research outcomes, the engagement still falls tive process of searching for resonance with conceptual ideas de-
a long way short of the prolonged and recursive interaction en- rived from ongoing literature searches. Both possibilities are
visaged in coproduction research. Research which rests solely on alluded to by Strauss and Corbin 共1994兲 共p. 273兲:
questionnaire surveys implemented-at-distance also risk extract-
“… theory may be generated initially from the data, or, if
ing research findings outwith the context which makes them
existing 共grounded兲 theories seem appropriate to the area
meaningful. While such approaches can have a role within copro-
of investigation, then these may be elaborated and modi-
duction research they tend to position practitioners as research
fied as incoming data are meticulously played against
subjects rather than active participants.
them.”
Recent years have seen various criticisms of the dominant re-
search approach within CM. Seymour and Rooke 共1995兲 and It is the second possibility alluded to above which accords
Bresnen and Marshall 共2001兲 have both championed the cause of strongest with our own approach; we remain suspicious of those

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 119

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
who claim that theories “emerge” from the data. In our enactment
of coproduction research we would consciously mobilize different
middle-range theories for the purposes of sensemaking 共cf., Der-
vin 1992; Green and Schweber 2008; Weick 1995兲. Sensemaking
in this context describes an ongoing process of creating situ-
ational awareness and understanding in situations characterized
by ambiguity and complexity. In many cases, this involves pro-
gressively interrogating the data through a succession of theoret-
ical lenses derived from the literature. This process would
continue for as long as the resultant insights were sufficient to
justify the resources expended. What sets coproduction research
apart is the extent to which academics and practitioners actively
collaborate in this ongoing process of sensemaking. We would
also point toward the way in which academics and practitioners
have equally important roles to play. Academic researchers are
very often able to mobilize different theoretical lenses, each of
which will accentuate different aspects of reality. The participat-
ing practitioners will in turn comment on the validity 共and use-
fulness兲 of the various insights on offer. They are also likely to
continue to provide empirical insights stimulated by the process
of engagement. But crucially, coproduction research works best Fig. 1. Iterative grounded theory approach
when such ascribed roles are interchangeable. The process of en-
gagement characterizes each stage of the research, right from the Semistructured Interviews
initial scoping through to the final dissemination. In many re-
spects, there is no harsher test for the validity of research findings Semistructured interviews are of central importance to the
than to subject them to the critical scrutiny of practitioners em- adopted methodology. It is useful to maintain flexibility to re-
bedded in the context within which the research was conducted. spond to interesting diversions, but it is crucial to have a robust
Methods of data collection typically include interviews and interview schedule from the outset, otherwise the resultant data
field observations, coupled with archival analysis of in-house risks being too unstructured to be meaningful. The interview
documents. We would also advocate the use of quantitative analy- schedule must clearly be carefully designed prior to engaging in
sis techniques where appropriate and would contest the common fieldwork. Furthermore, it needs to be explicitly informed by rel-
assumption that interpretive research is solely dependent upon evant literatures. Nevertheless, it is important to recognize that
qualitative methods. In the context of coproduction, we would the interviews are essentially exploratory; their prime purpose is
place a particular emphasis on focus groups and other means of to ascertain the extent to which the concepts derived from the
feeding back interim interpretations to the participating practitio- literature are meaningful to the participants in the selected re-
ners. Of further importance is the whole range of informal inter- search setting. Self-evidently, the interviewees need to be selected
actions which characterize the research process: telephone on the basis of having a relevant expertise about the topic under
conversations, informal chats, e-mail exchanges can be important investigation. In the context of coproduction, the initial focus of
not only as means of data collection, but also as means of testing inquiry is invariably negotiated with the participating company
interim interpretations. Participant observation can be another im- representatives who will also have a key role in providing access
portant method of understanding how practitioners’ behavior is to designated interviewees. In some cases, the sample of inter-
frequently situated within social and physical contexts. viewees can be usefully extended through the use of the snow-
balling technique, which essentially involves asking interviewees
who else the researchers could usefully talk to. This might apply
to others within the interviewee’s company, or to external con-
Beyond Simplistic Dichotomies: Mixing Methods tacts within other organizations. Of overriding importance is to
in Research recognize the pluralistic nature of organizational contexts, it is
therefore important to ensure that multiple interviews are
The preceding section is sufficient to provide the broad back- achieved from within each research setting.
ground to iterative grounded theory and to position it against
other research approaches. The purpose of this section is to offer
more details on the specific techniques which characterize the Informal Interactions
advocated approach. The research approach described has been The advocated research approach also values the full range of
developed over the course of several research projects completed informal interactions with industry participants. These are seen as
over the last ten years. What follows is a portrayal of six research valid and important means of enhancing data collection and
techniques which are central to the advocated approach: 共1兲 semi- thereby providing an improved understanding of the research con-
structured interviews; 共2兲 informal interactions; 共3兲 archival re- text. Examples include: e-mail exchanges, telephone conversa-
search; 共4兲 evolving literature reviews; 共5兲 feedback sessions; and tions, and informal meetings. Social interaction between
共6兲 industry workshops. Fig. 1 depicts the continuous iteration researchers and industry participants is especially important to
between the techniques employed. Of particular importance is the coproduction research which essentially rests on relationships be-
iterative process between theory and data, with an explicit recog- tween the research parties. Informal interactions offer a range of
nition of the role of practitioners in the coproduction of knowl- opportunities to access current thinking, or even to ask specific
edge. questions. Participant feedback on emergent interpretations can

120 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
also be achieved through informal interactions. Such interactions ment that they found the process to be useful in helping them to
are often crucial to fostering ongoing commitment to an unfolding reflect on pertinent issues. Some interviewees have even ex-
research process. Informal methods can also include participant pressed the view that they found the experience “cathartic.” Such
observation in a range of different empirical settings, including reactions are of more than marginal importance to coproduction
meetings, site visits, and promotional events to name but a few. and offer a sharp contrast to practitioners’ experiences in complet-
ing web-administrated questionnaires.
Archival Research
Industry Workshops
Archival research describes the process of collecting and analyz-
ing published and unpublished archival sources of direct rel- Confidence in the wider generalizability of the derived findings
evance to the research topic. For example, if research is being can be further enhanced through a series of industry workshops
conducted within a particular company, it is imperative to ascer- conducted to obtain feedback from practitioners drawn from a
tain as much relevant knowledge prior to engaging in fieldwork. variety of construction sector contexts. These differ from the
This entails analyzing all available sources such as: published aforementioned in that participation is not specifically limited to
company account reports, corporate publicity materials, unpub- those practitioners who were interviewed. Indeed, there are clear
lished in-house reports, and public domain press articles. Such advantages in extending participation beyond the interviewees
sources can provide a wealth of background information which is and others who had been previously involved; industry work-
often underused. In addition, archival sources can lead to ques- shops not only work as a validation process, but also enable the
tions being asked that might otherwise not have been raised; they research to embrace additional points of view concerning the is-
can also be used to legitimize the insights offered through the sues being discussed. Active participation in industry workshops
interview process. enables practitioners to be part of the research process, thereby
Archival research can also be used to provide an initial con- contributing toward the coproduction of knowledge.
textual understanding of the research setting. This is especially
useful when intertemporal data sets are available, thereby en-
abling emergent findings to be understood in the context of a Research Case Studies
time-series analysis of relevant exogenous factors. Important
sources of data in this respect include publicly available construc-
The case studies presented here will be restricted to two projects;
tion sector statistics and government policy reports.
the results of which have already been previously published
共Green and May 2005; Green et al. 2008b兲. However, the empha-
Evolving Literature Review sis here will not be on the research findings per se, but on the
adopted methodological perspective and supporting techniques.
One of the defining characteristics of iterative grounded theory is
an acceptance that the literature review is not limited to the be-
ginning of the project, but continues throughout the research pe- Case Study 1: Understanding Lean Construction
riod. New directions of literature search are often dictated by
emergent empirical findings, frequently taking the researchers in Background
unforeseen directions. An important caveat is that the researchers The project was predicated on a sense of unease regarding the
should not venture too far beyond their domain of expertise; but a way in which lean construction had been absorbed into the lexi-
willingness to absorb new ideas is an essential prerequisite to con of best practice following its endorsement by the Egan report
good research. A continuously evolving literature review facili- 共Egan 1998兲. Of particular concern was a failure among policy
tates the ongoing interplay between data and theory which is of makers to acknowledge the critical literature which equates lean
central importance to iterative grounded theory. Only through this production with regressive human resource management 共HRM兲
process can existing theories be mobilized to make sense of emer- practices 共cf., Dohse et al. 1985; Garrahan and Stewart 1992兲. A
gent empirical insights. Grounded theory conducted in absence of total of 14 firms were mobilized as research partners with a vari-
a commitment to an evolving literature review risks offending the ety of interests in lean construction. Some saw themselves as
key criterion of theoretical sensitivity. exemplars, while others were hoping to learn from the leaders in
the field. Others still were rather more interested in HRM prac-
tices in the construction sector, and saw the Egan agenda as the
Feedback Sessions with Interviewees
opportunity to improve their approach to people management in
A further cornerstone of the advocated approach is the practice of line with what they saw to be more enlightened practice in other
offering interviewees feedback sessions where they can share in sectors. The original research design sought to compare the HRM
and contribute to the researchers’ interpretation of the findings. practices within firms which had adopted lean construction to the
Such interim feedback is often crucial in strengthening the valid- HRM practices within those firms which continued to adhere to
ity of the research findings, and also important in ensuring that traditional practices. However, as the research unfolded it rapidly
key factors are not neglected. Often it is the feedback provided by became apparent that to distinguish firms who had adopted lean
interviewees that points toward literature sets previously not con- construction from those who had not was by no means straight-
sidered. This process of iterating between the data and different forward.
theoretical frameworks is central to the advocated methodology.
In the spirit of coproduction, the intention is that the interviewees Research Methods
should themselves derive benefit from the interview process and The initial stage of the research comprised a series of interviews
feedback sessions are seen to be important mechanisms for with U.K. policy makers who were advocating lean construction
achieving this end. Confidence in the coproduction process can as a means of industry improvement. The sample of interviewees
further be derived from those occasions when interviewees com- also included senior representatives of construction firms and cli-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 121

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
ent organizations who were widely recognized as exemplars of in which management fashions are characterized by a significant
lean construction 关see Green and May 共2005兲兴. The interview degree of “interpretive flexibility” 共Benders and Van Bijsterveld
questions were informed by the mainstream literatures on 共1兲 lean 2000兲. In other words, they are subject to multiple interpretations.
construction 共e.g., Ballard and Howell 1998兲 and 共2兲 HRM prac- This vagueness of definition aids diffusion by enabling managers
tices in construction and other comparative sectors 共e.g., Cully et to mobilize different storylines in different contexts. In this way, a
al. 1999兲. The interviews were semistructured and oriented to- variety of different approaches are able to derive legitimacy from
ward accessing the interviewees’ interpretations of “lean” with the institutionally sanctioned sources such as the Egan report. The
minimum amount of prompts from the interviewer. Of the 25 reinterpretation of the empirical findings from this perspective
interviews conducted, approximately one-half were recorded and provided a very different set of research findings from those en-
fully transcribed. During the course of these interviews it rapidly visaged. Rather than seek cause-and-effect relationships between
became apparent that there was a lack of consensus on the defin- lean construction and HRM, they were recognized to be constitute
ing characteristic of “lean construction.” Very few had read the parts of the same improvement discourse. The persuasiveness of
key texts on lean production 共e.g., Womack et al. 1990; Womack this alternative interpretation was in no small way coproduced
and Jones 1996兲 and even fewer were familiar with, or even with the industry partners through a process of gradual refine-
aware of, the work of the Lean Construction Institute or the In- ment. The ultimate validity of the derived interpretation was es-
ternational Group for Lean Construction. Bresnen and Marshall tablished through a wide process of dissemination; of crucial
共2001兲 suggested that numerous management improvement reci- importance was the extent to which results were judged to be
pes defy universal definition, and this was certainly found to be credible by those operating within similar contexts.
the case with lean construction.
Textual analysis of the interviews revealed several different Contextualization
interpretations of lean construction; several of which were fre- The researchers’ orientation toward contextualist research was
quently coexistent. One interpretation focused on the elimination also of primary importance in shaping the research outcomes.
of waste over-and-above all other considerations, thereby echoing From the outset, the researchers were much more sensitive to the
previous improvement storylines such as business process reengi- processes through which lean construction were enacted and le-
neering. A second interpretation emphasized the importance of gitimized. As such, we were sensitized to the idea that lean con-
improving relationships between contracting parties; the storyline struction could be assimilated differently in different contexts. We
here was almost inseparable from the notion of partnering. A third were also very conscious that lean construction was not being
interpretation of lean construction focused on the relationship be- diffused into an industry which was standing still. We sought to
tween design and construction with significant emphasis on so- understand the way in which institutionally embedded practices
called modern methods of construction such as standardization, shaped different outcomes. But more important, we sought to un-
prefabrication and the use of digital technologies. derstand the diffusion of lean construction in a way that took into
The second stage of the research methodology sought to ac- account the existing dynamics of change. This was especially
cess a collection of case studies of the implementation of lean important in respect of the industry’s HRM practices. Certainly, it
construction in practice. However, here again the proposed ap- readily became apparent that there are several conditions that
proach floundered on the difficulties of definition. Of particular must be met if an improvement recipe is to be accepted in the
note was the way in which any claimed implementation of lean construction industry. One of these conditions is that it should not
invariably transcended organizational boundaries. While some ac- present too great a challenge to the existing embedded model of
tors within firms claimed to be exemplars of lean construction, people management. In this respect, the Kinnie et al. 共1996兲 three-
others 共sometimes even within the same firm兲 were openly dis- stage model of leanness was found to be especially persuasive as
missive of the label lean construction. In some cases, managers a basis for pulling together the empirical findings. Kinnie et al.’s
paid lip service to the lexicon of lean construction while continu- model initially focuses on the transitional stage whereby firms
ing to operate much the same as they ever did. Others had cherry- strive to become “leaner” through restructuring, downsizing, and
picked particular techniques which they associated with lean changes in the contractual status of employees. This backcloth
construction and combined them with established performance resonated strongly with the recollections of numerous inter-
techniques. Others had been inspired by the ideas of lean con- viewees of where their “lean journey” had begun. In no small
struction to innovate in a broad variety of different ways. Whether way, the onset of leanness in the U.K. construction sector can be
or not they called the resultant way of working lean construction traced back to the mid-1970s, which saw the extensive outsourc-
was found to be much more dependent upon the interorganiza- ing of labor and the emergence of the hollowed-out firm. Self-
tional networks they frequented rather than the substance of their employment grew from 30% in 1977 to a peak of 60% in 1995
material practice. Many practitioners at the time were deliberately 关International Labour Office 共ILO兲 2001兴. Many contracting firms
aligning themselves with the Egan report as a means of legitimiz- thereby embarked upon Kinnie et al.’s journey to leanness long
ing their actions in the eyes of clients and colleagues alike. before the popularization of Toyota’s manufacturing system.

Emergent Findings Summary

The emergent findings were debated at length with the industry This interplay between the emergent empirical data and fresh per-
partners and there was a collective recognition of the need to spectives derived from the literature produced a contextualized
mobilize alternative literature to make sense of the underlying account of lean construction which provided a fresh contribution
processes. Certainly there were extensive complexities relating to to existing knowledge. However, for the purposes of the present
the diffusion of “new” management ideas into practice which did paper the key point of interest is the research process through
not even begin to be explained by the mainstream literature on which this revised understanding emerged. Such a radical reinter-
lean construction. Of particular relevance was the extensive lit- pretation of lean construction could not have been developed
erature which focuses on how management fashions are generated through a questionnaire survey derived from the existing litera-
and diffused 共e.g., Abrahamson 1996兲. Of further note is the way ture. Neither could it have emerged through a purely inductive

122 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
research process uninfluenced by different theoretical perspec- structured interviews among the firms’ directors and senior man-
tives. The research findings stand as an example of the richness agers. Part of the initial agreement was that the findings would be
that can be achieved through iterative grounded theory; i.e., by presented back to the firm’s senior management.
repeatedly iterating between the data and different theoretical per- Prior to entering the field, the researchers made sure that they
spectives derived from the literature. The findings also stand as were fully steeped in the relevant literatures. This applied not
testament to the benefits of involving practitioners in the research only to the specific literature on competitive strategy within con-
process. They not only provided a running critique of the inter- struction firms 共e.g., Lansley 1987; Hillebrandt and Cannon 1990;
pretations of the researchers, but also contributed to an environ- Hillebrandt et al. 1995兲, but also to the generic literature relating
ment which sought to understand lean construction in context. to competitiveness. Coverage included the competitive position-
This is in sharp contrast to the dominant approach within the CM ing school 共Porter 1985兲, the resource-based view 共Barney 1991兲
research community which tends to conceptualize lean construc- and the process school 共Pettigrew and Whipp 1991兲. However, in
tion as a substantive set of techniques in isolation of any under- the first instance the interview schedule was primarily informed
standing of the existing dynamics of change. by the dynamic capabilities perspective 共Teece et al. 1997兲.
The research also found support for the Gibbons et al. 共1994兲 Eight senior group and divisional directors were interviewed
ideas of Mode 2 knowledge production. Certainly the empirical within the first company in order to ensure a variety of different
evidence did not support the assumption that the theory of lean perspectives. Advocates of quantitative research methods would
construction was created by academics and that the role of prac- of course immediately criticize such a sample as being too small.
titioners was limited to its application. In contrast, the evidence However, there is clearly only a limited number of senior manag-
pointed toward the continuous reinvention of these ideas that ers who are in a position to comment on the enactment of strat-
egy. The secured sample of eight was close to saturation; few
were meaningful in particular localized contexts. The active par-
additional insights could have been achieved by interviewing ad-
ticipants in this process extended beyond academics and manage-
ditional people. Certainly, the interview schedule was successful
ment gurus to include practicing managers, various quangos
in revealing several rich stories relating to the way in which the
promoting change and the business media. An especially impor-
firm had adjusted to changing circumstances over the previous 30
tant function was seen to be conducted by consultants in terms of
years. And of course, the research methodology did not rely
the mediating role which lay between the management gurus and
solely on interviews. Also important were a selection of corrobo-
the end users of the promoted recipes. The relevance of interor-
rating internal documents coupled with an analysis of the compa-
ganizational networks and social networks is directly related to
ny’s accounts over a 30-year period.
the nature of knowledge creation as a socially embedded process
共Brown and Duguid 1991兲. Emergent Findings
The results of the first case studies are described in full in Green
Case Study 2: Enacting Competitive Strategy in the et al. 共2008b兲. One of the main themes which emerged from the
Construction Sector interviews was the way in which the firm had evolved from a civil
engineering contractor to become a major provider of social hous-
Background ing. The claims made were further corroborated by an analysis of
The second example of the advocated methodology involved construction sector statistics over the period of study. Of particu-
three consecutive case studies each of which was conducted with lar importance was the collapse of the speculative housing market
in the early 1990s and the subsequent slow-but-steady growth of
a separate industry partner. The research sought to bring fresh
the social housing sector. While the concept of dynamic capabili-
insights into the competitive strategy of construction firms
ties provided a useful script for the interview process, it must also
through a series of comparative case studies. Three regional con-
be noted that the adopted methodology provided few insights into
tracting firms were targeted and were approached through a vari-
how the associated transformational processes were enacted. The
ety of routes. The research was initiated by the observation that
research also raised doubts regarding the extent to which any such
the literature on competitive strategy within construction firm had
insights could ever be gained independently of the sociopolitical
stagnated within recent years with few fresh insights since the
machinations which invariably characterize any complex organi-
contributions of Lansley 共1987兲 and Hillebrandt and Cannon
zational context. However, the initial case study emphasized that
共1990兲. The research therefore sought to revisit competitive strat-
strategy is a collective endeavor to which many individuals
egy in construction firms with a particular focus on recent theo- contribute—often in the absence of any boardroom sanction. The
retical developments in terms of the contribution of “dynamic research also emphasized the way in which path dependency
capabilities” 共cf., Teece et al. 1997兲. In essence, the notion of shapes strategic choice. Different contractors are therefore very
dynamic capabilities relates to a firm’s ability to reconfigure its likely to initiate very different responses to prevailing circum-
resources in response to changing environments. stances.
Of particular note was the way in which the emergent findings
Research Method from the first case study caused different theoretical perspectives
The research again sought to enact an iterative grounded theory to be mobilized for the second and third case studies. Inter-
approach combined with a strong orientation toward contextualist viewees repeatedly emphasized the importance of relationships
research. The three case studies were broadly conducted consecu- and the need to become established in localized markets. This
tively, thereby presenting the opportunity to learn from one case resonated strongly with established concepts of embeddedness
study to another. In two cases, the participation of the CEO was and localized learning within the discipline of economic geogra-
secured which made access relatively easy to negotiate. In the phy 共i.e., Uzzi 1996; Maskell et al. 1998兲. According to Maskell
third case, the initial contact was limited to a contracts manager et al. 共1998兲 firms’ competitiveness is essentially rooted in their
and it was necessary to adopt a snowballing strategy. In each case, localized capabilities as constituted within local social networks.
emphasis was given to securing a representative sample of semi- These ideas were not explicitly present in the original research

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 123

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
design, but they did emerge from the research process. As such of practitioners with direct experience of the processes described.
the project provides a good example of iterative grounded theory. The role of the industry workshops was crucial in this respect.
The emergent findings from the first case study led to a revised
empirical focus for the second and third case studies.
Concluding Remarks
Contextualization
As was the case in the previous example, the orientation toward The approach which lies at the core of the methodology outlined
contextualist research was of pivotal importance. The adopted in this paper is that of iterative grounded theory. This has been
allegiance to a becoming ontology crucially focused attention seen to comprise a combination of inductive and deductive ap-
from conceptualizations of strategy as something which is pos- proaches which transcend simplistic dichotomies between quanti-
sessed toward conceptualizations of strategy as something which tative and qualitative research. We remain cautious of those who
is enacted. This has significant implications for the way in which claim to enact grounded theory on the basis of inductive ap-
dynamic capabilities were interpreted. The same also applies to proaches alone. One important characteristic of the approach ad-
the emergent concepts of embeddedness and localized learning. vocated is a reliance on mixing methods. The methods described
The orientation toward contextualist research was also crucial in are those which we have found to be of particular benefit to the
shaping the adopted temporal perspective. Maverick behavior was research we have conducted to date. Clearly the list of methods
seen to play a large part in accounting for the successes and used is by no means exhaustive. More emphasis could have been
failures of the past, and past actions irrevocably shape the rational given to observing participants in their own setting; such an ap-
choices that are available today. Hence if researchers are to un- proach is especially prevalent among those advocating the “prac-
derstand the present reality of construction firms, it is also neces- tice turn” 共cf., Hendry 2000; Jarzabkowski 2005兲. However,
sary to understand the past. In this respect construction access to observe in-house strategy meetings, for example, is
researchers have much to learn from historians. often difficult to negotiate. This is not only due to issues of con-
As was the case in the lean construction study, considerable fidentially, but also through fears that the presence of research
emphasis was given to understanding the dynamics of industry observers at such meetings might subtlety affect the behaviors of
change. But in this case additional emphasis was given to under- those involved. Nevertheless, we would wish to experience a
standing the strategic trajectory of the selected firm. Surprisingly, greater use of such quasi-ethnographic approaches in the future.
the analysis of previous company reports proved to be of signifi- It should also be repeated that we do not contend that the
cant interest to directors within the company who tended only to advocated approach to iterative grounded theory is especially
concern themselves with issues of current performance. The re- new, although it is certainly underused within the context of CM
search also serves as a direct challenge to those who view com- research. What perhaps is new is that we have drawn upon
petitiveness as if it were a tangible, acontextual, and generic broader ideas relating to coproduction and contextualist research.
entity that can be measured and benchmarked 共Green et al. These have been used to pattern a particular approach which goes
2008a,b兲. Much more important is to understand the way in which someway toward meeting the criticisms directed at currently
competitiveness is enacted with evolving and dynamic contexts. dominant approaches. It should also be made clear that there is
Such a shift in understanding has significant implications for the much that has gone unsaid. While advocating the importance of
research techniques that need to be adopted and directly chal- mixing methods it cannot be denied that our underlying expertise
lenges the hypothetico-deductive paradigm which continues to lies in interpretive research. We would hope that the legitimacy of
dominate the international CM research community. interpretive research would be accepted without extensive expla-
nation. To justify the contribution of interpretive research was not
Summary the objective that we set out to satisfy. It should further be made
What is especially relevant within this second example is the way clear that the notion of contextualist research has more to do with
in which the research used a mix of research approaches, thereby an underlying methodological orientation than with specific re-
transcending any simplistic dichotomization between quantitative search techniques. The idea of a becoming ontology presents a
and qualitative methods. The case studies each made significant deeply rooted challenge to the assumptions that shape so much
use of qualitative interview techniques coupled with the quantita- research within the context of CM 共Larsen et al. 2008兲. Likewise,
tive analysis of relevant exogenous factors. The research also pro- the orientation toward coproduction has much more to do with a
vided an excellent example of the process of iterating between generic research style than it does with any firm and fixed set of
empirical data and a secession of different theoretical frame- guiding methodological principles. It is further hoped that the
works. Of further importance was the continuous interaction with examples offered are sufficient to illustrate the essence of the
senior directors from within each of the three the companies, and approach advocated. Especially pertinent is the claim to produce
the way in which this enabled tentative interpretations to be ex- knowledge which is situated in the contexts within which the
plored and critiqued. However, the practitioners also readily con- research was conducted. The advocated model of iterative
tributed their own explanations of the emergent findings in a way grounded theory seeks to embed knowledge in research settings
which was an essential part of the coproduction process. Certainly and thereby satisfies the basic requirements of contextualist re-
the adopted modus operandi can be interpreted as a collective search 共cf., Rooke and Kagioglou 2007兲. The validity of such
sensemaking process involving both academics and practitioners. research findings does not depend upon statistical analysis tech-
This sensemaking directly contributed to the situational aware- niques or any other approach to the testing of predetermined hy-
ness of the participating practitioners, thereby helping better in- potheses. In contrast, validity is determined by the extent to
form future decisions. As was the case with the lean construction which the interpretations offered are found to be credible by those
study the subsequent dissemination and progressive fine-tuning of who operate in contexts similar to those described.
the resultant interpretations were an essential part of the research Finally, we would wish to emphasize that the approach de-
methodology. The validity of the research results were ultimately scribed is intended to complement other research approaches,
determined by demonstrating them to be credible to a wide range rather than replace them. All research methodologies operate on

124 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
the basis of underlying assumptions about the nature of reality enactment, models of diffusion and the meaning of ‘leanness’.” Build.
and the ways in which it can be accessed. The challenge is not to Res. Inf., 33共6兲, 498–511.
be “assumption free,” but rather to be reflexive and self-conscious Green, S. D., and Schweber, L. 共2008兲. “Theorising in the context of
of the assumptions upon which we operate. But one thing is for professional practice: The case for middle range theories.” Build. Res.
Inf., 36共6兲, 649–654.
sure; the CM research community needs more research of the
Greene, J. C. 共1994兲. “Qualitative program evaluation: Practice and
nature described in this paper.
promise.” Handbook of qualitative research, N. K. Denzin and Y. S.
Lincoln, eds., SAGE, London, 530–544.
Hendry, J. 共2000兲. “Strategic decision making, discourse, and strategy as
References social practice.” J. Manage. Stud. (Oxford), 37共7兲, 955–977.
Hillebrandt, P. M., and Cannon, J. 共1990兲. The modern construction firm,
Abrahamson, E. 共1996兲. “Management fashion.” Acad. Manage. Rev., Macmillan, London.
21共1兲, 254–285. Hillebrandt, P. M., Cannon, J., and Lansley, P. 共1995兲. The construction
Ballard, G., and Howell, G. 共1998兲. “Shielding production: An essential company in and out of recession, Macmillan, Basingstoke, U.K.
step in production control.” J. Constr. Eng. Manage., 124共1兲, 11–17. International Labour Office 共ILO兲. 共2001兲. The construction industry in
Barney, J. 共1991兲. “Firm resources and sustained competitive advantage.” the twenty-first century: Its image, employment prospects and skill
J. Manage., 17共1兲, 99–120. requirements, ILO, Geneva, Switzerland.
Benders, J., and Van Bijsterveld, M. 共2000兲. “Leaning on lean: The re- Jarzabkowski, P. 共2005兲. Strategy as practice—An activity-based ap-
ception of a management fashion in Germany.” N. Technol. Work proach, SAGE, London.
Employ., 15共1兲, 50–64. Kinnie, N., Hutchinson, S., and Purcell, J. 共1996兲. “The people implica-
Bresnen, M., and Marshall, N. 共2001兲. “Understanding the diffusion and tions of leaner ways of working.” Rep. No. 15, Institute of Personnel
application of new management ideas in construction.” Eng., Constr., Management, London, 6–63.
Archit. Manage., 8共5/6兲, 335–345. Lansley, P. 共1987兲. “Corporate strategy and survival in the UK construc-
Brown, J. S., and Duguid, P. 共1991兲. “Organizational learning and com- tion industry.” Constr. Manage. Econom., 5共2兲, 141–155.
munities of practice: Toward a unified view of working, learning, and Larsen, G. D., Kao, C. C., and Green, S. D. 共2008兲. “Partnering in flight:
innovation.” Org. Sci., 2共1兲, 40–57. From being to becoming.” J. Constr. Procure., 14共1兲, 35–50.
Chia, R. 共1995兲. “From modern to postmodern organisational analysis.” Lincoln, Y. S., and Guba, E. G. 共1985兲. Naturalistic inquiry, Sage, New-
Organ. Stud., 16共4兲, 579–604. bury Park, Calif.
Cox, R. F., Issa, R. R. A., and Ahrens, D. 共2003兲. “Management’s per- MacLean, D., and MacIntosh, R. 共2002兲. “One process, two audiences:
ception of key performance indicators for construction.” J. Constr. On the challenges of management research.” Eur. Manage. J., 20共4兲,
Eng. Manage., 129共2兲, 142–151. 383–392.
Cully, M., Woodland, S., O’Reilly, A., and Dix, G. 共1999兲. Britain at Maskell, P., Eskelinen, H., Hannibalsson, I., Malmberg, A., and Vatne, E.
work, Routledge, London. 共1998兲. Competitiveness, localised learning and regional develop-
Dainty, A. R. J., Bagilhole, B. M., and Neale, R. H. 共2000a兲. “Computer ment, Routledge, London.
aided analysis of qualitative data in construction management re- Orton, J. D. 共1997兲. “From inductive to iterative grounded theory: Zip-
search.” Build. Res. Inf., 28共4兲, 226–233. ping the gap between process theory and process data.” Scand. J.
Dainty, A. R. J., Bagilhole, B. M., and Neale, R. H. 共2000b兲. “A Manage., 13共4兲, 419–439.
grounded theory of women’s career underachievement in large UK Perkmann, M., and Walsh, K. 共2007兲. “University-industry relationships
construction companies.” Constr. Manage. Econom., 18共2兲, 239–250. and open innovation: Towards a research agenda.” Int. J. Manag. Rev.,
Dervin, B. 共1992兲. “From the mind’s eye of the user: The sense-making 9共4兲, 259–280.
qualitative-quantitative methodology.” Qualitative research in infor- Pettigrew, A. M. 共1997兲. “What is a processual analysis?” Scand. J. Man-
mation management, J. Glazier and R. Powell, eds., Libraries Unlim- age., 13共4兲, 337–348.
ited, Englewood, Colo., 61–84. Pettigrew, A. M. 共2003兲. “Strategy as process, power, and change.” Im-
Dohse, K., Jurgens, U., and Malsch, T. 共1985兲. “From Fordism to Toyo- ages of strategy, S. Cummings and D. Wilson, eds, Blackwell, Ox-
tism? The social organization of the labour process in the Japanese ford, U.K., 301–330.
automobile industry.” Polit. Soc., 14共2兲, 115–146. Pettigrew, A. M., and Whipp, R. 共1991兲. Managing change for competi-
Easterby-Smith, M., Thorpe, R., and Lowe, A. 共1991兲. Management re- tive success, Blackwell, Oxford, U.K.
search: An introduction, SAGE, London. Porter, M. E. 共1985兲. Competitive advantage, Free Press, New York.
Egan, J. 共1998兲. Rethinking construction: Department of the Environ- Rooke, J. 共1997兲. “Developing a more empirical approach to culture,
ment, Transport and the Regions, DETR, London. attitude, and motivation in construction management research: A cri-
Fernie, S., Leiringer, R., and Thorpe, T. 共2006兲. “Change in construction: tique and a proposal.” J. Constr. Procure., 3共2兲, 45–55.
A critical perspective.” Build. Res. Inf., 34共2兲, 91–103. Rooke, J., and Kagioglou, M. 共2007兲. “Criteria for evaluating research:
Fox, P., and Skitmore, M. 共2007兲. “Factors facilitating construction indus- The unique adequacy requirement of methods.” Constr. Manage.
try development.” Build. Res. Inf., 35共2兲, 178–188. Econom., 25共9兲, 979–987.
Garrahan, P., and Stewart, P. 共1992兲. The Nissan enigma: Flexibility at Runeson, G. 共1997兲. “The role of theory in construction management
work in a local economy, Mansell, London. research.” Constr. Manage. Econom., 15共3兲, 299–302.
Gibbons, M., Limoges, C., Nowotony, H., Schwartzman, S., Scott, P., and Seymour, D., Crook, D., and Rooke, J. 共1997兲. “The role of theory in
Trow, M. 共1994兲. The new production of knowledge: The dynamics of construction management: A call for debate.” Constr. Manage.
science and research in contemporary societies, Sage, London. Econom., 15共1兲, 117–119.
Glaser, B. G., and Strauss, A. L. 共1967兲. The discovery of grounded Seymour, D., and Rooke, J. 共1995兲. “The culture of the industry and the
theory: Strategies for qualitative research, Aldine, New York. culture of research.” Constr. Manage. Econom., 13共6兲, 511–523.
Green, S. D., Harty, C. F., Elmualim, A. A., Larsen, G., and Kao, C. C. Strauss, A., and Corbin, J. 共1994兲. “Grounded theory methodology: An
共2008a兲. “On the discourse of construction competitiveness.” Build. overview.” Handbook of qualitative research, N. Denzin and Y. Lin-
Res. Inf., 36共5兲, 426–435. coln, eds., Sage, Thousand Oaks, Calif.
Green, S. D., Larsen, G. D., and Kao, C. C. 共2008b兲. “Competitive strat- Suddaby, R. 共2006兲. “What grounded theory is not.” Acad. Manage J.,
egy revisited: Contested concepts and dynamic capabilities.” Constr. 49共4兲, 633–642.
Manage. Econom., 26共1兲, 63–78. Teece, D. J., Pisano, G., and Shuen, A. 共1997兲. “Dynamic capabilities and
Green, S. D., and May, S. C. 共2005兲. “Lean construction: Arenas of strategic management.” Strategic Manage. J., 18共7兲, 509–533.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 125

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Thomas, H. R., Sanders, S. R., and Bilal, S. 共1992兲. “Comparison of parison of cost for the construction sector: Purchasing power parity.”
labour productivity.” J. Constr. Eng. Manage., 118共4兲, 635–650. J. Constr. Eng. Manage., 131共2兲, 160–167.
Uzzi, B. 共1996兲. “The sources and consequences of embeddedness for the Weick, K. E. 共1995兲. Sensemaking in organizations, SAGE, Thousand
economic performance of organizations: The network effect.” Am.
Oaks, Calif.
Sociol. Rev., 61共4兲, 674–698.
Van Aken, J. E. 共2004兲. “Management research based on the paradigm of Womack, J. P., and Jones, D. T. 共1996兲. Lean thinking, Simon and
the design sciences: The quest for field-tested and grounded techno- Schuster, New York.
logical rules.” J. Manage. Stud. (Oxford), 41共2兲, 219–246. Womack, J. P., Jones, D. T., and Roos, D. 共1990兲. The machine that
Walsh, K. D., Sawhney, A., and Brown, A. 共2005兲. “International com- changed the world, Rawson Associates, New York.

126 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 Research Validation: Challenges and Opportunities
in the Construction Domain
Gunnar Lucko, Ph.D., A.M.ASCE1; and Eddy M. Rojas, Ph.D., A.M.ASCE2

Abstract: Validation of the research methodology and its results is a fundamental element of the process of scholarly endeavor.
Approaches used for construction engineering and management research have included experiments, surveys and observational studies,
modeling and simulation, theory building, case studies, and various subtypes thereof. Some studies use more than one approach. A
particular contribution of this paper is that it reviews different types of validation using examples of studies, analyzes the specific
challenges that were found to be significant, and presents how they were successfully overcome in each case. Another contribution is that
it describes new opportunities for research validation that are emerging at the horizon as well as ongoing collaborative efforts to enhance
the access of construction researchers to validation tools. This paper increases the awareness of the paramount role that validation
techniques play in scholarly work by providing readers with recommendations to properly validate their own research efforts.
DOI: 10.1061/共ASCE兲CO.1943-7862.0000025
CE Database subject headings: Control systems; Methodology; Models; Research; Sampling; Statistics; Surveys; Validation;
Verification.
Author keywords: Control methods; Delphi method; Methodology; Models; Research needs; Sampling design; Statistical analysis;
Surveys; Validation; Verification.

Introduction attain with human means, because scientific endeavor is by defi-

nition never-ending and iterative. The large body of knowledge in
Construction engineering and management research examines philosophy that explores the concept of truth underlines its fun-
real-world means and methods in an effort to improve the effec- damental nature, but is outside of the scope of this paper. As such,
tiveness and efficiency of the construction industry. It is essential the purpose of validation is to ensure that each phase of the cho-
for any scientific inquiry that researchers ensure the quality of sen research methodology rigorously adheres to the highest stan-
their work in every step of its methodology, including data col- dards of quality. This level of quality in planning, executing, and
lection, analysis, and interpretation of results, through appropriate evaluating research is measured as validity.
validation techniques. The results and the process by which they Law 共2007兲 stressed that validation always depends on the
were derived need to be accepted by the academic and the pro- specific purpose of the research. El-Diraby and O’Connor 共2004兲
fessional communities so that the new knowledge becomes an- asserted that “no single definition of the ingredients or subsets of
other stepping stone in the advancement of the state-of-the-art and the concept of validity” exists. Numerous writers discuss validity,
filters down to daily practice, ultimately contributing to the wel- yet each study uses a slightly different definition of the concept
fare of society. It is therefore also important that researchers col- 共e.g., Roschke 1994; Kamat and Martínez 2003兲. Sargent 共1991兲
laborate with industry practitioners to establish credibility. elaborated further that “currently no algorithms or procedures
The scientific method is a structured approach to generate new 关are兴 available to identify specific validation techniques, statistical
knowledge through critical inquiry. While the methodologies of tests, etc., to use in the validation process.” Therefore, validation
scientific research vary significantly depending on the particular is a challenge to all researchers, but especially so to those work-
subject and domain, its singular goal always remains the same, to ing in interdisciplinary fields such as construction engineering
discover truth. However, absolute scientific truth is impossible to and management.
In light of the absence of definite criteria for how to accom-
1
Assistant Professor of Civil Engineering and Director of Construc- plish validation, this paper first provides an overview of the vari-
tion Engineering and Management Program, Dept. of Civil Engineering, ous types of validity. The focus is not on particular research
The Catholic Univ. of America, 620 Michigan Ave. NE, Washington,
D.C. 20064 共corresponding author兲. E-mail: [email protected]
methods, because their inherent diversity renders a one-on-one
2
Professor and Executive Director, Pacific Northwest Center for Con- matching less meaningful, and because several different valida-
struction Research and Education, Dept. of Construction Management, tion techniques may be used in the same study. Rather, this paper
Univ. of Washington, 130G Architecture Hall, Box 351610, Seattle, WA examines various accepted validation techniques and illustrates
98195. E-mail: [email protected] how the challenge of establishing validity has been successfully
Note. This manuscript was submitted on July 23, 2008; approved on resolved in construction research with numerous examples. It also
December 29, 2008; published online on March 17, 2009. Discussion
introduces new validation opportunities emerging at the horizon
period open until June 1, 2010; separate discussions must be submitted
for individual papers. This paper is part of the Journal of Construction as well as collaborative efforts by various organizations to in-
Engineering and Management, Vol. 136, No. 1, January 1, 2010. crease the accessibility of validation resources for construction
©ASCE, ISSN 0733-9364/2010/1-127–135/$25.00. researchers.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 127

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
Validation and Related Concepts Criterion validity has been defined as “the extent to which the
results of an assessment instrument correlate with another, pre-
Validation stands in contrast to verification. Both terms are com- sumably related measure 共the latter is called the criterion”兲
mon in technical and managerial literature, but are frequently 共Babbie 1990兲. Criterion validity is thus established when the
used in an unclear or even interchangeable manner. Verification findings of a research study on a particular aspect of the construc-
establishes the technical correctness of a research product, e.g., a tion industry, e.g., factors that influence craft productivity, agree
simulation model 共Sargent 1991兲, in accordance with its specifi- in general with the outcomes of related studies, which e.g., inves-
cations that were established a priori. It is often performed inter- tigated training programs for skill development or the motiva-
nally by the researcher, who thoroughly checks all components in tional effects of working conditions, even though their detailed
a manner of proofreading. Errors that should be identified and approaches may differ.
eliminated during verification can range from trivial and innocu- Construct validity refers to whether operationalizations of the-
ous deviations to essential and destructive divergences. Examples oretical constructs are appropriate. In other words, construct va-
are “bugs” in computer code, transcription errors in survey an- lidity is concerned with ensuring that a research effort is
swers, transposition errors in numerical data, misconnections of measuring what it is supposed to measure according to its stated
schedule activities, and omitted or doubled components. Valida- objectives 共Leedy and Ormrod 2001兲. For example, a study about
tion is the more elusive one of the two concepts, as verification is the relationship between the motivation and productivity of con-
primarily a comparison of measurable performance parameters. struction workers is said to have construct validity if, and only if,
Together, they constitute an acceptance by both its developers and the instruments used actually measure motivation and productiv-
users as being credible, which particularly in the area of simula- ity. Even though this may seem obvious, there are many “ab-
tion is called accreditation or certification 共Law 2007兲. stract” constructs, especially traits of the human behavior or
In general terms, one could state that verification is concerned psyche that have varying definitions without universally accepted
with “doing things right,” while validation is concerned with metrics or that need to be detailed further to be usable. Motiva-
“doing the right things.” The process of validation can be broadly tion, for example, may have to be refined with more objective
divided into two main areas: establishing internal and external factors, e.g., need for work, salary and consequences of nonper-
validity. Internal validity is related to the concept of causality and formance. In such cases, researchers must justify the process by
is preoccupied with the derivability of relations within data which the instruments and their conceptual contents were devel-
共Leedy and Ormrod 2001兲. External validity is related to the con- oped to ensure construct validity.
cept of induction and focuses on the generalizability of results for Finally, another term that is sometimes confused with validity
prediction purposes 共Leedy and Ormrod 2001兲. Internal validity is reliability. It is related to the concepts of consistency and re-
can be threatened by many problems, including ill-defined theo- peatability in the data collection. According to Neuman 共2000兲, it
retical models that include spurious relationships or correlated “is necessary for validity and is easier to achieve than validity.”
explanatory variables, biases in data collection that render com- As with validity, there are different kinds of reliability, including
parisons ineffective, and failure to entertain alternative explana- interrater reliability, internal reliability, test-retest reliability, and
tions during data analysis. Note that a true causality can only be parallel forms reliability. Interrater reliability is concerned with
established under carefully controlled, laboratory-like environ- consistency between different measured subjects. For example, a
ment, but not in observational studies, where “only” correlation productivity study that relies on several individuals to visit con-
can be identified. External validity can be threatened by a variety struction sites and collect labor productivity data has interrater
of issues, including lack of statistical rigor in the selection of reliability if all observers classify the activities of workers in the
sample sizes and in collecting actual data, the presence of any same manner, i.e., do not impress personal views onto the data,
special circumstances during the research efforts, and oversimpli- and can thus be considered neutral or “interchangeable” in the
fication of the phenomena under study. analysis.
Besides internal and external, other types of validity that are Internal reliability focuses on the consistency within a measur-
commonly referred to in the literature include face validity, con- ing instrument, i.e., individual questions thereof, and can be quan-
tent validity, criterion validity, and construct validity. Face valid- tified statistically using Cronbach’s ␣ 共Cronbach 1951兲. For
ity is a subjective judgment of nonstatistical nature that seeks the example, if a questionnaire is created to survey the aforemen-
opinion of nonresearchers regarding the validity of a particular tioned laborers’ motivation, it should contain several questions
study 共Leedy and Ormrod 2001兲. For example, a simulation that “cross-check” each other without being apparent to the re-
model of an earthmoving operation can be said to have face va- spondent, i.e., are not just rewordings. Randomizing the order of
lidity if earthmoving experts looking at the model and its output questions can further remove the respondent’s attention from this
agree that it represents to a high degree what happens in reality. In internal feature. Rich literature on considerations for survey stud-
a practical field of study such as construction engineering and ies can be consulted by construction researchers for designing
management, the collaboration with appropriate representatives their measuring instruments 共e.g., Babbie 1990; Cresswell 1994;
from the private and public sectors, e.g., industry practitioners, Leedy and Ormrod 2001兲.
government agencies, and also the public at large, is very impor- Test-retest reliability refers to the consistency of results when
tant to secure the face validity of the research endeavors. the same measurement tool is repeatedly applied to the same sce-
Content validity is another nonstatistical approach that focuses nario. This repeatability of measurements is not only essential to
on determining if the content of a study fairly represents reality. remove the manner of applying a test as a potential error source,
Its primary concern is “the degree to which a measure covers the but also if other researchers want to reproduce the documented
range of meanings included within the concept” 共Babbie 1990兲. results. For example, a survey instrument is deemed to have test-
For example, a study of stress levels of construction supervisors is retest reliability if the responses are closely correlated when the
said to have content validity if the tasks, activities, events, and survey is applied to the same sample persons in two different
environmental factors that are included can be considered to be instances. However, researchers must take care that a possible
representative of those that generally occur in the larger industry. learning effect by the test subjects, who may remember details of

128 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
the previous test, is not mistakenly interpreted as test-retest reli-
ability. Regression
Estimate
Parallel forms reliability refers to the consistency of different, Analysis

but related, measurement tools when applied to the same sample.

As an illustration, parallel forms reliability can be achieved if two ±50%
Regression
Coefficients
different surveys to assess the level of construction workers’ job
satisfaction provide highly correlated results. Random
Dataset Split
Construction engineering and management research must be
±50% New Correlation
verified, validated, and reliable to achieve the highest level of Response ŷ and t-Test
quality. This paper, however, focuses mainly on the validation
process. Original
Predict Response y

Challenges
Fig. 1. Cross-validation technique applied by Lucko et al. 共2006兲
Given the interdisciplinary nature of construction engineering and
management research as well as the inherent difficulties of per-
forming studies in real-life settings, research validation in the
construction domain encounters a variety of challenges. This sec- gression model. Menches and Hanna 共2006兲 used “leave-one-out
tion presents examples of how such typical difficulties have been cross validation, 关where兴 one case is left out of the data set, and
successfully overcome by construction researchers. In their en- the remaining n−1 cases are used to construct the model” for their
tirety, these examples provide guidance to the research commu- performance index for project managers, similar to the resampling
nity for enhancing the quality of construction engineering and technique jackknifing. Multiple cross-validations yielded a mis-
management studies by delivering meaningful valid outcomes. classification rate, which was desired to be less than 10%. Resa-
mpling is also employed in bootstrapping, named after “pulling
oneself up by one’s own bootstraps,” i.e., with existing data. It
Internal Validity
assumes that the data set is representative of the population. Nu-
As mentioned, internal validity is related to causality. Establish- merous new data sets of the same sample size n are created from
ing causality is a significant challenge for construction research- the existing data set by repeated resampling with replacement.
ers, given that many studies are performed in real-life settings Different from validation, the technique estimates distributions
where multiple variables interact with one another, many of them for statistics of interest, e.g., mean or median 共Barton and
uncontrollable or unpredictable. While laboratory experiments are Schruben 2001兲.
less common in the construction engineering and management Multiple regression analysis must also satisfy a variety of con-
domain than in other areas of inquiry, such as e.g., structural ditions other than internal consistency reliability to ensure inter-
engineering, they represent one of the best strategies to study nal validity 共Lucko et al. 2006兲. The internal validity of such
causality. The tight control of environmental factors that is pos- studies can be threatened by issues such as misspecification of the
sible in a laboratory setting can strongly establish internal validity model components 共excluding important or including unimportant
of the research. Researchers should try to perform laboratory independent variables兲, multicollinearity 共independent variables
studies whenever possible. Ideally, the laboratory study should be are highly correlated and thus redundant兲, micronumerosity
augmented with field trials, as performed by Wu 共2005兲 while 共small sample size兲, heteroskedasticity 共nonconstant variance in
examining the flowable backfill material for pipelines. data points兲, autocorrelation 共repetitive pattern within the data
Internal validity can also be achieved by means other than points兲, nonzero expected disturbances 共regression assumption of
laboratory trials under controlled conditions. For example, re- zero error mean does not hold兲, and nonlinearity 共regression as-
searchers can use multiple regression analysis to isolate the effect sumption of linear relationship between independent and depen-
of individual independent variables within a model 共Montgomery dent variables does not hold兲 共Gujarati 1995兲. A variety of tests
et al. 2001兲. The results of multiple regression analysis, however, and techniques can be found in the literature to ensure that these
must exhibit internal consistency reliability as a necessary condi- problems are not present within a model. Zayed and Halpin
tion to be valid. Statistical resampling techniques can be used to 共2005b兲 performed F-tests on their regression models for dura-
ascertain internal consistency reliability without the need to col- tions, productivity, and cost of pile construction and t-tests for the
lect new data. Smaller samples are repeatedly taken from an ex- null hypothesis that each coefficient ␤ was identical to zero,
isting data set, provided its sufficient sample size. Cross- whose rejections together and individually established the linear
validation is a well-known technique that can test the stability of relationship of the variables.
a model. The existing data set is randomly split into two portions Researchers often attempt to define models based on statistical
and the independent and dependent variables of the first one are techniques for goodness-of-fit. While these techniques can be
used to estimate the model coefficients. The fitted model then very useful in building a functioning model, researchers must still
makes predictions from the independent variables of the second verify that all included variables are related to the dependent vari-
portion, which are compared statistically with its actual dependent able by some theoretical construct in order to ensure internal va-
variables as shown in Fig. 1. Zayed and Halpin 共2005a兲 split their lidity. In other words, there must be a clear and straightforward
data set of pile construction productivities into 70% estimation reason why a given variable belongs in a model, e.g., ambient
and 30% prediction portions. Lucko et al. 共2006兲 employed a 50% temperature and labor productivity, other than just simple corre-
to 50% split after checking Snee’s 共1977兲 recommendation of lation with the dependent variable, e.g., the consumption of gaso-
having at least n ⱖ 2 · p + 25 data points to allow splitting, where line and the number of housing starts in residential construction.
n = sample size and p = number of estimated parameters in the re- Otherwise, the variables would be judged as spurious, i.e., seem-

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 129

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
ingly related but in truth both caused by another, hitherto hidden If data are not measured directly by researchers but rather are
third variable, e.g., the income and debt levels of private house- created by respondents through a measuring instrument, e.g., a
holds. survey questionnaire, the response rate becomes especially impor-
tant. As the ratio of actually received responses divided by ran-
domly selected and requested responses, it directly influences the
External Validity
sample size. Achieving a high response rate is a challenge that
External validity is related to generalizability of results. In order construction researchers often face. However, simple techniques
to generalize from a sample, the sample must be representative of can boost the response rate and—by extension—increase their
the population. One of the most common means to ensure repre- external validity. Davis and Songer 共2003兲 used a combination of
sentativeness is to randomize the sampling procedure. In a survey techniques, starting with telephone and personal announcements
study on leadership characteristics among construction employ- of the survey followed by mailing “a cover letter introducing the
ees, Ozorovskaja et al. 共2007兲 selected companies at random from research study, an informed consent form, and the questionnaire,
listings in the yellow pages. This ensured that all listed companies along with an envelope in which to place the completed question-
had an equal chance of being included in their sample. Conse- naire to be either picked up later or mailed back . . . Follow-up
quently, external validity was established, i.e., conclusions drawn phone calls and office/site visits were performed as reminders for
from the sample could be generalized to that population. In some unreturned surveys,” which yielded a response rate of 58.4%.
cases, other techniques are also employed to support generaliz- Other techniques can seek to increase the motivation of respon-
ability. Stratification divides the population into groups that shall dents, e.g., by promising an executive summary of the results
be represented in the sample. Davis and Songer 共2003兲 used 共Ling 2002兲 or a small remuneration, gift, or prize lottery in ex-
“company size, industry sector, and profession, age, and gender” change for completing the survey.
as strata in their mail survey on resistance to technological change Research is by definition an incremental process and research-
to ensure that the proportions of the Unites States construction ers need to be keenly aware of the extent to which their work
industry were reflected in their samples. However, conscious sam- contributes to the body of knowledge. Describing the scope and
pling alone is not sufficient for external validity. The sample size limitations of the overall study is as important as revealing its
also determines whether the inherent variability within the popu- assumptions, possible bias, and possible error sources. External as
lation has been captured sufficiently. Depending on whether the well as content and face validity directly stem from the research
study is cross-sectional or longitudinal, the sample size describes being placed into its proper context while clearly acknowledging
the breadth or length of the data set. what it can accomplish versus what remains for future study.
Statistical equations provide guidance on necessary minimum Rojas and Kell 共2008兲 described the limitations of their empirical
sample sizes for research studies, beyond the obvious rule-of- comparison of project delivery systems in terms of geographic
thumb that more samples are always better, especially for very area, industry sector, time frame, convenience sampling from ex-
heterogeneous populations 共Leedy and Ormrod 2001兲. Note that isting government records, and how the sample size related to the
this differs from the resolution or “fineness” of the data points, Gaussian distribution assumption of the central limit theorem for
which should be appropriate to the desired level of detail of the statistical tests.
research results. Lucko et al. 共2006兲 evaluated the sample size n The most pertinent and arguably also most difficult form of
of their existing data sets of heavy equipment resale values using external and face validation that can be accomplished is the de-
several recommendations from the literature that included the ployment and application of a new model, method, or technology
number of explanatory variables k in the model, e.g., n ⱖ 15· k for into the ongoing industry practice. It is sometimes called pilot
multiple linear regression, n ⱖ 50+ 8 · k for multiple correlation,
testing or prototyping. Since the construction industry is known to
and n ⱖ 104+ k for testing explanatory variables. These guidelines
adopt innovations only hesitantly 共Davis and Songer 2003兲, this
consider the “level of detail” of the model by counting its vari-
can present a formidable challenge. Examples of construction re-
ables. The population size also should be considered in determin-
search that reported actual implementation include e.g., owner-
ing the sample size a priori. Gay et al. 共2006兲 provided sample
contractor work structures 共Anderson et al. 2004兲 and change
size percentages for ranges of population sizes based on earlier
management 共Price and Chahal 2006兲 in industry practice, and in
research that had assumed a statistical significance level ␣ = 0.05
the educational domain the interactive, collaborative, and
共Krejcie and Morgan 1970兲
computer-supported teaching of estimating and scheduling con-
• The larger the population size, the smaller the percentage of
cepts 共Jaafari et al. 2001兲.
the population required to get a representative sample.
• For smaller populations, say N = 100 or fewer, there is little
point in sampling; survey the entire population. Face Validity
• If the population size is around 500 共give or take 100兲, 50% of
the population should be sampled. Face validity requires the “approval” of nonresearchers regarding
• If the population size is around 1,500, 20% should be sampled. the validity of a study. Performing a research study directly under
• Beyond a certain point 共about N = 5 , 000兲, the population size the harsh and somewhat unpredictable conditions of a construc-
is almost irrelevant, and a sample size of 400 will be adequate. tion site can generate face validity, provided that such environ-
Other writers also give different recommendations depending on mental factors in the widest sense are measured and documented
whether the data are continuous or discrete 共Bartlett et al. 2001兲. as well. Thomas et al. 共1999兲, for example, studied the influence
Since sample size is intimately related to various statistical prop- of weather on labor productivity for steel erection projects in the
erties, e.g., “the variance, the desired significance level, the power field. Navon and Berkovich 共2005兲 evaluated a new materials
of hypothesis tests . . . and the widths of the confidence and management system on a commercial building project, which also
prediction intervals” 共Lucko et al. 2006兲, the writers recommend allowed a comparison with the existing approach. While a labo-
to confer with a statistician to ensure the external validity of a ratory setting works well for certain studies, e.g., testing new
new research project. materials, the aforementioned practice-oriented nature of con-

130 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
struction research often makes such a real-life setting necessary. most widely used qualitative method for validating expert sys-
Even though this makes creating internal validity challenging, it tems,” Roschke 共1994兲 noted that the “blind” Turing test conceals
goes a long way toward securing face validity. the identity of the decision-makers and thus reduces bias. It can
Arguably the strongest way to establish face validity is the also establish interrater reliability. For his expert system that se-
involvement of domain experts, also known as subject matter ex- lects bridge rails, he asked an expert to review fifteen field cases
perts, before 共a priori兲, during, after 共a posteriori兲, or throughout and for each one to distinguish the computer-generated decision
the research. Such participation can range from an advisory ca- from those of two human experts and concluded that the expert’s
pacity to active collaboration. Obviously, the earlier and the more inability to do so established face validity.
such input is sought, the stronger it can contribute to face validity. A technique that is specifically applicable for face validation
For example, a hallmark of research projects sponsored by the of discrete-event simulation computer models is scrutinizing its
Construction Industry Institute has been to include approximately trace file 共Sargent 1991兲, also known as event log, to match with
a dozen industry representatives in addition to the academic per- the sequence of events as expected by the model developer. This
sonnel. Gibson et al. 共1995兲 even reported the involvement “of
automatically generated record contains times at which any state
experienced industry professionals representing 16 companies,
changes occur. Such instances are activities changing between
which ensured expertise for model development” in their study on
being idle and active and resources being sent and received. Fol-
preproject planning. Especially for the construction industry,
lowing the entries in the trace file for at least one complete run
which is often viewed as resistant to innovation 共Blayse and Man-
and comparing them manually with the execution of the model
ley 2004兲, it is increasingly essential that through such involve-
can reveal small errors in links between model elements, their
ment it takes a more active role in initiating, supporting, and
implementing the research that seeks to provide solutions to its start and finish conditions, and the consistency of time and re-
current and future needs. source units as well as large errors in the logic of the entire
Another technique useful to establish face validity is the per- system. Such manual technique also contributes to the verification
formance of interviews. El-Diraby and O’Connor 共2004兲 con- of the simulation model.
ducted numerous structured interviews with government agencies Beyond checking the bare text of the trace file for validation, it
and consultants for their bridge plans evaluation model. They also finds an important use as an input file for simulation post-
deliberately sought a diverse group to attain a wide range of opin- processing. Visualization and animation can help users in natu-
ions. Considerations for conducting meaningful interviews are rally spotting inconsistencies in the simulation that have eluded
similar to those for surveys. Structured questions provide internal the developers 共Kamat and Martínez 2003兲. For users to accept
reliability. However, interviews allow a richer feedback, as the the model as a truthful representation of reality, its behavior must
interviewer can clarify and extend individual items ad hoc in a mirror their “mental image” of the real system. Research has been
semi-structured manner. An audio recording or at least detailed performed on both ends of the spectrum, from simple abstractions
notes during or directly after the session is vital. Note that an as in the animation of icons 共Zhang et al. 2002兲 to almost photo-
approval of research with human subjects is typically required. realistic environments of construction operations 共Kamat and
“Focus groups are carefully planned discussions stimulated Martínez 2003兲. A particular advantage of four-dimensional visu-
within a predefined group environment to obtain perceptions alizations, i.e., animations, is that it lets users perform face vali-
about a defined area of interest in a permissive, nonjudgmental dation even of complex logic without the need to learn a specific
environment” 共Yu et al. 2006兲. A more extensive technique than computer interface 共Ioannou and Martínez 1996兲. However,
interviews, they include “interaction and self-disclosure among González-Quevedo 共1995兲 also reminds that realism alone does
participants” and thus can lead to a consensus among experts 共Yu not guarantee a “good” model and that trivialities that may ob-
et al. 2006兲. However, they need to be carefully moderated to not scure its abstract essence should be avoided. Recent research is
deviate into tangential conversations. Refining the technique, Yu turning the previous one-way data flow from simulation to visu-
et al. 共2006兲 created transcriptions of their sessions on critical alization into a dynamic interaction, i.e., users can manipulate the
success factors and performed a content analysis on them. The underlying simulation model via the graphical user interface and
latter technique is described in detail in the literature of the social directly observe the impact of such changes 共Rekapalli and Mar-
sciences, where diverse qualitative data are often handled. tínez 2007兲.
The Delphi analysis, named after the ancient Greek oracle, is a
Construction research often creates face validity through case
validation technique that formalizes the collection of expert feed-
studies, meaning the discussion of a self-contained, detailed ex-
back. Its characterizing feature is that undergoes at least two it-
ample application of a new model or method. The issues sur-
erations. The first phase often collects data through a survey or a
rounding the assumptions, scope, limitations, and resolution of
group of interviews with experts, which are compiled by the re-
searcher. In the second phase the same experts receive all data the case study are similar to those encountered in modeling. The
and are asked to provide further comments. This “peer review” challenge of striking a balance between realism and abstraction as
can also generate a consensus. del Cano and de la Cruz 共2002兲 well as between inclusiveness and manageability becomes a mat-
used a Delphi analysis with interviews for validation in their ter of parsimony, applying “Occam’s razor,” the principle of being
study of project risk management. Demographic information, “as simple as possible 共but as complicated as necessary兲” Ad-
such as the age, education, position, and experience with different dressing this issue, Karam et al. 共2007兲 used a case study of a
project types of the interviewed experts as well as their company 2-km-long tunnel in Korea to illustrate and validate their com-
sizes and industry sectors were reported meticulously to support puter tool for decision-making for tunnel construction. Rojas and
the validation process. Dossick 共2008兲 presented a comprehensive case study of an inte-
The Turing test, which asks a user to decide whether the grated construction research and education center that exemplified
anonymous “conversation” partner is a human or a machine, is their experiential learning and interdisciplinary research ap-
still uncommon in the literature on construction research but its proaches. Different from their use in research validation, case
well known in the area of artificial intelligence. Considered “the studies can also be found in the form of scholarly reports on

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 131

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
innovations in construction engineering and management prac- Criterion Validity
tice, e.g., on physically moving an existing concrete building with
hydraulics 共Telem et al. 2006兲. Criterion validity requires correlation between related measure-
ments. A powerful, yet simple technique for criterion validity and
reproducibility of research is to compare predictions with those
Content Validity obtained from a different model or method. This can be combined
with extending the often less complex nature of such previous
Content validity relates the content of a study with reality. Col- model or method to new capabilities, e.g., increased accuracy or
lecting data from a population for scientific study is the phase wider applicability. An abundance of different types of compari-
within the research methodology that has a fundamental influence sons is found in construction engineering and management re-
on the quality of the subsequent analysis and interpretation. Since search, e.g., González-Quevedo et al. 共1993兲, who specifically
collecting data are time and cost intensive, researchers typically compared the performance of two simulation systems for model-
are not able to measure the spectrum of possible observations, ing construction operations in terms of productivity and duration.
i.e., the entire population. Rather, they collect samples in such a Lucko 共2007兲 validated the results from a new analytical model
way that they are representative of the characteristics of the popu- for linear schedules with manual calculations. Elazouni and Met-
lation. If sampling is performed consciously as to which data are wally 共2007兲 used genetic algorithms to automatically optimize
collected and how, it can establish significant content validity, schedules under financial constraints and validated the results of
e.g., for survey research. their model by performing linear programming on the same data.
In rare cases it may be possible to aggregate existing data from
several redundant sources. Such “triangulation” echoes its nauti-
cal meaning of increasing the accuracy by gauging from several Construct Validity
angles. It was used by Lucko et al. 共2006兲, who combined data Construct validity ensures that a research effort measures what it
from two independent equipment auction databases to substanti- is supposed to measure. One way to accomplish construct validity
ate their content validity. Data received a source label and were is through the performance of a pilot test. Different from the pilot
merged and sorted. In a two-step mechanism a computer macro implementation of research results, the word pilot in this context
then matched entries by sales date, location, and serial number to refers to fine-tuning the instrument before its use in the actual
identify identical events that had been recorded by both sources, data collection. Ling 共2002兲 established construct validity—and
filled any gaps with pieces of information from the respective also content validity—in this manner for a study on architects’
other source, and deleted all doublettes. In a discussion of com- and engineers’ performance. Col Debella and Ries 共2006兲 exam-
bining qualitative and quantitative research methodologies, Cre- ined project delivery methods and as the first step of their survey
swell 共1994兲 uses the term triangulation in the broader meaning of study pilot tested the questionnaire on several respondents to en-
removing “bias inherent in particular data sources, investigator, sure its effectiveness. No numeric measures exist to capture the
and method” by combining several of these listed items. improvement thus gained, but conceptually the effect is similar to
One of the most important ways in which researchers can involving domain experts during the course of the research.
show the integrity of their research methodology is documenting
their entire approach in detail with an open and self-critical mind.
Abstracting and modeling real-world processes entails simplify- Reliability
ing assumptions, which must be clearly stated so that readers can Reliability requires consistency and repeatability. An interesting
independently assess the quality of the study, creating content and approach to establish test-retest reliability of a new approach was
face validity. Neuman 共2000兲 illustrated the behavior of data used by Kallantzis et al. 共2007兲, who created a random input
points with a bull’s eye target, where the scatter 共or variability兲 generator for their linear scheduling research. Ranges of con-
around the center represents reliability and the drift 共or tendency兲 straints and buffers and a set of possible productivity values were
off the center represents validity. Scatterplots of the raw data randomized into 25 sample schedules with six sequential activi-
show their centrality and any patterns or trends in an approximate ties and six units of work so that “not only is the margin for error
form before any statistical analysis is performed. A systematic reduced, but also the possibility of a coincidental or biased result
bias would be visible as a recurring deviation from the center. is minimized and statistic conclusions can be drawn with relative
Other meanings of bias may also be encountered, e.g., in a statis- safety” 共Kallantzis et al. 2007兲. In rare cases, it may also be
tical sense as the opposite of random sampling and in seeking possible to use existing standardized inputs to evaluate the per-
statistical estimators to be unbiased. Even the researchers them- formance of new approaches, e.g., the project scheduling problem
selves may be biased, e.g., if they unconsciously favor a particu- library with 2,040 randomly generated resource-constrained net-
lar outcome. A related phenomenon is known as publication bias. works of four different sizes that was developed by Kolisch and
Conscientious researchers document and seek to remove bias as Sprecher 共1996兲. Vanhoucke 共2006兲 used it as a benchmark for
much as possible. Davis and Songer 共2003兲 acknowledged that examining work continuity in repetitive projects.
“the pilot sample is biased” because companies that were unlisted A powerful technique to test the internal consistency reliability
in the telephone book would have been ignored. They identified of models is sensitivity analysis. It seeks to quantify how vari-
another bias toward sampling from managerial positions, as field ability in the values of independent variable impacts the depen-
personnel were less likely to have office addresses, nor answer dent variable 共Elmaghraby 2000兲. In its most basic form, it entails
long surveys. Since participation was voluntary, respondents were varying each input either randomly or systematically while hold-
generally “more likely to be persons to be inclined to fill out ing other inputs fixed. Sargent 共1991兲 described a related tech-
surveys . . . or those that thought that the research sounded inter- nique of testing simulation models under extreme conditions, i.e.,
esting” 共Davis 2004兲. A survey was used instead of interviews to specifically setting inputs to their possible maxima and minima. A
“provide a high external validity, low internal validity solution” generalized version would be a factorial experimental design
共Davis 2004兲. 共Kelton and Barton 2003兲. Since inputs may not just vary by

132 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
themselves alone, but can possibly create interaction effects, it is considered as a critical element in the process of developing
can be sensible to extend the approach to a scenario analysis, and verifying metrics and tools to attain improvements in con-
where various combinations of input values are examined in con- struction productivity. FIATECH and NIST are currently collabo-
cert. The choice of the specific values depends on the number of rating to 共1兲 identify a draft set of requirements 共capabilities兲 for
inputs, their range and order of magnitude, continuous or discrete an IACJS test-bed; 共2兲 identify an initial set of high-impact target
nature, and the desired resolution of the output variability. This technologies, processes, or methods to evaluate within the IACJS
technique was adapted by Rojas and Aramvareekul 共2003兲, who test-bed; and 共3兲 identify industry partners interested in using de-
traced apparent variability in macroeconomic measures of con- veloped test-bed capabilities.
struction labor productivity to actual changes in the weights be- NSF and NIST are also working together toward the develop-
tween industry sectors, each with an “inherent” level of ment of NSF-NIST Advanced Building Infrastructure Test-Beds
productivity, rather than to a perceived decline of employee out- 共NABIT兲. “There is a variety of ongoing research related to intel-
put. In an interesting variation of sensitivity analysis of simula- ligent infrastructure, building commissioning, building informa-
tion models, Sargent 共1991兲 lists degeneracy as a validation tion integration systems that have special needs related to their
technique, which also entails “removing portions of the model” to verification and validation. These system-level projects employ-
evaluate the performance of changed “hardware” rather than just ing advanced information and communication technologies 共ICT兲
varying input values. Such testing technique is known in software and methods require sophisticated test-beds to realistically and
engineering as fault injection 共Hsueh et al. 1997兲. comprehensively evaluate the impact and process improvements
Computer models of phenomena that exhibit probabilistic brought about by the products of that research. The use of test-
properties can be tested for internal consistency reliability beds as a means of verifying and validating this ICT-related re-
through multiple replications 共Sargent 1991兲. In the construction search is critical to the successful evolution of this field of
domain, such phenomena include e.g., activity durations 共Lee inquiry, both among researchers as well as between researchers
2005兲, cost estimates and bids 共Chua et al. 2001兲, and soil condi- and the user community. There are many NSF-funded projects
tions 共Karam et al. 2007兲. A direct relationship exists with the that could benefit greatly from awareness of and access to exist-
quality of the input data, which are replaced by input distributions ing test-beds through which they could verify and validate their
共AbouRizk et al. 1994兲 from which samples are drawn in a quasi- research. There are many potential resources at NIST that would
random manner. In each repetition, or run, the specific realization make excellent test-beds for such research. What is needed is a
of the probabilistic variable is created via a random number gen- vehicle to bring together institutional needs of students and fac-
erator. Isidore and Back 共2002兲 analyzed risk from both schedule ulty, and potential verification and validation resources 共facilities,
and cost with “nested” repetitions of “25 times using 100 runs per equipment, and data兲 developed by NIST staff” 共Akin and Garrett
simulation.” A direct extension of the replication approach is the 2006兲.
widely used Monte Carlo method, which—more powerful than As a result of the NABIT initiative, a database of test-beds is
sensitivity analysis—seeks to quantify the probability parameters currently being developed under the umbrella of the Virtual Com-
of the output in cases when the probabilistic input is too complex munity of Construction Scholars and Practitioners 具 www.cschol-
for a direct, deterministic analysis 共Chantaravarapan et al. 2004兲. ars.washington.edu 典 . It is partially sponsored by the NSF and the
objective of its database is to provide an avenue for construction
researchers to easily identify potential test-beds that are available
Opportunities to validate their research. Once this database is populated, it will
include information such as the type of test-bed, a brief descrip-
The construction research community recognizes the importance tion, its past uses, and a contact person to gather additional infor-
of validation of research efforts as well as the challenges associ- mation and gain access to the test-bed. Government agencies and
ated within the construction domain. Indeed, the National Science members of the construction community will be invited to upload
Foundation 共NSF兲, the National Institute of Standards and Tech- information into the database.
nology 共NIST兲, and the Fully Integrated and Automated Tech- Construction researchers must realize that models and tools
nologies for Construction 共FIATECH兲 industry consortium, are created in the performance of their own research may very well
collaborating to foster the dialog about construction test-beds. serve as test-beds for other research, even when they were not
Test-beds include items such as physical laboratories, mathemati- developed with that purpose. For example, Rojas and Mukherjee
cal models, algorithms, computer simulations, and databases, as 共2006兲 developed a situational simulation termed “Virtual Coach”
well as their integration into sophisticated systems. They allow to support the educational process of future construction project
for rigorous and replicable testing to realistically and comprehen- managers. This gamelike environment was funded by the U.S.
sively evaluate the impact of research endeavors. Department of Education, given its potential impact on learning
FIATECH’s Intelligent and Automated Construction Job Site paradigms. However, as a result of this effort, the dynamics of the
共IACJS兲 test-bed was identified by its Capital Projects Technol- construction phase of commercial projects have been modeled
ogy Roadmap development team as one of the strategies needed and the system now also supports significant user interaction as
to help achieve the vision of the construction job site of the fu- well as the creation of new simulations by nonprogramers. There-
ture. The objective is to “focus on the design, development, de- fore, once face validity of this model is established, it can serve as
ployment, and use of an advanced set of test facilities that will a test-bed to evaluate various hypotheses. One could imagine the
provide the first major tests of the IACJS test-bed concepts. This performance of a study about information timeliness by testing
test-bed will provide realistic environments in a laboratory setting the sensitivity of the system to changes in the delivery time of
from which proposed and developed IAJCS technologies can be information to decision-makers. Some test groups may receive
tested for potential use on actual construction sites” 共FIATECH real-time data, while others may experience delays ranging from a
2004兲. FIATECH recognizes that having this capability will ac- few days to a couple of weeks. The advantage of such a scheme is
celerate industry adoption of new ideas that could improve safety, that test groups would be exposed to the same simulated construc-
cut costs, and reduce construction schedules. The IACJS test-bed tion environment and would experience the same simulation

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 133

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
events. Indeed, this may create a virtual laboratory where the comparative analysis of their performance within school districts.” J.
performance of each group can be evaluated by analyzing their Constr. Eng. Manage., 132共11兲, 1131–1138.
execution in terms of schedule and budget. Cresswell, J. W. 共1994兲. Research design: Qualitative and quantitative
approaches, SAGE, Thousand Oaks, Calif., 具http://www.sagepub.
com/home.nav典.
Conclusion Cronbach, L. J. 共1951兲. “Coefficient alpha and the internal structure of
tests.” Psychometrika, 16共3兲, 297–335.
Davis, K. A. 共2004兲. “Information technology change in the architecture,
Validation of the research methodology and its results is a funda- engineering, and construction industry: An investigation of individu-
mental element of the process of scholarly endeavor. Approaches als’ resistance.” Ph.D. dissertation, Virginia Polytechnic Institute and
used for construction engineering and management research have State Univ., Blacksburg, Va.
included experiments, surveys and observational studies, model- Davis, K. A., and Songer, A. D. 共2003兲. “Individuals’ resistance to tech-
ing and simulation, theory building, case studies, and various sub- nological change in the AEC industry.” Proc., 2003 Construction Re-
types thereof. Some studies use more than one approach. A search Congress, K. R. Molenaar and P. S. Chinowsky, eds., ASCE,
particular contribution of this paper has been reviewing different Reston, Va., 8.
types of validation using examples of studies, analyzing the spe- del Cano, A., and de la Cruz, M. P. 共2002兲. “Integrated methodology for
cific challenges that were found to be significant, and presenting project risk management.” J. Constr. Eng. Manage., 128共6兲, 473–485.
how they were successfully overcome in each case. Another con- Elazouni, A. M., and Metwally, F. G. 共2007兲. “Expanding finance-based
tribution has been describing new opportunities for research vali- scheduling to devise overall-optimized project schedules.” J. Constr.
dation that are emerging at the horizon as well as ongoing Eng. Manage., 133共1兲, 86–90.
El-Diraby, T. E., and O’Connor, J. T. 共2004兲. “Lessons learned in design-
collaborative efforts to enhance the access of construction re-
ing research methodology in field-based construction research.” J.
searchers to validation tools. This paper has increased the aware- Profl. Issues Eng. Educ. Pract., 130共2兲, 109–114.
ness of the paramount role that validation techniques play in Elmaghraby, S. E. 共2000兲. “On criticality and sensitivity in activity net-
scholarly work by providing readers with recommendations to works.” Eur. J. Oper. Res., 127, 220–238.
properly validate their own research efforts. FIATECH. 共2004兲. Capital Projects Technology Roadmapping Initiative:
Element 4 Tactical Plan, Intelligent and Automated Construction Job
Site (IACJS), Fully Integrated and Automated Technologies for Con-
Acknowledgments struction, Austin, Tex.
Gay, L. R., Mills, G. E., and Ariasian, P. 共2006兲. Educational research:
The writers thank Ms. Kathleen M. Mahoney, Graduate Library Competencies for analysis and applications, 8th Ed., Prentice-Hall,
Pre-Professional at the Engineering/Architecture/Mathematics Li- Upper Saddle River, N.J.
brary of The Catholic University of America, for her kind support Gibson, G. E., Kaczmarowski, J. H., and Lore, H. E. 共1995兲. “Preproject-
planning process for capital facilities.” J. Constr. Eng. Manage.,
in obtaining library materials for this study.
121共3兲, 312–318.
González-Quevedo, A., AbouRizk, S. M., Iseley, D. T., and Halpin, D. W.
共1993兲. “Comparison of two simulation methodologies in construc-
References tion.” J. Constr. Eng. Manage., 119共3兲, 573–589.
González-Quevedo, A. A. 共1995兲. “Sensitivity analysis of construction
AbouRizk, S. M., Halpin, D. W., and Wilson, J. R. 共1994兲. “Fitting beta simulation.” Proc., 2nd Congress on Computing in Civil Engineering,
distributions based on sample data.” J. Constr. Eng. Manage., 120共2兲, J. P. Mohsen, ed., ASCE, Reston, Va., 1427–1434.
288–305. Gujarati, D. N. 共1995兲. Basic econometrics, 3rd Ed., McGraw-Hill, New
Akin, O., and Garrett, J. H. 共2006兲. “Final report: NSF-NIST workshop York.
on advanced building infrastructure testbed 共NABIT兲.” Division of Hsueh, M.-C., Tsai, T. K., and Iyer, R. K. 共1997兲. “Fault injection tech-
Civil and Mechanical Systems, National Science Foundation, Arling- niques and tools.” Computer, 30共4兲, 75–82.
ton, Va. Ioannou, P. G., and Martínez, J. C. 共1996兲. “Animation of complex con-
Anderson, S. D., Patil, S. S., Gibson, G. E., and Sullivan, G. R. 共2004兲. struction simulation models.” Proc., 3rd Congress on Computing in
“Owner-contractor work structures: Process approach.” J. Constr. Civil Engineering, J. A. Vanegas and P. S. Chinowsky, eds., ASCE,
Eng. Manage., 130共5兲, 680–690. Reston, Va., 620–626.
Babbie, E. 共1990兲. Survey research methods, Wadsworth, Belmont, Calif., Isidore, L. J., and Back, W. E. 共2002兲. “Multiple simulation analysis for
具http://www.cengage.com/wadsworth典. probabilistic cost and schedule integration.” J. Constr. Eng. Manage.,
Bartlett, J. E., Kotrlik, J. W., and Higgins, C. C. 共2001兲. “Organizational 128共3兲, 211–219.
research: Determining appropriate sample size in survey research.” Jaafari, A., Manivong, K. K., and Chaaya, M. 共2001兲. “VIRCON: Inter-
Business Education Index, 19共1兲, 43–50. active system for teaching construction management.” J. Constr. Eng.
Barton, R. R., and Schruben, L. W. 共2001兲. “Resampling methods for Manage., 127共1兲, 66–75.
input modeling.” Proc., 2001 Winter Simulation Conf., B. A. Peters, J. Kallantzis, A., Soldatos, J., and Lambropoulos, S. 共2007兲. “Linear versus
S. Smith, D. J. Medeiros, and M. W. Rohrer, eds., IEEE, Piscataway, network scheduling: A critical path comparison.” J. Constr. Eng. Man-
N.J., 372–378. age., 133共7兲, 483–491.
Blayse, A. M., and Manley, J. 共2004兲. “Key influences on construction Kamat, V. R., and Martínez, J. C. 共2003兲. “Interactive discrete-event
innovation.” Constr. Innovation, 4共3兲, 143–154. simulation of construction processes in dynamic immersive 3D virtual
Chantaravarapan, S., Gunal, A., and Williams, E. J. 共2004兲. “On using worlds.” Proc., 2003 Conf. on Construction Applications of Virtual
Monte Carlo methods for scheduling.” Proc., 2004 Winter Simulation Reality, W. Y. Thabet and D. A. Bowman, eds., Virginia Polytechnic
Conf., R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters, Institute and State University, Blacksburg, Va., 197–201.
eds., IEEE, Piscataway, N.J., 1870–1975. Karam, K. S., Karam, J. S., and Einstein, H. H. 共2007兲. “Decision analy-
Chua, D. K. H., Li, D. Z., and Chan, W. T. 共2001兲. “Case-based reasoning sis applied to tunnel exploration planning. I: Principles and case
approach in bid decision making.” J. Constr. Eng. Manage., 127共1兲, study.” J. Constr. Eng. Manage., 133共5兲, 344–353.
35–45. Kelton, W. D., and Barton, R. R. 共2003兲. “Experimental design for simu-
Col Debella, D., and Ries, R. 共2006兲. “Construction delivery systems: A lation.” Proc., 2003 Winter Simulation Conf., S. Chick, P. J. Sánchez,

134 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright
 D. Ferrin, and D. J. Morrice, eds., IEEE, Piscataway, N.J., 59–65. and R. R. Barton, eds., IEEE, Piscataway, N.J., 2028–2031.
Kolisch, R., and Sprecher, A. 共1996兲. “PSPLIB—A project scheduling Rojas, E. M., and Aramvareekul, P. 共2003兲. “Is construction labor pro-
problem library.” Eur. J. Oper. Res., 96共1兲, 205–216. ductivity really declining?” J. Constr. Eng. Manage., 129共1兲, 41–46.
Krejcie, R. V., and Morgan, D. W. 共1970兲. “Determining sample size for Rojas, E. M., and Dossick, C. S. 共2008兲. “Developing a state-of-the-art
research activities.” Educ. Psychol. Meas., 30, 607–610. facility to support construction research and education: A case study.”
Law, A. M. 共2007兲. Simulation modeling and analysis, 4th Ed., McGraw- J. Profl. Issues Eng. Educ. Pract., 134共1兲, 67–74.
Hill, New York. Rojas, E. M., and Kell, I. 共2008兲. “Comparative analysis of project deliv-
Lee, D.-E. 共2005兲. “Probability of project completion using stochastic ery systems cost performance in Pacific Northwest public schools.” J.
project scheduling simulation.” J. Constr. Eng. Manage., 131共3兲, Constr. Eng. Manage., 134共6兲, 387–397.
310–318. Rojas, E. M., and Mukherjee, A. 共2006兲. “A multi-agent framework for
Leedy, P. D., and Ormrod, J. E. 共2001兲. Practical research planning and general-purpose situational simulations in the construction manage-
design, 7th Ed., Prentice-Hall, Upper Saddle River, N.J. ment domain.” J. Comput. Civ. Eng., 20共3兲, 165–176.
Ling, Y. Y. 共2002兲. “Model for predicting performance of architects and Roschke, P. N. 共1994兲. “Validation of knowledge-based system with mul-
engineers.” J. Constr. Eng. Manage., 128共5兲, 446–455. tiple bridge rail experts.” J. Transp. Eng., 120共5兲, 787–806.
Lucko, G. 共2007兲. “Flexible modeling of linear schedules for integrated Sargent, R. G. 共1991兲. “Simulation model verification and validation.”
mathematical analysis.” Proc., 2007 Winter Simulation Conf., S. G. Proc., 1991 Winter Simulation Conf., W. D. Kelton, G. M. Clark, and
Henderson, B. Biller, M.-H. Hsieh, J. Shortle, J. D. Tew, and R. R. B. L. Nelson, eds., IEEE, Piscataway, N.J., 37–47.
Barton, eds., IEEE, Piscataway, N.J. Snee, R. D. 共1977兲. “Validation of regression models: Methods and ex-
Lucko, G., Anderson-Cook, C. M., and Vorster, M. C. 共2006兲. “Statistical amples.” Technometrics, 19共4兲, 415–428.
considerations for predicting residual value of heavy equipment.” J. Telem, D., Shapira, A., Goren, Y. D., and Schexnayder, C. J. 共2006兲.
Constr. Eng. Manage., 132共7兲, 723–732. “Moving a reinforced-concrete building: Case study.” J. Constr. Eng.
Menches, C. L., and Hanna, A. S. 共2006兲. “Quantitative measurement of Manage., 132共2兲, 115–124.
successful performance from the project manager’s perspective.” J. Thomas, H. R., Riley, D. R., and Sanvido, V. E. 共1999兲. “Loss of labor
Constr. Eng. Manage., 132共12兲, 1284–1293. productivity due to delivery methods and weather.” J. Constr. Eng.
Montgomery, D. C., Peck, E. A., and Vining, G. G. 共2001兲. Introduction Manage., 125共1兲, 39–46.
to linear regression analysis, 3rd Ed., Wiley, New York. Vanhoucke, M. 共2006兲. “Work continuity constraints in project schedul-
Navon, R., and Berkovich, O. 共2005兲. “Development and on-site evalua- ing.” J. Constr. Eng. Manage., 132共1兲, 14–25.
tion of an automated materials management and control model.” J. Wu, J. Y. 共2005兲. “Soil-based flowable fill for pipeline construction.”
Constr. Eng. Manage., 131共12兲, 1328–1336. Proc., 2005 Pipeline Division Specialty Conf., C. Vipulanandan and
Neuman, W. L. 共2000兲. Social research methods: Qualitative and quan- R. Ortega, eds., ASCE, Reston, Va., 925–938.
titative approaches, 4th Ed., Allyn and Bacon, Needham Heights, Yu, A. T. W., Shen, Q., Kelly, J., and Hunter, K. 共2006兲. “Investigation of
Mass. critical success factors in construction project briefing by way of con-
Ozorovskaja, R., Voordijk, J. T., and Wilderom, C. P. M. 共2007兲. “Lead- tent analysis.” J. Constr. Eng. Manage., 132共11兲, 1178–1186.
ership and cultures of Lithuanian and Dutch construction firms.” J. Zayed, T. M., and Halpin, D. W. 共2005a兲. “Pile construction productivity
Constr. Eng. Manage., 133共11兲, 900–911. assessment.” J. Constr. Eng. Manage., 131共6兲, 705–714.
Price, A. D. F., and Chahal, K. 共2006兲. “A strategic framework for change Zayed, T. M., and Halpin, D. W. 共2005b兲. “Productivity and cost regres-
management.” Constr. Manage. Econom., 24共3兲, 237–251. sion models for pile construction.” J. Constr. Eng. Manage., 131共7兲,
Rekapalli, P. V., and Martínez, J. C. 共2007兲. “A message-based architec- 779–789.
ture to enable runtime user interaction on concurrent simulation- Zhang, H., Shi, J. J., and Tam, C. M. 共2002兲. “Iconic animation for
animations of construction operations.” Proc., 2007 Winter Simulation activity-based construction simulation.” J. Comput. Civ. Eng., 16共3兲,
Conf., S. G. Henderson, B. Biller, M.-H. Hsieh, J. Shortle, J. D. Tew, 157–164.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / JANUARY 2010 / 135

Downloaded 29 Dec 2009 to 158.42.244.86. Redistribution subject to ASCE license or copyright; see http://pubs.asce.org/copyright