Practical Assessment, Research, and Evaluation

Volume 12 Volume 12, 2007 Article 10

2007

The Delphi Technique: Making Sense of Consensus

Chia-Chien Hsu

Brian A. Sandford

Follow this and additional works at: https://scholarworks.umass.edu/pare

Recommended Citation
Hsu, Chia-Chien and Sandford, Brian A. (2007) "The Delphi Technique: Making Sense of Consensus,"
Practical Assessment, Research, and Evaluation: Vol. 12 , Article 10.
Available at: https://scholarworks.umass.edu/pare/vol12/iss1/10

This Article is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for
inclusion in Practical Assessment, Research, and Evaluation by an authorized editor of ScholarWorks@UMass
Amherst. For more information, please contact [email protected].
 Hsu and Sandford: The Delphi Technique: Making Sense of Consensus

A peer-reviewed electronic journal.

Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment,
Research & Evaluation. Permission is granted to distribute this article for nonprofit, educational purposes if it is
copied in its entirety and the journal is credited.

Volume 12, Number 10, August 2007 ISSN 1531-7714

The Delphi Technique:

Making Sense Of Consensus
Chia-Chien Hsu, The Ohio State University &
Brian A. Sandford, Oklahoma State University

The Delphi technique is a widely used and accepted method for gathering data from respondents within their
domain of expertise. The technique is designed as a group communication process which aims to achieve a
convergence of opinion on a specific real-world issue. The Delphi process has been used in various fields of
study such as program planning, needs assessment, policy determination, and resource utilization to develop a
full range of alternatives, explore or expose underlying assumptions, as well as correlate judgments on a topic
spanning a wide range of disciplines. The Delphi technique is well suited as a method for consensus-building by
using a series of questionnaires delivered using multiple iterations to collect data from a panel of selected
subjects. Subject selection, time frames for conducting and completing a study, the possibility of low response
rates, and unintentionally guiding feedback from the respondent group are areas which should be considered
when designing and implementing a Delphi study.

The Delphi technique, mainly developed by Dalkey and 1. To determine or develop a range of possible program
Helmer (1963) at the Rand Corporation in the 1950s, is a alternatives;
widely used and accepted method for achieving
2. To explore or expose underlying assumptions or information
convergence of opinion concerning real-world knowledge
leading to different judgments;
solicited from experts within certain topic areas.
Predicated on the rationale that, “two heads are better than 3. To seek out information which may generate a consensus on
one, or...n heads are better than one” (Dalkey, 1972, p. 15), the part of the respondent group;
the Delphi technique is designed as a group
4. To correlate informed judgments on a topic spanning a wide
communication process that aims at conducting detailed
range of disciplines, and;
examinations and discussions of a specific issue for the
purpose of goal setting, policy investigation, or predicting 5. To educate the respondent group as to the diverse and
the occurrence of future events (Ulschak, 1983; Turoff & interrelated aspects of the topic (p. 11).
Hiltz, 1996; Ludwig, 1997). Common surveys try to
CHARACTERISTICS OF THE
identify “what is,” whereas the Delphi technique attempts
DELPHI TECHNIQUE
to address “what could/should be” (Miller, 2006).
The Delphi technique is well suited as a means and method
In the literature, Delphi has been applied in various
for consensus-building by using a series of questionnaires
fields such as program planning, needs assessment, policy
to collect data from a panel of selected subjects (Dalkey &
determination, and resource utilization. Delbecq, Van de
Helmer, 1963; Dalkey, 1969; Linstone & Turoff, 1975;
Ven, and Gustafson (1975) specifically indicate that the
Lindeman, 1981; Martino, 1983; Young & Jamieson, 2001).
Delphi technique can be used for achieving the following
Delphi, in contrast to other data gathering and analysis
objectives:
techniques, employs multiple iterations designed to
Published by ScholarWorks@UMass Amherst, 2007 1
 Practical Assessment, Research, and Evaluation, Vol. 12 [2007], Art. 10
Practical Assessment, Research & Evaluation, Vol 12, No 10 2
Hsu & Sandford, Delphi Technique

develop a consensus of opinion concerning a specific topic. the information developed by previous iterations.
Ludwig (1994) indicates: Through the operation of multiple iterations, subjects are
expected to become more problem-solving oriented, to
Iterations refer to the feedback process. The process was viewed as a
offer their opinions more insightfully, and to minimize the
series of rounds; in each round every participant worked through a
effects of noise.
questionnaire which was returned to the researcher who collected, edited,
and returned to every participant a statement of the position of the whole Finally, the ability to use statistical analysis techniques
group and the participant’s own position. A summation of comments is a practice which further reduces the potential of group
made each participant aware of the range of opinions and the reasons pressure for conformity (Dalkey, 1972). More specifically,
underlying those opinions (p. 55). statistical analysis can ensure that opinions generated by
each subject of a Delphi study are well represented in the
More specifically, the feedback process allows and
final iteration because, “at the end of the exercise there may
encourages the selected Delphi participants to reassess
still be a significant spread in individual opinions” (Dalkey,
their initial judgments about the information provided in
1972, p. 21). That is, each subject would have no pressure,
previous iterations. Thus, in a Delphi study, the results of
either real or perceived, to conform to another participant’s
previous iterations regarding specific statements and/or
responses that may originate from obedience to social
items can change or be modified by individual panel
norms, customs, organizational culture, or standing within
members in later iterations based on their ability to review
a profession. The tools of statistical analysis allow for an
and assess the comments and feedback provided by the
objective and impartial analysis and summarization of the
other Delphi panelists.
collected data.
Other notable characteristics inherent with using the
Delphi technique are the ability to provide anonymity to THE DELPHI PROCESS
respondents, a controlled feedback process, and the Theoretically, the Delphi process can be continuously
suitability of a variety of statistical analysis techniques to iterated until consensus is determined to have been
interpret the data (Dalkey, 1972; Ludlow, 1975; Douglas, achieved. However, Cyphert and Gant (1971), Brooks
1983). These characteristics are designed to offset the (1979), Ludwig (1994, 1997), and Custer, Scarcella, and
shortcomings of conventional means of pooling opinions Stewart (1999) point out that three iterations are often
obtained from a group interaction (i.e., influences of sufficient to collect the needed information and to reach a
dominant individuals, noise, and group pressure for consensus in most cases. The following discussion,
conformity) (Dalkey, 1972). however, provides guidelines for up to four iterations in
One of the primary characteristics and advantages of order to assist those who decide to use the Delphi process
the Delphi process is subject anonymity which can reduce as a data collection technique when it is determined that
the effects of dominant individuals which often is a additional iterations beyond three are needed or valuable.
concern when using group-based processes used to collect Round 1: In the first round, the Delphi process
and synthesize information (Dalkey, 1972). Additionally, traditionally begins with an open-ended questionnaire. The
the issue of confidentiality is facilitated by geographic open-ended questionnaire serves as the cornerstone of
dispersion of the subjects as well as the use of electronic soliciting specific information about a content area from
communication such as e-mail to solicit and exchange the Delphi subjects (Custer, Scarcella, & Stewart, 1999).
information. As such, certain downsides associated with After receiving subjects’ responses, investigators need to
group dynamics such as manipulation or coercion to convert the collected information into a well-structured
conform or adopt a certain viewpoint can be minimized questionnaire. This questionnaire is used as the survey
(Helmer & Rescher, 1959; Oh, 1974; Adams, 2001). instrument for the second round of data collection. It
Controlled feedback in the Delphi process is designed should be noted that it is both an acceptable and a common
to reduce the effect of noise. Based upon Dalkey (1972), modification of the Delphi process format to use a
noise is that communication which occurs in a group structured questionnaire in Round 1 that is based upon an
process which both distorts the data and deals with group extensive review of the literature. Kerlinger (1973) noted
and/or individual interests rather than focusing on that the use of a modified Delphi process is appropriate if
problem solving. As a result, the information developed basic information concerning the target issue is available
from this kind of communication generally consists of bias and usable.
not related to the purposes of the study. Basically, the Round 2: In the second round, each Delphi participant
controlled feedback process consists of a well organized receives a second questionnaire and is asked to review the
summary of the prior iteration intentionally distributed to items summarized by the investigators based on the
the subjects which allows each participant an opportunity information provided in the first round. Accordingly,
to generate additional insights and more thoroughly clarify Delphi panelists may be required to rate or “rank-order

https://scholarworks.umass.edu/pare/vol12/iss1/10 2
 Hsu and Sandford: The Delphi Technique: Making Sense of Consensus
Practical Assessment, Research & Evaluation, Vol 12, No 10 3
Hsu & Sandford, Delphi Technique

items to establish preliminary priorities among items. As a Rescher (1959), Klee (1972), and Oh (1974) concur that
result of round two, areas of disagreement and agreement choosing individuals who are simply knowledgeable
are identified” (Ludwig, 1994, p. 54-55). In some cases, concerning the target issue is not sufficient nor
Delphi panelists are asked to state the rationale concerning recommended. Considering the necessity of selecting the
rating priorities among items (Jacobs, 1996). In this round, most qualified individuals, Delbecq, Van de Ven, and
consensus begins forming and the actual outcomes can be Gustafson (1975) specifically state that three groups of
presented among the participants’ responses (Jacobs, people are well qualified to be subjects of a Delphi study.
1996). The authors recommend:
Round 3: In the third round, each Delphi panelist receives “(1) the top management decision makers who will utilize the
a questionnaire that includes the items and ratings outcomes of the Delphi study;
summarized by the investigators in the previous round and
(2) the professional staff members together with their support
are asked to revise his/her judgments or “to specify the
team; and
reasons for remaining outside the consensus” (Pfeiffer,
1968, p. 152). This round gives Delphi panelists an (3) the respondents to the Delphi questionnaire whose judgments
opportunity to make further clarifications of both the are being sought” (p. 85).
information and their judgments of the relative importance
Delphi subjects should be highly trained and
of the items. However, compared to the previous round,
competent within the specialized area of knowledge related
only a slight increase in the degree of consensus can be
to the target issue. Investigators need to closely examine
expected (Weaver, 1971; Dalkey & Rourke, 1972; Anglin,
and seriously consider the qualifications of Delphi subjects.
1991; Jacobs, 1996).
Oh (1974) indicates that choosing appropriate subjects is
Round 4: In the fourth and often final round, the list of generally based on the judgment and discretion of the
remaining items, their ratings, minority opinions, and items principal investigators. Jones and Twiss (1978) state that
achieving consensus are distributed to the panelists. This the principal investigators of a Delphi study should identify
round provides a final opportunity for participants to and select the most appropriate individuals through a
revise their judgments. It should be remembered that the nomination process. Ludwig (1994) also states that,
number of Delphi iterations depends largely on the degree “solicitation of nominations of well-known and respected
of consensus sought by the investigators and can vary from individuals from the members within the target groups of
three to five (Delbecq, Van de Ven, Gustafson, 1975; experts was recommended” (p. 52). Generally, the pool of
Ludwig, 1994). selecting possible Delphi subjects is likely to use positional
leaders (Kaplan, 1971; Ludwig, 1994), to follow a review of
Subject Selection
authors of publications in the literature (Meyer, 1992;
Regarding the selection of subjects for a Delphi study, Miller, 2001), and/or to make contacts with those who
choosing the appropriate subjects is the most important have firsthand relationships with a particular issue (Jones,
step in the entire process because it directly relates to the 1975; Anderson & Schneider, 1993). The latter basically
quality of the results generated (Judd, 1972; Taylor & Judd, consists of individuals who are primary stakeholders with
1989; Jacobs, 1996). Since the Delphi technique focuses on various interests related to the target issue or research
eliciting expert opinions over a short period of time, the effort.
selection of Delphi subjects is generally dependent upon
Concerning the appropriate number of subjects to
the disciplinary areas of expertise required by the specific
involve in a Delphi study, Delbecq, Van de Ven, and
issue.
Gustafson (1975) recommend that researchers should use
Regarding any set standards of selecting Delphi the minimally sufficient number of subjects and should
subjects, there is, in fact, no exact criterion currently listed seek to verify the results through follow-up explorations.
in the literature concerning the selection of Delphi Ludwig (1994) notes that the number of experts used in a
participants. That is, “throughout the Delphi literature, the Delphi study is "generally determined by the number
definition of [Delphi subjects] has remained ambiguous” required to constitute a representative pooling of
(Kaplan, 1971, p. 24). Regarding the criteria used to guide judgments and the information processing capability of the
the selection of Delphi subjects, individuals are considered research team” (p. 52). However, what constitutes an
eligible to be invited to participate in a Delphi study if they optimal number of subjects in a Delphi study never reaches
have somewhat related backgrounds and experiences a consensus in the literature. Delbecq, Van de Ven, and
concerning the target issue, are capable of contributing Gustafson (1975) suggest that ten to fifteen subjects could
helpful inputs, and are willing to revise their initial or be sufficient if the background of the Delphi subjects is
previous judgments for the purpose of reaching or homogeneous. In contrast, if various reference groups are
attaining consensus (Pill, 1971; Oh, 1974). Helmer and involved in a Delphi study, more subjects are anticipated to

Published by ScholarWorks@UMass Amherst, 2007 3

 Practical Assessment, Research, and Evaluation, Vol. 12 [2007], Art. 10
Practical Assessment, Research & Evaluation, Vol 12, No 10 4
Hsu & Sandford, Delphi Technique

be needed. Witkin and Altschuld (1995) note that the of criteria to use to both define and determine consensus in
approximate size of a Delphi panel is generally under 50, a Delphi study is subject to interpretation. Basically,
but more have been employed. Ludwig (1997) documents consensus on a topic can be decided if a certain percentage
that, “the majority of Delphi studies have used between 15 of the votes falls within a prescribed range (Miller, 2006).
and 20 respondents” (p. 2). In sum, the size of Delphi One criterion recommends that consensus is achieved by
subjects is variable (Delbecq, Van de Ven, & Gustafson, having 80 percent of subjects’ votes fall within two
1975). If the sample size of a Delphi study is too small, categories on a seven-point scale (Ulschak, 1983). Green
these subjects may not be considered as having provided a (1982) suggests that at least 70 percent of Delphi subjects
representative pooling of judgments regarding the target need to rate three or higher on a four point Likert-type
issue. If the sample size is too large, the drawbacks scale and the median has to be at 3.25 or higher. Scheibe,
inherent within the Delphi technique such as potentially Skutsch, and Schofer (1975) reveal that the use of
low response rates and the obligation of large blocks of percentage measures is inadequate. They suggest that a
time by the respondents and the researcher(s) can be the more reliable alternative is to measure the stability of
result. subjects’ responses in successive iterations.
Time Requirements In the Delphi process, data analysis can involve both
qualitative and quantitative data. Investigators need to deal
Conducting a Delphi study can be time-consuming.
with qualitative data if classic Delphi studies, which use
Specifically, when the instrument of a Delphi study
open-ended questions to solicit subjects’ opinions, are
consists of a large number of statements, subjects will need
conducted in the initial iteration. Subsequent iterations are
to dedicate large blocks of time to complete the
to identify and hopefully achieve the desired level of
questionnaires. Delbecq, Van de Ven, and Gustafson
consensus as well as any changes of judgments among
(1975), Ulschak (1983), and Ludwig, (1994) recommend
panelists. The major statistics used in Delphi studies are
that a minimum of 45 days for the administration of a
measures of central tendency (means, median, and mode)
Delphi study is necessary. With regard to the time
and level of dispersion (standard deviation and
management between iterations, Delbecq et al. (1975) note
inter-quartile range) in order to present information
that giving two weeks for Delphi subjects to respond to
concerning the collective judgments of respondents
each round is encouraged.
(Hasson, Keeney, & McKenna, 2000). Generally, the uses
Ludwig (1994) indicates, “a drawback to Delphi was of median and mode are favored. However, in some cases,
that the questionnaire method may slow the process greatly as manifested by Murray and Jarman (1987), the mean is
as several days or weeks may pass between rounds" (p. 54). also workable. Witkin (1984) questions the
More specifically, since developing the instrument, appropriateness of using the mean to measure the subjects’
collecting the data, and administering the questionnaire are responses if scales used in Delphi studies are not delineated
interconnected between iterations, ensuring Delphi at equal intervals. In the literature, the use of median score,
subjects respond to the investigators on time does in many based on Likert-type scale, is strongly favored (Hill &
ways either promote or prohibit the ability of the Fowles, 1975; Eckman, 1983; Jacobs, 1996). As Jacobs
investigators in analyzing the data, developing a new (1996) states, “considering the anticipated consensus of
instrument based upon the prior responses, and opinion and the skewed expectation of responses as they
distributing subsequent questionnaires in a timely fashion. were compiled, the median would inherently appear best
These are challenging aspects of conducting a Delphi study suited to reflect the resultant convergence of opinion” (p.
and do require proper planning and management. 57). The use of mode is also suitable when reporting data
in the Delphi process. Ludwig (1994) specifically
The use and prevalence of electronic technologies (i.e.,
addressed that “the Delphi process has a tendency to create
e-mail, teleconferencing, etc.) may facilitate those who are
convergence, and though this was usually to a single point,
interested in using the Delphi technique. Witkin and
there was the possibility of polarization or clustering of the
Altschuld (1995) note that electronic technology provides
results around two or more points. In these instances, the
an opportunity for individuals to more easily employ the
mean or median could be misleading” (p. 57).
Delphi process by taking advantages of, “(1) the storage,
processing, and speed of transmission capabilities of
computers; (2) the maintenance of respondent anonymity,
and; (3) the potential for rapid feedback” (p. 204). CONSIDERING DELPHI
SHORTCOMINGS AND WEAKNESSES
Data Analysis
Potential of Low Response Rates
Regarding data analysis, decision rules must be established
to assemble and organize the judgments and insights Due to the multiple feedback processes inherent and
provided by Delphi subjects. However, the kind and type integral to the concept and use of the Delphi process, the

https://scholarworks.umass.edu/pare/vol12/iss1/10 4
 Hsu and Sandford: The Delphi Technique: Making Sense of Consensus
Practical Assessment, Research & Evaluation, Vol 12, No 10 5
Hsu & Sandford, Delphi Technique

potential exists for low response rates and striving to Potential of Identifying General Statements vs.
maintain robust feedback can be a challenge. “In the Specific Topic Related Information
Delphi technique, [poor response rate] is magnified
An assumption concerning Delphi participants is that they
fourfold because a maximum of four surveys may be sent
are equivalent in knowledge and experience (Altschuld &
to the same panelists” (Witkin & Altschuld, 1995, p. 196).
Thomas, 1991). However, this assumption might not be
If a certain portion of the subjects discontinue their
justified. More specifically, the expertise of Delphi
responses during various stages of the Delphi process, the
panelists could be unevenly distributed, especially in the
quality of information obtained could be discounted or at
field of high technology (Marchant, 1988; Altschuld &
least critically scrutinized. As such, Ludwig (1994)
Thomas, 1991). “Some panelists may have much more
specifically addresses subject motivation as the key to the
in-depth knowledge of certain topics, whereas other
successful implementation of a Delphi study and
panelists are more knowledgeable about different topics”
investigators need to play an active role in this area to help
(Altschuld & Thomas, 1991, p. 187). Therefore, subjects
ensure as high a response rate as possible.
who have less in-depth knowledge of certain topics are
Consumption of Large Blocks of Time unable to specify the most important statements which
have been identified by those subjects who possess
The Delphi technique can also be time-consuming and
in-depth knowledge concerning the target issue. The
laborious. Unlike other data collection techniques such as
outcomes of a Delphi study could be the results of
the telephone survey and the face-to-face administration,
identifying a series of general statements rather than an
which can be simultaneously conducted by a group of
in-depth exposition of the topic (Altschuld & Thomas,
people and can be completed in a short period of time if
1991).
the sample size is small, the Delphi technique is iterative
and sequential. As a result, the necessity of taking large SUMMARY
block of time to successively complete a Delphi process is
The Delphi technique provides those involved or
inescapable. Ludwig (1994) indicates that, “a drawback to
interested in engaging in research, evaluation, fact-finding,
Delphi was that the questionnaire method may slow the
issue exploration, or discovering what is actually known or
process greatly as several days or weeks may pass between
not known about a specific topic a flexible and adaptable
rounds” (p. 54). Optimally speaking, the iteration
tool to gather and analyze the needed data. Subject
characteristics of the Delphi process provide the
selection and the time frames for conducting and
opportunities for investigators and subjects to improve the
completing a Delphi study are two areas which should be
accuracy of the results. In contrast, the same characteristic
considered carefully prior to initiating the study. The
also increases the workload of investigators and the
additional precautions concerning low response rates,
amount of time needed to successfully complete the data
unintentionally guiding feedback, and surveying panelists
collection process (Cunliffe, 2002).
about their limited knowledge of the topic rather than
Potential of Molding Opinions soliciting their expert judgments should also be built into
The iteration characteristics of the Delphi technique can the design and implementation of the study. The Delphi
potentially enable investigators to mold opinions technique has and will continue to be an important data
(Altschuld, 2003). An experiment, conducted by Scheibe, collection methodology with a wide variety of applications
Skutsch, and Schofer (1975), indicated that Delphi subjects and uses for people who want to gather information from
would rate their responses differently after receiving a those who are immersed and imbedded in the topic of
distorted feedback. Dalkey and Helmer (1963) also noted interest and can provide real-time and real-world
that, “some ‘leading’ by the experimenters inevitably knowledge.
resulted from the selection of the information supplied” (p.
467). Moreover, Cyphert and Gant (1971) illustrated that a REFERENCES
statement in their study was initially rated below average.
However, Delphi subjects rated the statement above Adams, S. J. (2001). Projecting the next decade in safety
average after receiving false feedback. Therefore, Cyphert management: A Delphi technique study. Professional
and Gant (1971) concluded that the Delphi technique Safety, 46 (10), 26-29.
could, “be used to mold opinion as well as to collect [data]”
(p. 273). Indeed, “subtle pressure to conform with group Altschuld, J. W. (2003). Delphi technique. Lecture, Applied
ratings” was one of the major drawbacks in the Delphi evaluation design. The Ohio State University.
technique (Witkin & Altschuld, 1995, p. 188). Delphi Altschuld, J. W., & Thomas, P. M. (1991). Considerations
investigators need to be cognizant, exercise caution, and in the application of a modified scree test for Delphi
implement the proper safeguards in dealing with this issue. survey data. Evaluation Review, 15 (2), 179-188.

Published by ScholarWorks@UMass Amherst, 2007 5

 Practical Assessment, Research, and Evaluation, Vol. 12 [2007], Art. 10
Practical Assessment, Research & Evaluation, Vol 12, No 10 6
Hsu & Sandford, Delphi Technique

Anderson, D. H., & Schneider, I. E. (1993). Using the Alliance for Health, Physical Education, Recreation,
Delphi process to identify significant recreation and Dance, Spokane, WA.
research-based innovations. Journal of Park and
Hasson, F., Keeney, S., & McKenna, H. (2000). Research
Recreation Administration, 11 (1), 25-36.
guidelines for the Delphi survey technique. Journal of
Anglin, G. L. (1991). Instructional technology past, present and Advanced Nursing, 32 (4), 1008-1015.
future. Englewood, CO: Libraries Unlimited Inc.
Helmer, O., & Rescher, N. (1959). On the epistemology of
Brooks, K. W. (1979). Delphi technique: Expanding the inexact science. Management Science, 6, 25-53.
applications. North Central Association Quarterly, 54 (3),
Hill, K. Q., & Fowles, J. (1975). The methodological worth
377-385.
of the Delphi forecasting technique. Technological
Cunliffe, S. (2002). Forecasting risks in the tourism Forecasting and Social Change, 7, 179-192.
industry using the Delphi technique. Tourism, 50 (1),
Jacobs, J. M. (1996). Essential assessment criteria for physical
31-41.
education teacher education programs: A Delphi study.
Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The Unpublished doctoral dissertation, West Virginia
modified Delphi technique: A rotational modification. University, Morgantown.
Journal of Vocational and Technical Education, 15 (2), 1-10.
Jones, C. G. (1975). A Delphi evaluation of agreement
Cyphert, F. R., & Gant, W. L. (1971). The Delphi between organizations. In H. A. Linstone, & M. Turoff
technique: A case study. Phi Delta Kappan, 52, 272-273. (Eds.). The Delphi method: Techniques and applications (pp.
160-167). Reading, MA: Addison-Wesley Publishing
Dalkey, N. C. (1969). An experimental study of group
Company.
opinion. Futures, 1 (5), 408-426.
Jones, H., & Twiss, B. C. (1978). Forecasting technology for
Dalkey, N. C. (1972). The Delphi method: An
planning decision. London, UK: Macmillan Press Ltd.
experimental study of group opinion. In N. C. Dalkey,
D. L. Rourke, R. Lewis, & D. Snyder (Eds.). Studies in Judd, R. C. (1972). Use of Delphi methods in higher
the quality of life: Delphi and decision-making (pp. 13-54). education. Technological Forecasting and Social Change, 4 (2),
Lexington, MA: Lexington Books. 173-186.
Dalkey, N. C., & Helmer, O. (1963). An experimental Kaplan, L. M. (1971). The use of the Delphi method in
application of the Delphi method to the use of experts. organizational communication: A case study. Unpublished
Management Science, 9 (3), 458-467. master’s thesis, The Ohio State University, Columbus.
Dalkey, N. C., & Rourke, D. L. (1972). Experimental Kerlinger, F. N. (1973). Foundations of behavioral research. New
assessment of Delphi procedures with group value York: Holt, Rinehart, and Winston, Inc.
judgments. In N. C. Dalkey, D. L. Rourke, R. Lewis, &
Klee, A. J. (1972). The utilization of expert opinion in
D. Snyder (Eds.). Studies in the quality of life: Delphi and
decision-making. AICHE Journal, 18 (6), 1107-1115.
decision-making (pp. 55-83). Lexington, MA: Lexington
Books. Lindeman, C. A. (1981). Priorities within the health care system:
A Delphi survey. Kansas City, MO: American Nurses’
Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H.
Association.
(1975). Group techniques for program planning. Glenview,
IL: Scott, Foresman, and Co. Linstone, H. A., & Turoff, M. (1975). Introduction. In H.
A. Linstone, & M. Turoff (Eds.). The Delphi method:
Douglas, D. C. (1983). A comparative study of the effectiveness of
Techniques and applications (pp. 3-12). Reading, MA:
decision making processes which utilize the Delphi and leaderless
Addison-Wesley Publishing Company.
group methodologies. Unpublished doctoral dissertation,
The Ohio State University, Columbus. Ludlow, J. (1975). Delphi inquiries and knowledge
utilization. In H. A. Linstone, & M. Turoff (Eds.). The
Eckman, C. A. (1983). Development of an instrument to evaluate
Delphi method: Techniques and applications (pp. 102-123).
intercollegiate athletic coaches: A modified Delphi study.
Reading, MA: Addison-Wesley Publishing Company.
Unpublished doctoral dissertation, West Virginia
University, Morgantown. Ludwig, B. G. (1994). Internationalizing Extension: An
exploration of the characteristics evident in a state university
Green, P. J. (1982, March). The content of a college-level outdoor
Extension system that achieves internationalization.
leadership course. Paper presented at the Conference of
Unpublished doctoral dissertation, The Ohio State
the Northwest District Association for the American
University, Columbus.

https://scholarworks.umass.edu/pare/vol12/iss1/10 6
 Hsu and Sandford: The Delphi Technique: Making Sense of Consensus
Practical Assessment, Research & Evaluation, Vol 12, No 10 7
Hsu & Sandford, Delphi Technique

Ludwig, B. (1997). Predicting the future: Have you Scheibe, M., Skutsch, M., & Schofer, J. (1975).
considered using the Delphi methodology? Journal of Experiments in Delphi methodology. In H. A.
Extension, 35 (5), 1-4. Retrieved November 6, 2005 Linstone, & M. Turoff (Eds.). The Delphi method:
from http://www.joe.org/joe/1997october/tt2.html Techniques and applications (pp. 262-287). Reading, MA:
Addison-Wesley Publishing Company.
Marchant, E. W. (1988). Methodological problems
associated with the use of the Delphi technique: Taylor, R. E., & Judd, L. L. (1989). Delphi method applied
Some comments. Fire Technology, 24 (1), 59-62. to tourism. In S. Witt, & L. Moutinho, (Eds.). Tourism
marketing and management handbook. New York: Prentice
Martino, J. P. (1983). Technological forecasting for decision
Hall.
making. New York: North-Holland.
Turoff, M., & Hiltz, S. R. (1996). Computer based Delphi
Meyer, J. H. (1992). Rethinking the outlook of colleges whose roots
process. In M. Adler, & E. Ziglio (Eds.). Gazing into the
have been in agriculture. Davis, CA: University of
oracle: The Delphi method and its application to social policy and
California.
public health (pp. 56-88). London, UK: Jessica Kingsley
Miller, G. (2001). The development of indicators for Publishers.
sustainable tourism: Results of a Delphi survey of
Ulschak, F. L. (1983). Human resource development: The theory
tourism researchers. Tourism Management, 22, 351-362.
and practice of need assessment. Reston, VA: Reston
Miller, L. E. (2006, October). Determining what could/should Publishing Company, Inc.
be: The Delphi technique and its application. Paper presented
Weaver, W. T. (1971). The Delphi forecasting method. Phi
at the meeting of the 2006 annual meeting of the
Delta Kappan, 52 (5), 267-273.
Mid-Western Educational Research Association,
Columbus, Ohio. Witkin, B. R. (1984). Assessing needs in educational and social
programs. San Francisco, CA: Jossey-Bass Publishers.
Murray, W. F., & Jarman, B. O. (1987). Predicting future
trends in adult fitness using the Delphi approach. Witkin, B. R., & Altschuld, J. W. (1995). Planning and
Research Quarterly for Exercise and Sport, 58 (2), 124-131. conducting needs assessment: A practical guide. Thousand
Oaks, CA: Sage Publications, Inc.
Oh, K. H. (1974). Forecasting through hierarchical Delphi.
Unpublished doctoral dissertation, The Ohio State Young, S. J., & Jamieson, L. M. (2001). Delivery
University, Columbus. methodology of the Delphi: A comparison of two
approaches. Journal of Park and Recreation Administration,
Pill, J. (1971). The Delphi method: Substance, context, a
19 (1), 42-58.
critique and an annotated bibliography. Socio-Economic
Planning Science, 5, 57-71.

Citation
Hsu, Chia-Chien & Sandford, Brian A. (2007). The Delphi Technique: Making Sense of Consensus. Practical
Assessment Research & Evaluation, 12(10). Available online: http://pareonline.net/getvn.asp?v=12&n=10

Editors Note:
Another paper on the Delphi Technique that appeared in Practical Assessment Research & Evaluation is:

Yousuf, Muhammad Imran (2007). Using Experts’ Opinions through Delphi Technique. Practical Assessment
Research & Evaluation, 12(4). Available online: http://pareonline.net/getvn.asp?v=12&n=4 .

Authors
Chia-Chien Hsu
Post-doctoral Studies
The Ohio State University
393 Schrock Road
Worthington, OH 43085

Published by ScholarWorks@UMass Amherst, 2007 7

 Practical Assessment, Research, and Evaluation, Vol. 12 [2007], Art. 10
Practical Assessment, Research & Evaluation, Vol 12, No 10 8
Hsu & Sandford, Delphi Technique

Tel: (614) 885-0763

E-mail: hsu.127 [at] osu.edu

Brian A. Sandford
Assistant Professor
214 Willard Hall
Occupational Education
Oklahoma State University
Stillwater, Oklahoma 74074
405-744-3461
brian.sandford [at] okstate.edu

https://scholarworks.umass.edu/pare/vol12/iss1/10 8