PARTICIPATORY METHODS TOOLKIT

A practitioner’s manual

Delphi
Colophon

Participatory Methods Toolkit. A practitioner’s manual

Method: Delphi

This is an extract of the publication ‘Participatory Methods Toolkit. A practitioner’s manual’, a joint publication of the
King Baudouin Foundation and the Flemish Institute for Science and Technology Assessment (viWTA).

The full version of the manual includes:

- Introduction about participatory methods
- General guidelines and tips for participatory methods
- Complete description of 13 participatory methods: 21st Century Town Meeting® ; Charrette ; Citizens Jury ; Consensus
Conference ; Deliberative Polling® ; Delphi ; Expert Panel ; Focus Group ; Participatory Assessment, Monitoring and
Evaluation ; Planning Cell ; Scenario Building Exercise ; Technology Festival ; The World Café
- Brief descriptions of 50 methods and techniques

All publications are available free of charge, online via www.kbs-frb.be or www.viWTA.be.

Editors
Stef Steyaert (viWTA) and Hervé Lisoir (King Baudouin Foundation)

Author
Nikki Slocum (United Nations University – Comparative Regional Integration Studies)

Layout and printing

Belgian Advertising (B.AD)

Legal depot
D/2005/2893/19

ISBN
90-5130-506-0

With the support of the Belgian National Lottery

September 2005
Delphi
(and Policy Delphi, Public Delphi, Delphi Conference, Delphi face-to-face)

I. DEFINITION

Delphi involves an iterative survey of experts. Each participant completes a questionnaire and is then given feedback on
the whole set of responses. With this information in hand, (s)he then fills in the questionnaire again, this time providing
explanations for any views they hold that were significantly divergent from the viewpoints of the others participants.
The explanations serve as useful intelligence for others. In addition, (s)he may change his/her opinion, based upon
his/her evaluation of new information provided by other participants. This process is repeated as many times as is
useful. The idea is that the entire group can weigh dissenting views that are based on privileged or rare information.
Thus, in most Delphi processes the mount of consensus increases from round to round.

While traditionally conducted via mail, other variations of Delphi can be conducted online or face-to-face. In the original
Delphi process, the key characteristics of this method were (1) structuring of information flow, (2) feedback to the
participants and (3) anonymity for the participants. In a face-to-face Delphi, the anonymity is eliminated. Another

DELPHI
variation of the Delphi is the ‘Policy Delphi’, the main goal of which is to expose all the different options and opinions
regarding an issue and the principal pro and con arguments for these positions.

II. WHEN TO USE

A dialectical process, Delphi was designed to provide the benefits of a pooling and exchange of opinions, so that
respondents can learn from each others’ views, without the sort of undue influence likely in conventional face-to-face
settings (which are typically dominated by the people who talk the loudest or have most prestige). The technique allows
experts to deal systematically with a complex problem. From round to round the relevant information is shared, further
educating the panel members. Recommendations can thus be made on the basis of more complete information.

Usually one or more of the following properties of the application leads to the need or usefulness of employing Delphi:
The problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on
a collective basis.
The individuals needed to contribute to the examination of a broad or complex problem have no history of
adequate communication and may represent diverse backgrounds with respect to experience or expertise.
More individuals are needed than can effectively interact in a face-to-face exchange (except through the
face-to-face Delphi’s shuttle process between plenary and sub-groups).
Time and cost make frequent group meetings infeasible.
The efficiency of face-to-face meetings can be increased by a supplemental group communication process.

109

1 The Delphi method was originally developed at the RAND Corporation by Olaf Helmer and Norman Dalkey.
 Disagreements among individuals are so severe or politically unpalatable that the communication process must
be refereed and/or anonymity assured.
heterogeneity of the participants must be preserved to assure validity of the results, i.e. avoidance of
domination by quantity or by strength of personality.

The Policy Delphi serves any one or a combination of the following objectives:
to ensure that all possible options have been put on the table for consideration
to estimate the impact and consequences of any particular option
to examine and estimate the acceptability of any particular option.

In general, the Delphi method was invented in an attempt to overcome various social-psychological challenges
associated with committee processes, including:
the domineering personality or outspoken individual that takes over the committee process
the unwillingness of individuals to take a position on an issue before all the facts are in or before it is known
which way the majority is headed
the difficulty of publicly contradicting individuals in higher positions
the unwillingness to abandon a position once it is publicly taken
the fear of bringing up an uncertain idea that might turn out to be undesirable and result in a loss of face.
DELPHI

III. PROCEDURE

A. Overview

Refer to the Delphi Method Flowchart below for a graphic overview of this method.

Delphis – whether conventional, real-time on computer or face-to-face – usually undergo four phases. In the first phase
the subject under discussion is explored and each individual contributes the information (s)he feels is pertinent to the
issue. In the second phase an overview is reached on how the group views the issue, for example, where there is
dis/agreement over what is meant by relative terms such as ‘feasible’, ‘important’, ‘desirable’, etc. If there is significant
disagreement, then this is explored in the third phase in order to illuminate the reasons for the differences and evaluate
them. The fourth phase entails a final evaluation that occurs when all previously gathered information has been initially
analysed and the evaluations have been fed back for reconsideration.

In the following sections a step-by-step description of conventional Delphi is presented, followed by a description of the
variations that constitute the Policy Delphi. Finally, the steps of the Delphi Conference are presented.

110
Delphi Method Flowchart 1

Start

Problem
definition

Select panel members based on

the expertise required

Prepare and distribute questionnaire

Prepare and
distribute 2nd/3rd Analyse questionnaire responses
questionnaire

Has

DELPHI
No consensus
been
reached?

Yes

Provide requested information and

summary of responses

Develop final report

111

1 From http://www.ryerson.ca/~mjoppe/ResearchProcess/841TheDelphiMethod.htm
 B. Realisation

1. PERSONNEL AND TASKS

(a) Organisational Team

The tasks of the organisational team are as follows:

Develop the questionnaires.
Identify and recruit experts.
Distribute questionnaires.
Analyse the comments and give feedback to the experts after each round.
Write the final report.

(b) Experts

Complete the questionnaires.

If the Delphi is face-to-face, attend the scheduled events.
DELPHI

(c) Moderator(s)

If the Delphi is conducted face-to-face, one or two moderators will be required to facilitate the process.

2. CONVENTIONAL (PEN-AND-PAPER) DELPHI

The procedure of the original Delphi can be described in the following steps. (For a graphic overview, please refer to the
Delphi Method Flowchart.)

(1) Form a team to undertake and monitor a Delphi on a given subject.

(2) Select and recruit panel(s) to participate in the exercise.

Customarily the panellists are experts in the area to be investigated. Some literature suggests that while the panellists
should be well informed about the topic, a high degree of expertise is not necessary. Of course, the required level of
expertise will depend upon the specific topic and questions being addressed. The number of panellists varies greatly
between Delphis, but should include a very minimum of four persons per panel.

TIP: The panellists should be assured that they are participating in an exercise that involves a peer group. Therefore,
in the letter of invitation indicate the types of backgrounds reflected in the participant group.

112
(3) Develop the first-round Delphi questionnaire.
A month or more is needed to develop the first-round questionnaire. Ideally, the questions posed should be specific
enough to eliminate most irrelevant information, but otherwise place as few constraints on the information as possible.
In addition to the questionnaire, a factual summary of background material is usually supplied. In some cases single or
multiple sets of scenarios are provided that specify certain items that the respondents are to assume as given for the
purpose of evaluating the issues. (Typically these scenarios deal with aspects like future economic conditions, such as
the rate of inflation.)

Often, various alternatives are presented along with rating scales, which give the respondents an opportunity to
quantify their preferences. An example of a commonly used scale follows.

1 * 2 * 3 * 4 * 5 * 6
strongly disagree somewhat somewhat agree strongly
disagree disagree agree agree

If the rating procedure is used, take care not to use compound statements (such as ‘Do you think y, if x….’; rather break
down such statements into two simple statements (e.g. ‘Do you think x?’ and ‘Do you think y?’).

DELPHI
TIP: If new to Delphi, the respondents will often respond with compound and lengthy comments. It is useful to
provide some examples of the form you would like their answers to take, in terms of being short, specific and s
ingular in nature. Allow the panellists to suggest changes in the wording of items and introduce them as new
items. Policy issues are often very sensitive to precise wording.

TIP: Sometimes it is appropriate to introduce a set of alternative assumptions making up scenarios and let the
respondents form a group scenario by voting on the validity of each.

(4) Test the questionnaire for proper wording (e.g. ambiguities, vagueness).

Each questionnaire should be pre-tested with people who have not been involved in the design. Identify any items that
are phrased in a confusing manner and revise them.

(5) Transmit the first questionnaires to the panellists.

(6) Analyse the first-round replies.

(7) Prepare the second-round questionnaires (and possible testing).

113
 In this round the discrepancies between the participants’ views are brought to the fore (but still kept anonymous).
Participants are asked to try to explain the differences between their views and others’, providing their reasoning and
any influential information to which the others may not be privy. In each round such information and reasoning are
shared with the other participants (still maintaining anonymity).

(8) Transmit the second-round questionnaires to the panellists.

TIP: When asking for re-votes on an item, show the individuals their original votes and provide them with two
copies of the questionnaire so that they may retain one for later reference or do draft work.

(9) Analyse the second-round replies.

Steps 7 – 9 are reiterated as long as desired or necessary to achieve stability in the results.

(10) Prepare a report by the analysis team to present the conclusions of the exercise.

TIP: It is very important that all of the participants understand the aim of the Delphi exercise; otherwise they may
answer inappropriately or become frustrated and lose interest.
DELPHI

3. POLICY DELPHI 2

Delphi, as it originally was introduced and practised, tended to deal with technical topics and seek a consensus among
a homogeneous group of experts. In contrast, the Policy Delphi is employed to generate the strongest possible opposing
views on the potential resolutions of a major policy issue. A policy issue can be seen as an issue for which there are
no ‘experts’, only informed advocates and referees. An expert or analyst may contribute a quantifiable or analytical
estimation of some effect resulting from a particular resolution of a policy issue, but it is unlikely that a clear-cut
(to all concerned) resolution of a policy issue will result from such an analysis. The expert becomes an advocate for
effectiveness or efficiency and must compete with the advocates for concerned interest groups within the society.

The Policy Delphi rests on the premise that the decision maker is not interested in having a group generate his/her
decision, but rather in having an informed group present all the options and supporting evidence for his/her
consideration. Therefore, the Policy Delphi is a tool for the analysis of policy issues and not a mechanism for making a
decision. Generating a consensus is not the prime objective. The structure of the communication process, as well as the
choice of the respondent group, may make achieving consensus on a particular resolution very unlikely. In fact, in some
cases the sponsor may even request a design that inhibits consensus formulation.

The procedure for the Policy Delphi is the same as for the traditional Delphi, but the survey questions posed to the
panellists will aim more at exploring all possibilities, opinions and reasons rather than at achieving consensus.

114

2 From Turoff, M. (Internet). The Policy Delphi. In The Delphi Method: Techniques and Applications, pp. 80 – 96.
http://www.is.njit.edu/pubs/delphibook/ch3b1.html
The following questions should guide the planning and implementation phases of a Policy Delphi:
Formulation of the issues. What is the issue that really should be under consideration? How should it be stated?
Exposing the options. Given the issue, what are the policy options available?
Determining initial positions on the issues. Which are the ones everyone already agrees upon and which are the
unimportant ones to be discarded? Which are the ones exhibiting disagreement among the respondents?
Exploring and obtaining the reasons for disagreements. What underlying assumptions, views or facts are being
used by the individuals to support their respective positions?
Evaluating the underlying reasons. How does the group view the separate arguments used to defend various
positions, and how do they compare to one another on a relative basis?
Re-evaluating the options. Re-evaluation is based upon the views of the underlying ‘evidence’ and the
assessment of its relevance to each position taken.

In principle, this process would require five rounds in a paper-and-pencil Delphi procedure. However, in practice most
Delphis on policy try to maintain a three-round or four-round limit by doing the following:
The monitor team devotes considerable time to carefully pre-formulating the issues.
The questionnaires provide a list(s) of an initial range of options but allow for the panellists to add to the list(s).
The panellists are asked for their positions on an item and their underlying assumptions in the first round.

TIP: It has been suggested that the best vehicle for a policy Delphi is a computerised version of the process, in

DELPHI
which the round structure disappears and each of these phases is carried through in a continuous process.

In a Policy Delphi it is necessary that informed people, representative of the many sides of the issues under examination,
are chosen as participants. The initial design must ensure that all of the ‘obvious’ questions and sub-issues have been
included and that the participants are being asked to supply the more subtle aspects of the problem. Thus, the monitors
must understand the subject well enough to recognise the implications of the participants’ abbreviated remarks.

TIP: In some cases the participants may over-concentrate their efforts on some issues to the detriment of others.
This may occur because the group is not as diversified as the total scope of the exercise should be. With proper
knowledge of the subject material, the design team can stimulate consideration of the neglected issues by
interjecting comments in the summaries for consideration by the group. It is a matter of integrity to use this
privilege sparingly to stimulate dialogue on all sides of an issue and not to sway the participants toward one
particular perspective.

4. DELPHI CONFERENCE (ALSO FACE-TO-FACE OR GROUP DELPHI)

This face-to-face group version of Delphi allows for more discussion and debate and takes less time than the traditional
version, but the participants forego anonymity.

115
 Recruit a design-monitor team, group facilitator and an assistant to undertake and monitor the group Delphi.
The design-monitor team should consist of at least two professionals, so that one can check the other. Ideally,
one should be knowledgeable in the issue at hand and the other should have editorial talents.
A management team must decide on (and usually narrow down) the topic(s), as well as the number of Delphi
panels that will be conducted on the topic(s).
Decide on the date that the panel will be held. One full day will allow for several rounds of the process, in
addressing one question. More time will be required to address very complex issues or more than one major
question.
Reserve a location for the workshop. One large room to accommodate all panellists is required. It would be ideal
to have access to a few smaller rooms, in which the sub-groups can do their work.
Select and recruit participants for each panel.
Customarily the panellists are experts in the area to be investigated.
Reserve accommodation for those who require them. Make catering arrangements.
Development of the Delphi questionnaire.
Individual question replies.
Working individually and without discussion, each participant responds to the question.
Small groups.
Participants divide into sub-groups of ‘similar’ people and prepare a list of information, arranged in order of
importance. Here ‘similar’ refers to their views on the topic being addressed. The purpose of having homogenous
DELPHI

sub-groups is to help ensure that all information that is important to a particular perspective or interest group
will reach the plenary list.
Plenary group.
Gather the important items from each group and list them where everyone can see them (newsprint, flipcharts,
etc.). To do this, ask each group in turn to contribute the most important item on their list that has not already
been added to the plenary group’s list.
Plenary vote.
A multiple-vote procedure is used to rank the items from most to least important. A natural cut-off point is
chosen between items with high scores and those with low scores. Somewhere between six and nine items are
appropriate for most topics.
Individual changes.
Each individual considers what changes (s)he wishes to make to his/her small-group list after having seen the
plenary list.
Small groups.
Members compare the list of top items on their small-group list to those on the plenary list. Where the small
group list differs from the plenary list, the small group has two options. It can either change its list to conform
more closely to the plenary list or it can develop evidence for changing the plenary list more in the direction of
its list. This is done as follows: add to the small-group list the items from the plenary list that the small group
previously omitted but is ready to accept. Prepare a brief report supporting any of the top items from the
small-group list that the group believes should be added to the plenary list. NOTE: A time limit for the report,
of one minute for example, should be enforced. The purpose of this report is not to persuade others to adapt

116
 their point of view, but to present evidence that the group thinks others may have overlooked. Each small group
then documents its revised list on one sheet of newsprint and its evidence (in note form) on another.
Small-group reports to the plenary:
The revised lists of the small groups are displayed without comment. Each group in turn displays its sheet of
evidence and explains it briefly. Each group report is followed by a brief session of questions for clarification
only. Strict time limits are reinforced.
Plenary consensus development: return to step 9 and repeat the cycle until consensus emerges.
Time constraints may require a fixed number of cycles. Consensus can be increased by having two rounds of
voting, instead of one, at step 9.

For additional in-depth information on the philosophy behind the Delphi method, as well as various applications,
refer to: http://www.is.njit.edu/pubs/delphibook/

IV. RESOURCE CONSIDERATIONS (TIME, BUDGET)

A. Timing

The following table presents a general weekly schedule for an online version of Delphi. However, this is just to provide

DELPHI
a general guideline. It should be noted that schedules will vary greatly and that face-to-face Delphis will require
significantly more time than those conducted online.

General Schedule for Delphi-online Week

1. Preparation of the Delphi project 1&2
General preparation
Compose expert panel
Email addresses participants
Make the collaboration more concrete
Develop accompanying texts

2. Start up and configuration of Delphi online system 1&2

Questions first round
Develop invitation-email;
Call participants for formal consent
Create first round in Delphi online system (users, passwords, texts, …)

3. First question round Delphi 3&4

Call the nonrespondents 5

117
 4. Treatment of results of first round and start up second round 6&7
Data-analysis: reduce the answers to the open questions to a more limited
set without losing content
Introduce the system
Invite participants by email

5. Second question round Delphi 8&9

Call the nonrespondents 10

6. Treatment of results of second round and start up third round 11&12

Data-analysis: response to closed questions and arguments
Invite participants by email

7. Third question round Delphi 13&14

Call the nonrespondents 15

8. Treatment of results of third round

Intermediate report 16&17
DELPHI

9. Project management continuous

Elaborate project plan
Project meetings

10.Final report Delphi project 18&19

B. Budget

The following items listed are the main budgetary items in a Delphi:
Personnel
- organisational team
- stipends for experts
- moderator(s)
Travel
- only for Face-to-Face Delphi: travel for experts and moderator(s)
Accommodation
- only for Face-to-Face Delphi: accommodation for experts and moderator(s)
Food
- only for Face-to-Face Delphi: meals for experts and moderator(s)
Recruitment and promotion
- recruitment of experts

118
 Communications
- printing and postage costs for surveys (if done by traditional mail)
- printing of draft and final report and dissemination
Facilities
- only for Face-to-Face Delphi: location for event
Materials and Supplies
(These depend upon the type of Delphi conducted.)

V. ADDITIONAL BEST PRACTICES AND POTENTIAL PITFALLS

Some common reasons for the ‘failure’ of a Delphi are:

the imposition of the monitor’s views and preconceptions of a problem upon the panellists by over-specifying
the structure of the Delphi and thus not allowing for the contribution of other perspectives related to the
problem
the assumption that Delphi can be a surrogate for all other human communications in a given situation
poor techniques of summarising and presenting the group response and thus failing to ensure common
interpretations of the evaluation scales utilised in the exercise
ignoring, rather than exploring, disagreements so that discouraged dissenters drop out and an artificial

DELPHI
consensus is generated
underestimating the demanding nature of a Delphi; failing to recognise the respondents as consultants and
properly compensate them for their time if the Delphi is not an integral part of their job function.

For a successful Delphi, it is important to:

carefully select the group of respondents/panellists
adapt the Delphi design to your particular application
assure the honesty and lack of bias in the monitoring team
assure a common language and logic, particularly if participants come from diverse cultural backgrounds.

References and Resources

Brown, B. (1968) The Delphi Process: A Methodology Used for the Elicitation of Opinions of Experts. Rand Document
No. P-3925. www.rand.org

Dalkey, N. and Helmer-Hirschberg, O. (1962) An Experimental Application of the Delphi Method to the Use of Experts.
RM-727-PR. www.rand.org.

Dick, B. (2000) Delphi face to face [On line]. Available at http://www.scu.edu.au/schools/gcm/ar/arp/delphi.html

Glenn, J. (Ed.) Futures Research Methodology. Version 1.0. AC/UNU The Millennium Project.

119
 Linstone, H. and Turoff, M. (2002) Introduction. In H. Linstone and M. Turoff (Eds.), The Delphi Method: Techniques and
Applications. pp. 3 – 12. http://www.is.njit.edu/pubs/delphibook/

Practical Guide to Regional Foresight in the United Kingdom.

Turoff, M. (Internet). The Policy Delphi. In The Delphi Method: Techniques and Applications, pp. 80 – 96.
http://www.is.njit.edu/pubs/delphibook/ch3b1.html

Numerous publications on the Delphi method and its applications, of which only a few are given below, can be
found on the Rand Corporation website, www.rand.org.

Sackman, H. (1974). Delphi Assessment: Expert Opinion, Forecasting, and Group Process. Rand Document No.
R-1283-PR. www.rand.org.
DELPHI

120