AC/UNU Millennium Project

Futures Research Methodology V2.0

3.

THE DELPHI METHOD

By

Theodore J. Gordon

I. History of the Method II. Description of the Method III. How To Do It IV. Strengths and Weaknesses of the Method V. Frontiers of the Method VI. Samples of Applications Bibliography

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

Acknowledgments Some contents of this report have been taken, in some cases verbatim, from internal papers of The Futures Group with their permission. These papers were written by John G. Stover, Theodore J. Gordon, and others describing the Delphi method and its applications. The managing editor also gratefully acknowledges the contributions of reviewers of the draft of this paper: Dr. Ian Miles of The Programme of Policy Research in Engineering Science and Technology, in the United Kingdom; Dr. Brian Free of Futures Environment Council of Alberta Canada; Dr. Mika Mannermaa of Futures Research Centre at Turku School of Economics, Turku Finland; Dr. Harold A. Linstone of Portland State University, United States; and Dr. Peter Bishop of the University of Houston, in the United States. And finally, special thanks to Elizabeth Florescu and Neda Zawahri for project support, Barry Bluestein for research and computer operations and Sheila Harty for editing.

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

I. HISTORY OF THE METHOD

The modern renaissance of futures research began with the Delphi technique at RAND, the Santa Monica, California, "think tank" in the early 1960s. The questions of Rand thinkers, at the time, primarily dealt with the military potential of future technology and potential political issues and their resolution. The forecasting approaches that could be used in such applications were quite limited and included simulation gaming (individuals acting out the parts of nations or political factions) and genius forecasting (a single expert or expert panel addressing the issues of concern). Quantitative simulation modeling was quite primitive, and computers, which would ultimately make such quantitative techniques practical, were a decade away. The RAND researchers explored the use of expert panels to address forecasting issues. Their reasoning went something like this: experts, particularly when they agree, are more likely than nonexperts to be correct about questions in their field. However, they found that bringing experts together in a conference room introduces factors that may have little to do with the issue at hand. For example, the loudest voice rather than the soundest argument may carry the day; or, a person may be reluctant to abandon a previously stated opinion in front of his peers. As with normal thinkers, the give-and-take of such face-to-face confrontations often gets in the way of a true debate. One of the little known in-house research projects undertaken by RAND at the time involved combining opinions of horse-racing handicappers. These people, after all, are supposedly experts in their field. Furthermore, their opinions about the future (the outcome of horse races) are published daily and can be checked against reality within 24 hours. So a project was implemented to determine just how to combine horse-race forecasts by different experts to improve the likelihood that the composite opinion was better than any single expert. The work on the Delphi method followed. Olaf Helmer, Nicholas Rescher, Norman Dalkey, and others at RAND developed the Delphi method, which was designed to remove conference room impediments to a true expert consensus. The name, of course, was drawn (humorously, they thought) from the site of the Greek oracle at Delphi where necromancers foretold the future using hallucinogenic vapors and animal entrails. They began from a philosophical base and asked initially, "just how much could be known about the future?" (Helmer and Rescher, 1959) The Delphi method was designed to encourage a true debate, independent of personalities. Anonymity was required in the sense that no one knew who else was participating. Further, to eliminate the force of oratory and pedagogy, the reasons given for extreme opinions were synthesized by the researchers to give them all equal "weight" and then fed back to the group as a whole for further analysis. These aspects, anonymity and feedback, represent the two irreducible elements of the Delphi method. This general approach has been used thousands of times since the first published Delphi study, Report on a Long-Range Forecast by Gordon and Helmer. This all-time RAND "best seller," published in 1964, contained forecasts of scientific and technological breakthroughs through 2000 and beyond; the 82 panelists who contributed included Isaac Asimov, Arthur Clarke, Bertrand de Jouvenel, Ithiel de Sola Pool, Dennis Gabor, Peter Goldmark, Harold Guetzkow, and

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

William Pickering, to name a few. (Gordon and Helmer, 1964) The temptation to review all of the forecasts with the advantage of hindsight is great (one such review was made by Amant in 1970), but a cursory review shows many forecasts that were on target, such as: economically useful desalination of sea water oral contraceptives advent of ultra light materials automated language translation transplanting organs more reliable weather forecasts centralized data banks artificial organs X Ray lasers psychotropic drugs self replicating molecules synthetic protein feasibility of control over hereditary defects There were big misses, too, including: controlled thermo nuclear power biochemical general immunization limited weather control world population by 2000 less than 6 billion manned landing on Mars Nevertheless, the study and, more generally, RAND's interest in developing systematic methods for forecasting apparently legitimized the field. Since this first Delphi, literally thousands have been performed on topics as wide ranging as the future of religion and the family to space exploration. Several of these are described in more detail in Section 5 of this report.

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

II. DESCRIPTION OF THE METHOD

This description will be illustrated by a hypothetical example: suppose we want to establish a forecast for the date by which a manned Mars landing would occur. With the Delphi method, experts from the required disciplines are first identified and asked to participate in the inquiry. For this example, the experts might include rockets scientists, geologists, and bioscientists who are experts on the planet, planners from space agencies that might be involved in such a project, and others who expressed an opinion that manned landings might be a bad idea. During the initial contact, the nominated persons are told about the Delphi and invited to participate. They are assured of anonymity in the sense that none of their statements will be attributed to them by name. The questions are refined by the researchers and pursued through a number of sequential questionnaires. In the first questionnaire, participants might be asked to provide their judgment on a date at which a manned landing might take place. The analysis would identify the range of opinions about the date. In a second questionnaire, the range would be presented to the group, and persons holding opinions at the extremes of the range would be asked to reassess their opinion in view of the group's range and provide reasons for their positions. For example, a reason for a late manned landing might be that robot landers can do all that men can do, so little reason exists to spend the money for human on-site involvement. These reasons would be synthesized by the researchers at the end of round two; the synthesized reasons would form the basis for the third questionnaire. In this third questionnaire, the new group judgment on a date would be presented to the participants, along with reasons for the extreme opinions. Each member of the group would be asked to reassess his or her position in view of the reasons presented. They might also be asked to refute, if appropriate, the extreme reasons with any facts at their disposal. For example, someone might attempt to refute the robot argument by saying that human presence will be required to perform a certain class of engineering activity or to construct habitats for later colonization. In a fourth and final round, these arguments would be presented, along with the evolving group consensus, and a reassessment requested. In a sense, the Delphi method is a controlled debate. The reasons for extreme opinions are made explicit, fed back coolly and without anger or rancor. More often than not, experts groups move toward consensus; but even when this does not occur, the reasons for disparate positions become crystal clear. Planners reviewing this material can make judgments based on these reasons and their own knowledge and goals. Because the number of respondents is usually small, Delphis do not (and are not intended to) produce statistically significant results; in other words, the results provided by any panel do not predict the response of a larger population or even a different Delphi panel. They represent the synthesis of opinion of the particular group, no more, no less. The value of the Delphi method rests with the ideas it generates, both those that evoke consensus and those that do not. The arguments for the extreme positions also represent a useful product.

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

Questions included in a Delphi may be of any sort that involve judgment, including, for example, the size of a future market, whether or not the CEO should receive a raise, or the proper policy to achieve a goal. In planning applications, the questions generally are of three types. forecasts on the occurrence of future developments. Forecasts of future developments call for answers about when an event is expected to occur or about the future value of some parameter; desirability of some future state. Questions dealing with desirability ask for judgments about whether an event ought to occur, and the basis for the recommendation; and the means for achieving or avoiding a future state. Questions dealing with policy involve the traditional reporter's questions about implementation: who, what, when, where, and how much? But to this set we must add: to what end. In other words, questions about policy ought to be linked closely to the objectives sought and the likelihood that any policy will, in fact, accomplish its intended goals. These three types of questions may require different kinds of experts. The likelihood questions may involve hands-on experience and intimate knowledge of the frontiers of research. The desirability questions may involve a moral, political, or social dimension quite distinct from the disciplinary expertise involved in judging likelihood. The policy question may involve knowledge of the art of the possible. In some modern applications of Delphi: 1. The questions relate to the value of independent variables that are used in quantitative simulation models. In this application, a consensus is not required; rather, if disagreement exists about the value of any variable, the extremes can be tested in quantitative models to determine whether or not the difference has any important significance. 2. In-depth interviews with experts have been used at The Futures Group and elsewhere with great success as an alternative to questionnaires. In this approach, the same kinds of experts are first identified, invited to participate, assured of their anonymity, and, in most instances, promised a report based on the interview sequence. Appointments are made at the convenience of the interviewees. Interview protocols are prepared and tested to elicit judgments. High-level staff members, familiar with the study's objectives, act as interviewers. Feedback can be introduced if two rounds of interviews are employed; however, single-round studies are used more frequently. In these, "feed-forward" is often employed, presenting to respondents information about emerging consensus derived from the prior interviews. True, this process introduces differences among the various interviews, but remember that the exercise is not designed to be statistically significant but rather to elicit ideas that can be important to subsequent analyses. Expert in-depth interviews are an excellent means of obtaining such ideas. An advantage of one-on-one interviews is that they provide flexibility, which is absent in questionnaires. For example, an interview provides the opportunity to probe the reasons behind

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

the forecasts, to search for biases in the forecasts, and to follow up on unexpected hints dropped by the interviewees. 3. For some applications, group meetings among experts have now become practical. Delphi had its birth in concern about spurious factors that intrude in face-to-face meetings among experts. New technology can minimize these factors. Some Delphi-like studies have been performed on line (Shota, 1993); but these are generally cumbersome, and access to the appropriate technology limits the selection of participants. Conference room voting machines have proved useful. The Consensor (Applied Future, Westport, CT) built and rents one such machine. Others include OptionFinder (Minneapolis) and the PC Voter (The Futures Group). Typically, these machines provide each participant with a small terminal, which is connected through a serial circuit to a personal computer. In the PC Voter, each small terminal has two knobs. The first knob allows the user to provide quantitative judgments about a question posed by the meeting's moderator for example: "What is the probability that limited weather control will be available by 2020?" Using the second knob, participants can provide a percentage assessment of confidence in their answer. The computer's software integrates the answers of the experts at the meeting, discounting those who have low confidence in their answers, and provides on a display screen a histogram showing the distribution of the group's opinions. Anonymity is preserved because the inputs are private and unseen by others; the display provides feedback. It is true that anonymity is lost in any discussion of the results, but discussion is an option of the participants. This approach has been found quite useful when quick results are needed and the issue is, "hot"; that is, the topic of the study is likely to evoke strong emotional responses. An example of a "hot" issue is the future of executive compensation among the executives and compensation committee members of a corporation.

III. HOW TO DO IT
The key to a successful Delphi study lies in the selection of participants. Since the results of a Delphi depend on the knowledge and cooperation of the panelists, persons who are likely to contribute valuable ideas are essential to include. In a statistically based study, such as a public opinion poll, participants are assumed to be representative of a larger population; in Delphi, non representative, knowledgeable persons are needed. So the first problem is how to select potential participants. Knowledgeable persons are usually identified through literature searches for who has published on the subject under study, recommendations from institutions (e.g., The World Future Society) and other experts in a process known as "daisy chaining." Literature searches necessarily result in lists of people who have published on the topic of interest; this approach misses people who may have something to contribute but have not published. Recommendations from institutions suffer from the same shortcoming: the recommendations are limited to only those who are known to the institutions. "Daisy chaining" has the potential disadvantage of identifying cliques. One possibility that helps ensure that the required skills are represented is to form a matrix in

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

which the required skills are listed. In the case of the Mars landing question, for example, the columns might be: solar system planetary geology, rocket system design, robotics, data collection, and telemetry, etc. The rows of the matrix consist of the names of the prospective participants. The cells are checked to indicate the "coverage" that the nominees are expected to provide. As for "unknown" people who are outside of the normal lines of communication but who may be able to contribute new and innovative ideas, here are some suggestions: use bulletins boards so that contributors who have something to say in this informal environment may identify themselves; get recommendations from university professors about bright students; and advertise for participants and qualified applicants through preliminary Delphi's. Most studies use panels of 15 to 35 people. The length of the list should anticipate an acceptance rate between 35 and 75 percent. Once the list of nominees is formed, each person should be contacted individually. Form letters should not be used. The initial contact may be by telephone, but letters should confirm the invitation. These letters should contain a description of the project, its objectives, the number of rounds to be included (or the time commitment anticipated), the promise of anonymity, and, if appropriate, a confirmation of the panelist's acceptance. The next step is to formulate the questions. The questions must be sharp and answerable. A small panel might be used to help formulate the questions. For example, say our question is: What effective and practical means exist to prevent proliferation of nuclear weapons? The question would be sent to a small expert panel (10 persons or so) working with the research team. Essay-type answers would be permitted at this stage. The answers might include: Military intervention, greatly enhanced customs operations, new nonproliferation treaties, space surveillance, etc. The research staff would collate the answers and form a nonessay questionnaire. The question might be: The following means have been suggested for preventing nuclear proliferation. Please rate each of these in terms of effectiveness and practicality. Add other suggestions to the list if you have an idea that might be as effective and practical as those listed. If you think any of these means are extremely good or, at the other extreme, counterproductive, please provide your reasons. This nonessay question would serve as the basis for the first questionnaire. The questionnaire would be tested, perhaps using the small advisory panel. The testing would include actually

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

filling in the questionnaire. This test is designed to find flaws in the way the questions are asked and to find any lurking possibilities for misinterpretation. Once qualified in this way, the questionnaire would be sent to the participants. The cover letter would remind the participants about the objectives of the study, establish the schedule for the response, and include the return address. The media that are practical for transmitting the questionnaires and responses are airmail, fax, and e-mail. Our experience indicates that a response rate from 40 to 75 percent of the participants can be anticipated. The turn around time is on the order of weeks, no matter what the medium of communications. In this example, respondents would provide two numbers for each option depicting their judgments about effectiveness and practicality. They would be asked to provide reasons for their judgments if they saw an alternative as particularly promising or potentially counterproductive. The research staff would collate the results. A feedback round would be used to present the results, the reasons for the extreme positions, and a call for reassessment. The reasons, in Delphi style, would be placed in front of the participants who answered the first round. They would be asked to reconsider their former answers in view of the reasons for the extreme opinions. Questions that have quantitative answers or "check the box" responses are always easier to use. When the questions are in this form, they can be collated using spreadsheet software. However, it is not always possible or desirable to ask questions in this form. For example, we may want suggestions from the panel about policies for diminishing population growth: here, the essence is in the detail that the panel might provide. In addition, even when quantitative rather than narrative answers are called for, notes from the panelists may contain the most valuable information: references to other people, past experiences, uncertainty, pointing to data, etc. For this reason, all questionnaires should be read and digested by senior members of the study team. Phrasing of questions is important. A common mistake is to include two events in the same question. Example: "When will bicycles be made mostly of plastics and used in urban transportation?" Differences in the way people answer this question may stem not only from their perceptions about future uses of plastic in bicycles, but also from differences in their beliefs about when bicycles will likely be used for urban transportation. Even subtle changes in wording may affect answers. For this reason, it is desirable to conduct a test of the instrument, not only by obtaining answers from a test group, but also by discussing with that group their interpretations of the questions. The data from a Delphi can be displayed in several ways. The group judgment should be based on the median rather than the mean, since single extreme answers can "pull" the mean unrealistically. Furthermore, it is incumbent on the analyst to show the spread of opinion, which can be done by showing the interquartile range (the range that contains the answers of 50 percent of the respondents). An example is shown below:

The Delphi Method

AC/UNU Millennium Project

Futures Research Methodology V2.0

The Delphi Method

10

AC/UNU Millennium Project

Futures Research Methodology V2.0

IV STRENGTHS AND WEAKNESSES OF THE METHOD

Delphi studies are difficult to perform well. A great deal of attention must be given to the choice of participants; the questionnaires must be meticulously prepared and tested to avoid ambiguity. Multi-round studies require a great deal of time; inevitably, some participants will drop out during the process. Therefore, other techniques for collecting expert judgment are welcome. Some of these techniques were described in Section 3. The primary strength of Delphi is its ability to explore, coolly and objectively, issues that require judgment; a weakness of Delphi is the ease with which questions can be asked for which better techniques exist. At one extreme are questions about the future for which factual answers exist and thus require minimal judgment. For example: "Given continued trends in immigration, birth, and death rates, how many people over the age of 70 will be in the United States in 2010?" This question has a factual answer that can and should be computed. Factual questions should not be candidates for Delphi. The question of whether or not current population trends will continue is a matter of judgment and, therefore, appropriate. In short, Delphi is a powerful technique when used to seek answers to appropriate questions. Suppose, for example, that we want to forecast the future size of the market for a given product. A Delphi study might involve sales and marketing personnel, retailers, and experts in consumer preferences. These people might be asked for a direct estimate; to do so would require that they somehow integrate all factors affecting the market, such as pricing, changing fashion, competition, consumer spending, etc. Participants might focus on different factors and account for them differently; without a direct question about the market determinants, the sizedetermining factors would remain hidden. Therefore, a better technique would be to ask for the respondents to identify the factors important to future market size and to estimate probabilities and consequent sales, were they to occur. Then, using this information and a quantitative technique, such as Trend Impact Analysis, the size of the future market can be estimated, not only in view of the past market trend but also in consideration of trend-changing factors uncovered by Delphi participants. Finally, a weakness of the Delphi method is the time that it takes. A single round can easily require three weeks; a three-round Delphi is at least a three to four month affair, including preparation and analysis time. A recent article raises questions about Delphi. Fred Woudenberg reports that "the main claim of Delphi - to remove the negative effects of unstructured, direct interaction - cannot be substantiated." He found that Delphi does not produce more accurate answers than other methods, and that consensus occurs as a result of pressure brought on participants that have extreme opinions. These questions are well worth exploring in more depth. (Woudenberg, 1991; Kasten et al., 1993; and Rowe G. et al., 1991) True, the Delphi method makes participants with extreme opinions work harder than others. If opinions are not strongly held, participants may switch positions rather than write reasons for their estimates. On the other hand, those with strong opinions state why. The original impetus behind the method was to seek consensus, because expert consensus was believed more likely to

The Delphi Method

11

AC/UNU Millennium Project

Futures Research Methodology V2.0

be accurate than an individual forecast. Dalkey tested this proposition using obscure questions with known answers and found weak support for the concept (see the following section). Today, consensus is less important for many investigators than previously; now a useful product of the Delphi method is crystallization of reasons for dis-sensus. Furthermore, Delphi is now seen as no more or less than a systematic means of synthesizing the judgments of experts - the aggregate judgment representing a kind of composite expert composed, in the domain of interest, of the expertise of all participants. Does the method produce an accurate view of the future? It is no more accurate, probably, than any expert, single or composite. But suppose we wanted to form a scenario based on expert views of what might be possible. Or suppose we needed a judgment about whether or not we could mount a manned Mars mission and if so, how. Or suppose we wanted to explore the range of future events that could affect population growth or weaponry or war. No better way exists to collect and synthesize opinions than Delphi.

V. FRONTIERS OF THE METHOD

The central question on the Delphi frontier is "How can questions be addressed to the persons most likely to answer them well?" Here are some thoughts that were developed in the Phase 1 Millennium Feasibility Study (Gordon and Glenn, 1993). Self-Rated Expertise. If the number of questions and participants were small, then one might say "let's have everybody answer all questions." This approach is essentially taken in most Delphis. If the answers of all participants are accorded equal weight, the analysis is easy: for questions that can be answered in numerical form, the answers are often simply displayed as an average (or in statistical terms, such as median and standard deviation). Since sample sizes are small, this kind of analysis can give the appearance of numerical respectability when none really exists. For nonnumeric questions, the participants' answers can be displayed as a spread of opinions. Some researchers (e.g., Dalkey and Helmer) attempted to introduce a refinement: self- rated expertise. The reasoning went something like this: an expert ought to be believed more than a nonexpert; therefore, we ought to give an expert's opinion more weight than a nonexpert. In fact, weak evidence exists that for obscure questions with knowable answers (e.g., what was the true value of the mark against the dollar in January 1983?) experts, without use of reference books or data bases, are able to give more accurate answers then nonexperts. But how can experts be identified a priori? Dalkey and others have used various systems of selfrating; for example, 1. Are you an expert in this field, working in it daily? 2. Do you work in this field occasionally?

The Delphi Method

12

AC/UNU Millennium Project

Futures Research Methodology V2.0

3. Are you knowledgeable about this field through occasional professional reading? 4. Would you classify yourself as an informed layman? 5. Are you uninformed about this field? By assigning "points" for each level of self-rated expertise in the analysis of the group's response, answers can be discounted for lack of expertise to arrive at a group opinion. If this approach were to be used in the Millennium Project, then all questions would be sent to all participants. Each question would have appended a self-rating section. This approach would provide a weighing yardstick but has several disadvantages. Experts for many of the questions posed may not exist; Even if experts exist, should their answers carry more weight than nonexperts? (After all, if experts could answer the questions associated with poverty, for example, why haven't they already done so?) The approach is inefficient because everybody would have to read every question; and People with good and useful suggestions could be dissuaded from contributing since they might have to admit a self-depreciating level of ignorance.

Lock-and-Key Approach. In this approach, administrators attempt to match the capabilities of participants with the requirements of the questions. Each question and each respondent is "profiled," and questions are addressed only to those respondents whose profiles match. Respondent profiles can come from a questionnaire that is completed by the respondent at the time they join the panel and can be updated periodically. The profile would consist of best (or several) choice alternatives of the following sort: 1. What term best describes your occupation? Scientist______ Politician________ Physician_____ Etc. Business person____ Cleric_____ Service worker________ Engineer________ Artist________ Manufacturing________ Teacher________ Publisher_______ Retailing___________ 2. If scientist, what is your discipline? (Parallel questions for engineers, business persons teachers, etc.) Particle physics Organic chemistry Materials Social Science Genetics Biomedicine Etc. Nuclear physics Astronomy Psychologist Economics Political science Agriculture

The Delphi Method

13

AC/UNU Millennium Project

Futures Research Methodology V2.0

3. Would you consider yourself a: Generalist Specialist 4. Are you interested in (or do you have experience in): Technological forecasting Value-related issues Science Policy Political processes Policy analysis World affairs Utopias Etc.

In other words, the respondents would be asked to identify themselves with respect to the same descriptors that will be used to profile the questions. Before any questionnaire is sent out, the profile would be categorized according to the same set of descriptors that the respondents used to describe themselves. Then, through a matching process of the sort described below, the degree of match between a question and each respondent could be scored: Desired Question Attributes Scientist Materials Generalist Science Policy Techn Fcsting SCORE Weight 5 10 3 2 1 Resp No. 1 0 0 0 1 1 8 Resp No. 2 1 1 1 0 0 18 Resp No. 3 1 0 1 1 1 11 Resp No. 4 0 1 1 0 0 13 Resp No. 5 0 0 0 0 0 0

In this example, the administrators felt that the question could best be answered by a materials scientist, who considered himself a generalist, with experience in science policy and technological forecasting. Not each of these descriptors was viewed as being equally important, however. Being a scientist was weighed at 5; being an expert in materials was weighed at 10; etc. Data exists for each potential respondent with respect to each attribute (from the questionnaire completed at the time they joined the panel), so a simple scoring matrix could be constructed that assessed each respondent with respect to the desired attributes. By taking weighted sums, each respondent could be assigned a score. In this example, respondent 1 scored 8, and respondent 2 scored 18. After completing such an exercise, respondents could be rank ordered by score and the top set selected as a subpanel to which the question would be addressed. The number of people in a subpanel might be kept common across all questions, determined by budget considerations, or limited to say 90 percent of the top possible score. Of course, if the number of respondents were large, this process would have to be automated. The program would have access to a data base of the respondent's characteristics. It would call for question attributes and weights, automatically scan the data base, score each respondent and present a different rank ordered list of respondents for each question. The selected number of these could then be addressed automatically.

The Delphi Method

14

AC/UNU Millennium Project

Futures Research Methodology V2.0

Free Text Search. This approach is similar to the lock-and-key approach except that it envisions a more open ended and fully automated means of matching question requirements to respondent expertise. Suppose that each respondent were asked to submit a free text resume to the project when they joined. The resumes might be constrained to a given length and follow a predetermined format requiring, for example, that each respondent describe their current and past activities, interests, publications, etc. A data base would be formed of these resumes. Now, when each question is formulated, a set of key words would be associated with the question, or the significant words in the question itself would be viewed as a key word set. As before, each key word would be assigned a weight. The data base of resumes would be searched for these key words and, based on "hits," each potential respondent would receive a score. They could be rank ordered as before. Narrowing the Universe. In this approach, rather than asking about the respondent's expertise and interest, we let them select questions of interest and use their past selections to guide the routing of questions in future inquires. Suppose, after a respondent joined the network they received, for a time, all questions. They would, however, be instructed to answer only questions of interest to them and on which they felt competent to address. Then, over time, the administrators could learn which categories of questions each respondent included. When sufficient data were available, only the types of questions of past interest would be sent to them. The danger, of course, is that this approach is limiting. If the scanning process is cut off too early, then respondent/question matches that could have been productive might be missed. To minimize this potential detriment, respondents might be sent brief descriptions of other questions not included in their questionnaires and asked if any of these might be of interest to them. Past Performance Scoring. Here we define an expert as a person who can provide correct answers to difficult questions. We want experts to answer questions about the future on the network; therefore, we will first pose qualifying questions that experts should get right. Answering these qualifying questions correctly is the "ticket of admission" to the real inquiry. For example, if the administrators wanted to make a forecast of the future value of the German mark, the qualifying question might be "On yesterday's date, what was the value of the German mark vis a vis the U.S. dollar?" Only those people who answered close to correctly would be presented the future oriented question. This approach has obvious problems. The test questions would have to be designed with meticulous care to be appropriate. More important, many respondents might feel that such a test of their competence is inappropriate. A variation on this theme is to keep a record of each respondent's "batting average" by subject; that is, the ratio of correct to total forecasts made in the past. Important questions would be sent to those who have done well in the past. While initially attractive, this approach has a number of shortcomings. First, it would have to be built up over time. Different respondents would have a different number of "times at bat." Finally, whether a forecast has indeed occurred is often difficult to tell, even in retrospect.

The Delphi Method

15

AC/UNU Millennium Project

Futures Research Methodology V2.0

The Tree Approach. In this approach, a sequential series of questions is asked of the respondents, each requiring a more detailed knowledge of the field. At some point along the line, the respondents self disqualify. Here's an example. Suppose our question has to do with the future of the electoral college system of elections in the United States: Will the electoral college system of election in the United States change; if so, to what form and when? The administrators might ask: Domain Question: Have you ever thought much about how Presidents are elected in the United States? (Yes, No) If "no," no further questions. Qualifying Question: Can you name three other ways in which this might be done? (Yes, No) If "no," no further questions. Actual Question: Do you think the electoral system in the United States will change within the next 20 years? (Yes, No) If "no," then ask: Why not? If "yes," then ask: Why? To what form? When? The disadvantage of this approach is its complexity and its need for formulating precise questions. If this could be tolerated, it would certainly lead to appropriate matching of question to respondent. The general principals of a Delphi study, anonymity, multiple rounds, and feedback of prior round information to the current round, have been used in several novel ways in recent work by the Millennium Project. Some highlights of these applications are listed below to illustrate the wide range of possibilities for this method. Delphi has been used in scenario construction. One example of the use of Delphi techniques in scenario construction dealt with anti-terrorism. The study began with a request for anti-terrorist scenarios distributed to listserves of the Millennium Project of the American Council for the United Nations University and the World Futures Studies Federation. The scenarios submitted in response to this request were analyzed to identify and rate policies and actions that might be useful in counterterrorism strategies. This work was posted on-line with a further request for comments, modifications to existing scenarios and added scenarios. The submitted scenarios and others from outside of this effort were reviewed to identify actions and policies that might be useful in counterterrorism strategies. The fifty-nine actions/policies identified were then submitted to an international panel for judgments about their effectiveness, plausibility, and potential unexpected downside risks. 1
1

Jerome Glenn and Theodore Gordon, State of the Future 2002, Millennium Project, American Council for the United Nations University, Washington, DC. 2002.
The Delphi Method 16

AC/UNU Millennium Project

Futures Research Methodology V2.0

Another example of the use of the technique was made by the Millennium Project in forming normative scenarios. Millennium Project participants identified and rated norms that were to be used as the scenario framework. In the first round participants selected the following top four norms around which to form the scenario: environmental sustainability, plenty, global ethics (the identified and accepted), and peace. The others in order of preference were health, freedom, universal education access, equity, preservation of the human species, enlightenment, exciting and meaningful life, self-actualization, longevity, everyone has everything they want, and security. A normative scenario was devised around these norms; the scenario focused on actions to address the Global Challenges being tracked by the Project. These actions connected the present world to the normative future of 2050. For the second round, a scenario review panel was formed of long-term normative-oriented participants; they were asked to review and improve the draft of the scenario to illustrate optimistic possibilities for the future.2 The Delphi approach has been used in construction of a State of the Future Index. In 1999 2000, the Global Lookout Panel of the Millennium Project was asked to identify indictors by which the status of 15 global challenges could be measured. These nominated indicators were subsequently evaluated by the panel in terms of their availability and usefulness. The results, plus a review of other index studies, were submitted to the Global Lookout Panel in 2001 to collect judgments about potential indicators for the SOFI. The respondents provided judgments about what the best (norm) and worst (dystopic) status was for the indicator in 2011. They also rated the importance of reaching the norm and dystopic state. The criteria for assigning a high weight to a variable were: the number of people affected; the significance of the effect; whether some groups seem to be affected differentially; the time over which the effect will be felt; and whether the effect is reversible. Millennium Project staff worked with the variables identified in this questionnaire, obtaining 20 years (where possible) of historical data from the most authoritative sources, and forecasting each variable using a time series approach (and later, using Trend Impact Analysis) to for a State of the Future forecast. 3 Delphi principles have been used in policy studies. As an example, in 2002-2003, the Millennium Project performed a study of routes to peace in the Middle East. In this application, the first questionnaire listed pre-conditions to peace and actions that could be taken to satisfy the preconditions. The panel was asked to add to the list of actions and to evaluate the set in terms of importance, likelihood of implementation and their potential to evoke undesired or unintended downstream consequences. Extensive written commentary was generated by both the respondents and those who elected to not participate. The data were analyzed not only from the panel as a whole but also from the standpoint of the antagonists using sub samples of responses from Arab countries and Israel.. From this body of data, the Project will form a set of normative scenarios that begin from points of agreement and illustrate paths to peace.4

2 3

Ibid. Ibid. 4 Jerome Glenn and Theodore Gordon, State of the Future 2003, Millennium Project, American Council for the United Nations University, Washington DC. 2003.
The Delphi Method 17

AC/UNU Millennium Project

Futures Research Methodology V2.0

VI. SAMPLES OF APPLICATIONS

The Delphi method can be used in almost any forecasting application; reference Lindstone and Turoff, 1975, presents many prior applications. The Millennium Feasibility Project, performed by the United Nations University for the U.S. Environmental Protection Agency in early 1993, includes an application of potential interest. This study included a Delphi that examined important future events and policies that could affect world population growth and the environment in the next 25 years. The respondents for this Delphi had self-identified themselves as expert or greatly interested in population or in the environment in a prior study; this list was augmented by recommendations by the study's sponsor (EPA), a literature scan, and recommendations of people who had already accepted our invitation to participate. In the first round, 76 questionnaires were sent out via airmail, fax, and e-mail; 42 responses were received. The second round went to the same list. Of those who completed the first round, 25 responded to the second around (at the time of this writing). The questionnaires were first sent to a small planning committee, largely in open-ended form; the responses from the smaller group helped define a much more extensive multiple-choice format. The questions dealt with past history (e.g., "What developments and trends have been important in shaping today's population growth patterns?") and future developments (e.g., "Do you think these trends will continue? What other novel developments will deflect the trends in the future? What will be their consequences?"). The design of the questions was straightforward. The subjects of population and environment were dealt with in separate sections of the questionnaire, but the flow for each was largely the same. In the first round, questions focused on past forces for change, both from the standpoint of historical importance and future influence. The researchers provided the group with an initial list and asked them to extend the list and provide judgments about historical and future importance. They then asked about new forces for change and potential future unprecedented events; again, the researchers provided a list, asked for other ideas that might extend the list, and requested judgments about the likelihood and impact of the future developments. In the case of population, the questionnaire also called for direct estimates about the future size of the population and population growth rates in several countries and regions. It also asked about factors that might be different between developed and developing countries. In the case of the environment, the questionnaire included an additional section dealing with the interaction between population and the environment. The second round fed back the newly suggested items to the panel and requested judgments similar to those called for in the first round. It also asked for judgments about policies that might be implemented to improve the future situation. These policy questions focused on the one or two future events that were judged to have low probability but high favorable impact, if they were to occur. This question asked, in effect, what might be done to improve the probabilities of these events.

The Delphi Method

18

AC/UNU Millennium Project

Futures Research Methodology V2.0

To illustrate the nature of the questions asked, the population section of the second questionnaire is reproduced below. The full questionnaires and the results of the study are included in reference (Gordon and Glenn, 1993). UNU MILLENNIUM PROJECT FEASIBILITY STUDY DELPHI ON ENVIRONMENT AND POPULATION ROUND 2 QUESTIONNAIRE 1. POPULATION SECTION The first round of this Delphi on Population and Environment asked the panel to estimate population and growth rates for several regions 25 years from today (or to the year 2018). For your information, Table 1 below lists the panel's median responses. The inter quartile ranges for the population estimates are shown in parentheses next to the population estimates. (The inter quartile range is 25% from the lowest to 25% from the highest estimates; hence, 50% of the estimates fall within this range.) For purposes of comparison, 1992 year-end population and growth rates are listed first (cities are 1990 data).
Table 1. 1992 and 25 year forecasts of population from Population & Environment Delphi I

Region
World Developed Countries Developing Countries Africa China India United States Brazil Japan Iran

1992 Population
5.4 Billion 1.2 Billion 4.2 Billion 654 Million 1.2 Billion 883 Million 256 Million 151 Million 124 Million 60 Million

Rates
1.7% 0.5% 2.0% 3.0% 1.3% 2.0% 0.8% 1.9% 0.3% 3.3 %% 2.3% % 0.9% 2.9%

Median Forecast (& Ranges)

8.0 Billion (7.0 - 8.5) 1.4 Billion (1.3 - 1.4) 6.5 Billion (5.7 - 7.0) 1.2 Billion (1.0 - 1.4) 1.5 Billion (1.4 - 1.6) 1.3 Billion (1.2 - 1.4) 300 Million (300 - 320) 230 Million (210 - 250) 130 Million (130 - 137) 105 Million (100 - 120)

Rates
1.5 % 0.35% 1.8 % 2.5 % 1.0 % 1.5 % 0.65% 1.6 % 0.2 % 3.0 % %%%% %% 2.0 % 0.5 % 2.0 %

Mexico Tokyo Mexico City

88 Million 18.1 Million 20.2 Million

140 Million (128 - 140) 21 Million (20 - 22) 32.5 Million (30 - 35)

In the first round, the panelists were asked to rate some forces that led to the reduction of the

The Delphi Method

19

AC/UNU Millennium Project

Futures Research Methodology V2.0

world population growth rate from 2.06% in the late 1960s to 1.7% currently and to assess how these forces might change over the next 25 years. For your information, Table 2 below shows the average of the panel's judgments. In making these assessments, they used Scale A shown below. Scale A Historic Influence Future Influence 1 = Very important 1 = Greatly increasing in importance 2 = Important 2 = Increasing in importance 3 = Marginally important 3 = Remaining the same in importance 4 = Unimportant 4 = Decreasing in importance 5 = Counter impact 5 = No longer a factor, or mixed Table 2 - The importance of some historic factors (first number) on global population growth and possible future changes in importance (second number) 2.0 2.0 2.0 2.1 2.1 2.1 2.5 2.0 2.0 2.1 2.1 2.1 2.5 2.2 2.1 2.8 2.1 2.0 2.3 2.4 2.1 2.8 2.1 2.0 2.3 2.4 Availability of inexpensive, simple effective contraceptives; Family planning and public health programs China's population policy Government policies in developing countries that encourage smaller families Increasing number of years that women attend school Rising incomes and the spread of middle class values Rise of "woman's power." Family planning and public health programs China's population policy Government policies in developing countries that encourage smaller families Increasing number of years that women attend school Rising incomes and the spread of middle class values Rise of "woman's power."

QUESTION 1.1 We asked the panel to suggest additional forces that might be responsible for the historic changes in population growth. Many new suggestions were received. Please review the list presented below and provide your judgments about their historical and possible future importance using Scale A. Please use the first space in front of each item for the historic influence over the past 25 years and the second space for your judgment of the potential changes in importance over the next 25 years.. 1.1.1 ___ ___ 1.1.2 ___ ___ wealth 1 .1.3 ___ ___ 1.1.4 ___ ___ Move away from agricultural society/primary sector Demonstration by the North that fewer children can mean more

Legitimization of contraception Availability of male contraceptives

The Delphi Method

20

AC/UNU Millennium Project

Futures Research Methodology V2.0

1.1.5 ___ ___ 1.1.6 ___ ___ 1.1.7 ___ ___ 1.1.8 ___ ___ 1.1.9 ___ ___ 1.1.10 ___ ___ 1.1.11 ___ ___ 1.1.12 ___ ___

Decreasing Catholic Church's social influence Increasing futuristic orientation War, famine, disease, and pestilence Environmental deterioration Spread of new communications media (Television, etc) Education about relation of environment and population Decreases in infant and child mortality Improved literacy by improved children's schooling

Population growth rates have remained high in developing countries. Using the Scale A above, the following historical influences were rated in order of importance and rated in how they might change in importance over the next 25 years. Table 3 presents the panel's estimates. Historical assessment is the first number; future assessment is the second number. Table 3 - Some reasons for high population growth in developing countries 1.6 1.8 2.0 2.2 2.2 2.8 3.0 2.4 2.7 2.8 3.1 3.4 Need of children for social security, to support parents in their old age. Poverty Low levels of literacy Lack of information and access to contraceptives High infant mortality Government policies supporting large families

QUESTION 1.2 In the first round, the panelists were also asked to suggest other forces that might account for population growth rates remaining high in many developing countries. Many new suggestions were received. Please review the developments listed below and provide your judgments about their historical and possible future importance using Scale A. Please use the first space in front of each item for the historic influence over the past 25 years and the second space for your judgment of the item's future influence over the next 25 years. 1.2.1 ___ ___ Masculinity associated with increasing numbers of children 1.2.2 ___ ___ Close adherence to religious tenets that lead to avoidance of contraceptives 1.1.3 ___ ___ Low levels of literacy and lack of understanding ecological view of planet earth

The Delphi Method

21

AC/UNU Millennium Project

Futures Research Methodology V2.0

1.2.4 ___ ___ Discrimination against women (little autonomy, education, and lack of social power) 1.2.5 ___ ___ Belief by governments that larger populations mean greater political strength 1.2.6 ___ ___ Rural areas receive less attention from population programs than urban areas 1.2.7 ___ ___ Family-based, labor-intensive economies needing children's input 1.2.8 ___ ___ Liberal immigration policies in richer countries In the first round, the panelists were asked to assess new forces and unprecedented events that might influence population growth in the future. They were asked for judgements about the likelihood of occurrence and impacts over the next 25 years. For your information, Table 4 shows the average of the panel's responses about the events included in the first round. In making these assessments, they used Scale B shown below. Scale B Likelihood Eventual impact on population within of occurrence the next 25 years of growth 1 = almost certain 1 = reduces growth rate by 30% or more 2 = likely 2 = reduces growth rate by 5-30%. 3 = even or 50/50 chance 3 = no impact. 4 = unlikely 4 = increases growth rate by 5-30%. 5 = almost impossible 5 = increases growth rate by 30% or more

Table 4 Likelihood of occurrence of new forces or unprecedented events (first number) that might influence future population growth and eventual impact over the next 25 years (second number) 1.8 2.7 Simple test for identifying the sex of unborn children 1.9 3.4 Increasing survival in middle age and early old age due to curing or improved therapy for heart disease, cancer and stroke 2.1 2.3 Simple, safe, effective male birth control pill 2.3 2.4 Changes in death rates due to spread of AIDS in developing countries 2.4 2.6 Massive starvation of the scale of Somalia at least once every three years 2.5 2.0 Long term (at least one year) contraceptives widely used and accepted as control pills are today 2.6 2.4 New deadly viruses, including AIDS mutations 2.8 2.6 Simple method for selecting sex at conception 2.9 2.1 Rising incomes in most developing countries 3.3 2.3 Change in the Vatican's position: use of contraceptives becomes available without limit 3.4 2.1 Two children per family becomes the social norm in the majority of the developing world

birth

The Delphi Method

22

AC/UNU Millennium Project

Futures Research Methodology V2.0

QUESTION 1.3 In the first round, the panelists were asked to suggest other new forces or unprece-dented events that could influence population growth. Many new suggestions were received. Please review the list below and provide your judgments about their likelihood and impacts, if they were to occur over the next 25 years using Scale B. Please use the first space in front of each item for the likelihood of occurrence and the second space for the item's future impact over the next 25 years. 1.3.1 ___ ___ Wide spread use of relatively cheap and easy ways to affect the aging process, resulting in diminished mortality and extension of the life span about 5 years 1.3.2 ___ ___ Increasing sterility, worldwide, by 10% due to environmental degradation, higher level of stress, and other factors 1.3.3 ___ ___ Increasing impotency by 10% due to environmental degradation, level of stress, and other factors 1.3.4 ___ ___ Public health programs decrease mortality of infant and young adults by 5-10% 1.3.5 ___ ___ 3% of births via new methods of impregnation and prenatal development ("Outside womb" fertility, artificial inseminating, surrogate motherhood, other such techniques) 1.3.6 ___ ___ Important negative changes in the environment (e.g., accumulation of toxic wastes, failures of mono-agriculture crops, contamination of drinking water) resulting in increased mortality 1.3.7 ___ ___ Doubling, worldwide, of today's level of male and female 1.3.8 ___ ___ Basic economic needs met for 90% of global population (minimum acceptable health care, food, and shelter) 1.3.9 ___ ___ Successful new prototype habitats in oceans, cold regions, or in earth orbit gives stimulates popular frontier spirit and alternatives to previous urbanization patterns 1.3.10___ ___ 20% increase of people active in religions that encourage higher fertility homosexuality higher

by

QUESTION 1.4 With the information provided by the panel in round I, we identified two possible future events that were judged to be unlikely, but nevertheless potentially effective in reducing population growth rates, if they were to occur. Both have been considered before and one is the focus of large international programs. Without repeating what has been said and tried many times, we ask below for your suggestions about novel policy approaches that you think

The Delphi Method

23

AC/UNU Millennium Project

Futures Research Methodology V2.0

might be practical, and if implemented, improve the probability of these developments. Please write your suggestions in the space provided below. Change the statements if you wish. Add other policy domains if you wish. Please be as specific as possible and use separate sheets of paper or electronic space as you need. 1.4.1 Novel policies that could lead to a social norm of two children per family throughout the majority of the developing world:

1.4.2 Novel policies that could lead to the Vatican's acceptance of contraception without limit:

1.4.3. Other policy areas of your selection and how you would address it:

To illustrate the quality of the responses received, some of the results of this questionnaire are summarized below; a complete analysis- including summaries of answers to all questionsappears in Gordon and Glenn, 1993. Using the scale for historical influence, the panel found the following ten developments most significant in the evolution of world population growth.

The Ten Most Important Historical Factors 2.0 2.0 2.0 2.0 2.0 2.1 2.1 2.1 2.2 2.3 Availability of inexpensive, simple effective contraceptives; Family planning and public health programs China's population policy Legitimization of contraception Decreases in infant and child mortality Government policies in developing countries that encourage smaller families Increasing number of years that women attend school Rising incomes and the spread of middle class values Move away from agricultural society/primary sector Spread of new communications media (Television, etc.)

And, the panel judged, using the scale for future influence, that among forces already in play, the following would be the ten of most greatly increasing importance over the next 25 years The Ten Historical Forces That Promise to be of Increasing Importance

The Delphi Method

24

AC/UNU Millennium Project

Futures Research Methodology V2.0

1.8 2.0 2.0 2.1 2.1 2.1 2.2 2.2 2.2 2.2

Spread of new communications media (Television, etc.) Increasing number of years that women attend school Environmental deterioration Government policies in developing countries that encourage smaller families Family planning and public health programs Improved literacy by improved children's schooling Availability of inexpensive, simple effective contraceptives Legitimization of contraception Education about relation of environment and population Decreases in infant and child mortality

The questionnaire included similar questions about developing countries: reasons for high population growth in these countries and the potential importance of these forces over the next 25 years. Here is a summary of the top items: The Ten Most Important Reasons for High Population Growth in Developing Countries 1.6 Need of children for social security, to support parents in their old age. 1.7 Discrimination against women (little autonomy, education, and lack of social power) 1.8 Poverty 1.8 Low levels of literacy and lack of understanding ecological view of planet earth 2.0 Family-based, labor-intensive economies needing children's input 2.1 Close adherence to religious tenets that lead to avoidance of contraceptives 2.1 Rural areas receive less attention from population programs than urban areas 2.2 Lack of information and access to contraceptives 2.2 High infant mortality 2.3 Masculinity associated with increasing numbers of children The Ten Historical Forces That Promise to be of Increasing Importance in Developing Countries over the Next 25 Years 2.2 2.4 2.6 2.7 2.8 2.8 3.0 3.1 3.1 3.2 Low levels of literacy and lack of understanding ecological view of planet earth Poverty Rural areas receive less attention from population programs than urban areas Low levels of literacy Discrimination against women (little autonomy, education, and lack of social power) Lack of information and access to contraceptives Need of children for social security, to support parents in their old age. Family-based, labor-intensive economies needing children's input High infant mortality Belief by governments that larger populations mean greater political strength

The panelists were asked to assess new forces and unprecedented events that might influence population growth in the future and suggest and assess additional forces as to their likelihood of occurrence and impacts over the next 25 years using the scale shown below

The Delphi Method

25

AC/UNU Millennium Project

Futures Research Methodology V2.0

Likelihood Eventual impact on population within the next 25 years of growth 1 = almost certain 1 = reduces growth rate by 30% or more 2 = likely 2 = reduces growth rate by 5-30%. 3 = even or 50/50 chance 3 = no impact. 4 = unlikely 4 = increases growth rate by 5-30%. 5 = almost impossible 5 = increases growth rate by 30% or more

of occurrence

Using this scale, the items that were seen to have a retarding affect on population growth, together with their probabilities, are shown below. Impact of New Forces and Unprecedented Events on Population Growth Over the Next 25 Years Impact Probab 2.0 Development

2.5 Long term (at least one year) contraceptives widely used and accepted as birth control pills are today 2.9 Rising incomes in most developing countries

2.1 2.1

3.4 Two children per family becomes the social norm in the majority of the developing world 2.1 Simple, safe, effective male birth control pill

2.3 2.3

3.3 Change in the Vatican's position: use of contraceptives becomes available without limit 2.9 Increasing sterility, worldwide, by 10% due to environmental degradation, higher level of stress, and other factors 2.3 2.6 Changes in death rates due to spread of AIDS in developing countries New deadly viruses, including AIDS mutations

2.3

2.4 2.4 2.5

2.5 Important negative changes in the environment (e.g., accumulation of toxic wastes, failures of mono-agriculture crops, contamination of drinking water) resulting in increased mortality 2.4 2.8 1.8 3.4 Massive starvation of the scale of Somalia at least once every three years Simple method for selecting sex at conception Simple test for identifying the sex of unborn children Increasing impotency by 10% due to environmental degradation, higher

2.6 2.6 2.7 2.7

The Delphi Method

26

AC/UNU Millennium Project

Futures Research Methodology V2.0

level of stress, and other factors 2.7 2.9 3.4 Doubling, worldwide, of today's level of male and female homosexuality

2.6 3% of births via new methods of impregnation and prenatal development ("Outside womb" fertility, artificial inseminating, surrogate motherhood, such techniques) 2.9 3.9 Basic economic needs met for 90% of global population (minimum acceptable health care, food, and shelter)

other

With the information provided by the panel in round I, we identified two possible future events that were judged to be unlikely, but nevertheless potentially effective in reducing population growth rates, if they were to occur. These were;

Impact 2.1

Probab 3.4

Development Two children per family becomes the social norm in the majority of the developing world. Change in the Vatican's position; use of contraceptives becomes available without limit.

2.3

3.3

These items were of particular interest to us since they represented potential policy opportunities. Both have been considered before and one is the focus of large international programs. We asked for suggestions about novel policy approaches that might be practical, and if implemented, improve the probability of these developments. We also asked our respondents to not repeat, if possible, previously made suggestions. Some of the panel's ideas follows: Novel policies that could lead to a social norm of two children per family throughout the majority of the developing world: - Family-size taxes that increase substantially for every child over two. - Prohibition of child labor. - Policies should focus on women's access to work and education. - Free circulation of contraceptives in public health programs supported by churches and other religious organizations. - International information utilities in education, health, and training in information society.

The Delphi Method

27

AC/UNU Millennium Project

Futures Research Methodology V2.0

- With enough effort and resources, the present policies (national, bilateral, and multlateral) that try to make family planning services universally available and to promote information, education and communication about family planning, then the two child norm should be virtually universal within 25 years. - Without significant reductions in poverty and infant mortality, and increase in women's education and empowerment, these policies would need coerce in nature or at least provide strong economic disincentives to having many children. Some Novel Policies That Could Lead to the Vatican's Acceptance of Contraception Without Limit: - Remove Holy See from the United Nations on grounds that it is not really a country and give it the same status as the World Council of Churches. - Promotion of contraceptives accompanied with strong dissemination of moral values. - Theological doctrine developed by US. Catholic Bishops in support of sustainable development (at UNCED 1992). - Policies that focus on responsibilities of woman to make choices. - Only decline of organized religion is likely to affect the situation. - Allow priests and nuns to marry and pay for the raising of children. - Reconsideration of the theology of St. Thomas Aquinas - especially the discarding of the Thomastic view of "natural law" as it applied to human sexuality. Christ said nothing whatsoever about human sexuality. Separate the notion of procreation as the only "natural end" from that of enjoyment. We also asked respondents to suggest other areas that were ripe for policy intervention. Here are some they suggested: - Research for long-term implantable ovulation suppression device. - Global televised debate on population policy, environmental protection, and social ethics. - Sex education in schools - All nations will have to eventually adopt policies that clearly state the freedom of individual choice. - Policies that emphasizes people to take charge of their lives and reduce dependence on governments.

The Delphi Method

28

AC/UNU Millennium Project

Futures Research Methodology V2.0

BIBLIOGRAPHY
Amant, R., "Comparison of Delphi Forecasting Studies in 1964 and 1969," Futures, Vol 2, No. 1, March 1970. This article is an early attempt to determine just how accurate earlier forecasts made by the Delphi method have proven to be. It showed that scientific and technological forecasts were indeed likely to be more accurate than forecasts in softer areas, such as fashion, politics, or social behavior. It also showed that sometimes, whether an event, even one succinctly stated, had indeed occurred was quite difficult to tell. Azani, Hossein and Khorramshahgol, Reza, "Analytic Delphi Method," Engineering Costs and Production Economics, July 1990. Biji, Rob, "Delphi in a Future Scenario Study on Mental Health and Mental Health Care," Futures, April 1992. Gibson, Lay James and Miller, Mark, "A Delphi Model for Planning 'Preemptive' Regional Economic Diversification," Economic Development Review, Spring 1990. Ginsburg, Alan, "Integrating Evaluation Into Decisionmaking," Public Manager, Winter 199293. Goldfisher, Ken, "Modified Delphi: A Concept for Product Forecasting," Journal of Business Forecasting, Winter 1992-93. Gordon, T. J. and Glenn, J. C., Issues in Creating the Millennium Project: Initial Report from the Millennium Project Feasibility Study, United Nations University, August 1993. This work was, in part, an exploration of the feasibility of conducting worldwide Delphis on issues of global importance, using scholars, futurists, and other interested and informed participants in panels that communicated by e-mail, fax, and air mail. Gordon, T. J. and Helmer, Olaf, Report on a Long Range Forecasting Study, R-2982, 1964. This report is still interesting to read. It contains many innovations that are used in the analysis and presentation of Delphi results. For example, the reports presents arguments for using median rather that means of the group's responses, shows how ranges of opinions can be presented graphically, distinguishes between likelihood and probability, etc. Helmer, Olaf and Rescher, Nicholas, "On the Epistemology of the Inexact Sciences," Management Sciences, Vol. 6, No.1 (1959). In this landmark work Helmer and Rescher set out the philosophical backdrop for Delphi and set limits of expectation about what can and cannot be known when the questions being addressed fall into the category of "inexact science." They include all of social science and political science as "inexact." Kastien, M. R., et al., "Delphi, The Issue of Reliability," Technological Forecasting and Social Change, Nov. 1993. Lindstone, H., and Turoff, M. (ed.), The Delphi Method, Addison Wesley Publishing Co., 1975.

The Delphi Method

29

AC/UNU Millennium Project

Futures Research Methodology V2.0

One of the best detailed discussions of the Delphi method. McCarthy, Kevin, "Comment on the 'Analytic Delphi Method'," IInternational Journal of Production Economics, May 1992. Mitchell, Vincent-Wayne, "Using Delphi to Forecast New Technology Industries," Marketing Intelligence and Planning, Vol 10, Issue 2, 1992. Nimgade, Ashok and Sonk, Joseph, "Ordinal Quantification of the Concept of Technology Through Pooled Consensus," R&D Management, Jan. 1991. Olshfski, Dorthy, and Joseph, Alma, "Assessing Training Needs of Executives Using the Delphi Technique," Public Productivity and Management Review, Spring 1991. Rohde, William, "Past Predictions Take Shape," Healthcare Executive, May/June 1991. Rowe, Gene, George Wright, and F. Bolger, "Delphi: A Revaluation of Research and Theory," Technological Forecasting and Social Change, Vol. 39, May 1991. Russo, David M., and McLaughlin, "The Year 2000: A Food Industry Forecast," Agribusiness, November 1992. Shota, Ushio, "The Future of High Tech: Forecasts for the Next Decade," Tokyo Business Today, April 1993. Technological Forecasting and Social Change, Special Issue on Delphi, 1975, no. 2. Discusses critiques of the method. Vickers, Brent, "Using GDSS to Examine the Future European Automobile Market," Futures, October 1992. Woudenberg, Fred, "An Evaluation of Delphi," Technological Forecasting and Social Change, Sept. 1991. Contains a broad review of the literature on Delphi and many references to the method and past studies.

The Delphi Method

30