International Institute for Tel: 43 2236 807 342

Applied Systems Analysis Fax: 43 2236 71313

Schlossplatz 1 E-mail: [email protected]
A-2361 Laxenburg, Austria Web: www.iiasa.ac.at

Interim Report IR-09-09

Requirement Analysis and Implementation

of Multicriteria Analysis in the NEEDS Project
Marek Makowski ([email protected])
Janusz Granat ([email protected])
Hongtao Ren ([email protected])
Warren Schenler ([email protected])
Stefan Hirschberg ([email protected])

Approved by
Detlof von Winterfeldt ([email protected])
Director, IIASA
November 2009

Interim Reports on work of the International Institute for Applied Systems Analysis receive only limited
review. Views or opinions expressed herein do not necessarily represent those of the Institute, its National
Member Organizations, or other organizations supporting the work.
M. Makowski et al. - ii - MCA-Needs: requirements&implementation

Foreword

Public participation in decision-making is almost always desired but difficult to effectively

achieve, especially in situations where the decision problem is complex and a policy rec-
ommendation requires involving large and diversified groups of stakeholders. Making
recommendations for European future energy technologies is an example of such a prob-
lem. Its analysis implicitly defines a class of multicriteria analysis problems composed
of large sets of alternatives, each characterized by a large number of criteria organized in
a hierarchical structure. These criteria are diverse and conflicting, and are organized into
three sets composed of economic, environmental, and social criteria respectively. Each
of these sets has the hierarchical structure of the corresponding criteria. Moreover, the
stakeholders invited to make the analysis not only had different preferences for trade-offs
between such criteria, but also diversified backgrounds and thus typically very limited
experience in analyzing problems using formal multicriteria approaches, and especially
in defining preferences.
There existed neither suitable methods nor tools to support multicriteria analysis of
such a problem. Therefore the team that both developed the methods and implemented
them as a dedicated Web-site had to solve a number of methodological and technological
challenges. This report first describes the requirement analysis for the corresponding class
of problems, and then summarizes the implementation of the tool used for interactive
multicriteria analysis of future energy technologies by the large and diversified group of
stakeholders.
The lessons learned from the documented process are of interest to both researchers
and practitioners involved in development of methods and/or tools for multicriteria anal-
ysis, especially of large sets of discrete alternatives.
M. Makowski et al. - iii - MCA-Needs: requirements&implementation

Abstract

This report specifies the requirements for and implementation of the multicriteria analysis
of future energy technologies performed by a large number of stakeholders within the
EU-funded integrated project NEEDS. The report is composed of two main parts and the
appendix.
The first part starts with a summary of the objectives of the analysis followed by a
detailed specification of the analyzed problem, in particular the analysis context, discus-
sion of the sets of criteria and alternatives, and the participation of the stakeholders. Next,
the planned problem analysis process is first outlined, and then discussed in more detail.
Finally, the requirements for the multicriteria analysis are specified.
The second part deals with the implementation of the dedicated Web-site developed
for this analysis, and later extended to support analysis of any multicriteria choice between
discrete alternatives. It starts with an overview of the problem analysis process and the
corresponding basic assumptions. The architecture of the application and its features
are then presented. Lessons learned from the development and use of this application
conclude this part of the report.
The appendix contains a review of the state-of-the-art of applying multicriteria anal-
ysis to energy problems, as well as characteristics of three applications that exploit the
multicriteria analysis methods for energy problems considered relevant to the analysis
reported in this paper.
M. Makowski et al. - iv - MCA-Needs: requirements&implementation

Acknowledgments

The authors gratefully acknowledge help of Bartosz Kozłowski of IIASA, who over the
period of four years has systematically contributed his expertise in several areas, includ-
ing data basis, Web-based applications, and software development. Moreover, he has
provided the so-called Framework composed of software modules supporting implemen-
tation of Web-based applications.
The authors also thank colleagues from the Laboratory for Energy Systems Analysis,
Paul Scherrer Institute, Villigen, Switzerland for their comments and suggestions. We
especially thank Peter Burgherr for his numerous comments on the MCA design and
implementation, as well as for contributions to the preparations and running the analysis.
We also thankfully acknowledge the comments of members of other teams participating in
the Research Stream 2b of the NEEDS project provided during testing of the application
described in this report.
The research reported in this paper was partly financially supported by the EC-funded
Integrated Project NEEDS (project no: 502687), and by the Austrian Federal Ministry of
Science and Research.
M. Makowski et al. - v - MCA-Needs: requirements&implementation

About the authors

Marek Makowski leads the IIASA Integrated Modeling Environment Project. His re-
search interests focus on model-based support for solving complex problems, which
incorporates three interlinked areas. First, integration of interdisciplinary knowl-
edge and its representation by mathematical models. Second, creation of knowl-
edge by comprehensive model analysis, including multicriteria methods. Third,
tailoring the modeling process to meet the needs of decision-making processes.
Thus Marek’s research interests cover a cluster of areas relevant to the adaptation
(whenever possible) or development (when needed) of methodology, algorithms,
and software for model-based decision-making support. This includes more spe-
cific topics in Operations Research such as: multicriteria problem analysis, large
scale optimization, optimization of badly conditioned problems, use of database
management systems for complex models, decision analysis and support, user in-
terfaces in decision support systems, effective treatment of uncertainty and risk.
Marek has published over 130 papers and book-chapters, co-edited four books, co-
ordinated or led several scientific projects, and has been twice guest editor of the
European Journal of Operational Research.

Janusz Granat is a leader of the Division of Advanced Information Technology at the

National Institute of Telecommunications. He also lectures on decision support sys-
tems and management information systems at the Warsaw University of Technol-
ogy. His scientific interests include decision support systems, multi-criteria analy-
sis, modeling, data mining, event mining, techno-economic analysis and the design
of the telecommunications network. He has been involved in various industrial and
scientific projects e.g., data warehousing and decision support systems for telecom-
munication industry, building data mining models for marketing departments, de-
velopment of decision support systems for energy management.

Hongtao Ren is a researcher in the IIASA Integrated Modeling Environment Project

(IME). His research interests include modeling environments, computerized sup-
port for knowledge creation, integrated systems for scientific creativity, knowledge
engineering, the semantic web, and text mining. He has developed diverse Web-
based applications in collaboration with IIASA colleagues, including Multiple Cri-
teria Analysis of Discrete Alternatives, and Multiple Criteria Analysis of Future
Energy Technologies. He has participated in various activities of the Center of
Excellence at the Japan Advanced Institute of Science and Technology, and in par-
ticular has developed Creative Environments for supporting scientific research.

Warren Schenler is a researcher in the Interdepartmental Laboratory for Energy Sys-

tems Analysis at the Paul Scherrer Institute (PSI), Switzerland. He has worked on
energy research projects, and especially on electric power system research issues,
M. Makowski et al. - vi - MCA-Needs: requirements&implementation

related to the United States, Switzerland, Germany, the EU and China. His core
interests include system and technology interactions, including sector and societal
economics, system operation and risk, and social impacts like employment. He
has experience in applying multicriteria decision analysis to questions of energy
and sustainability, including the design of sustainability criteria sets, multi-scenario
simulation and data mining. Other interests include the interaction between the
electric power sector and other energy sectors such as hydrogen and transportation.

Stefan Hirschberg is the head of the Interdepartmental Laboratory for Energy Systems
Analysis at the Paul Scherrer Institute (PSI), Switzerland. The Laboratory con-
sists of three Groups: Technology Assessment, Energy Economics, and Risk and
Human Reliability. Since 1992 he has been coordinating interdisciplinary activ-
ities on ”Comprehensive Assessment of Energy Systems”. He manages a num-
ber of projects for energy and environmental authorities, for electrical utilities and
vendors, and within international programs. His main research interests currently
include: Life Cycle Assessment, Environmental Impact and External Cost Assess-
ment, Comparative Risk Assessment, Sustainability Assessment and Development
of Integrated Tools for Decision Support. Stefan has about 200 publications and
lectures at the Swiss Federal Institutes of Technology in Zurich and Lausanne. He
has been a member of numerous advisory, consultant and expert groups supporting
national and international organizations. In 2008 Stefan was elected an individual
member of the Swiss Academy of Engineering Sciences (SATW). Before joining
PSI he was responsible for Risk and Reliability Assessment within ABB, Sweden
and during a leave of absence for Probabilistic Safety Assessment at the IAEA.
M. Makowski et al. - vii - MCA-Needs: requirements&implementation

Contents

1 Introduction 1

2 Problem specification 3
2.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 The NEEDS Project, relevance to energy/electricity sectors and
importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 The MCDA problem as it applies to NEEDS . . . . . . . . . . . 6
2.1.3 Why MCDA is needed . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Elements of multicriteria analysis . . . . . . . . . . . . . . . . . . . . . 7
2.3 Set of criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Sets of alternatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4.1 Preparation of alternatives . . . . . . . . . . . . . . . . . . . . . 10
2.5 Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5.2 Information that was provided to the stakeholders . . . . . . . . . 11
2.5.3 Use of the preference information in a problem analysis . . . . . 12
2.5.4 Preference information from stakeholders . . . . . . . . . . . . . 12

3 Problem analysis 13
3.1 The purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Individual stakeholder analysis . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Analysis of results corresponding to stakeholders’ preferences . . . . . . 14

4 Requirements for MCA-Needs 15

4.1 The user perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2 Infrastructure and organization . . . . . . . . . . . . . . . . . . . . . . . 17

5 Implementation 18
5.1 Overview of the problem analysis . . . . . . . . . . . . . . . . . . . . . 18
5.2 Basic assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.2.1 General assumptions . . . . . . . . . . . . . . . . . . . . . . . . 19
5.2.2 Specification of the user preferences . . . . . . . . . . . . . . . . 20
5.3 Architecture of the Web-based MCA . . . . . . . . . . . . . . . . . . . . 21
5.4 Summary of the MCA implementation . . . . . . . . . . . . . . . . . . . 24
5.4.1 Versions of the software . . . . . . . . . . . . . . . . . . . . . . 24
5.4.2 Summary of the basic features of the MCA . . . . . . . . . . . . 25
5.4.3 Summary of the issues specific to the MCA-NEEDS . . . . . . . 26
M. Makowski et al. - viii - MCA-Needs: requirements&implementation

6 Experience 26
6.1 Beyond the state-of-the-art . . . . . . . . . . . . . . . . . . . . . . . . . 26
6.2 Lessons learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

7 Conclusions 28

References 30

A Selected applications of MCDA to energy planning 33

A.1 Prior MCDA applications in the electric and energy sectors . . . . . . . . 33
A.1.1 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . 34
A.2 CETP (China Energy Technology Program) . . . . . . . . . . . . . . . . 36
A.2.1 Multi-scenario, multi-attribute tradeoff analysis . . . . . . . . . . 36
A.2.2 Concordance/Discordance MCDA analysis using stakeholder
preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
A.2.3 Interactive weighted average MCDA for DVD presentation . . . . 36
A.3 MARKAL goal programming formulation . . . . . . . . . . . . . . . . . 37
A.3.1 The substantive model . . . . . . . . . . . . . . . . . . . . . . . 37
A.3.2 The preference structure . . . . . . . . . . . . . . . . . . . . . . 38
A.4 Reference point method - an energy planning model for Vienna . . . . . . 39
A.4.1 The substantive model . . . . . . . . . . . . . . . . . . . . . . . 39
A.4.2 The preference specification . . . . . . . . . . . . . . . . . . . . 40
M. Makowski et al. - ix - MCA-Needs: requirements&implementation

List of Tables

1 Illustration of specification of alternatives. . . . . . . . . . . . . . . . . . 10

M. Makowski et al. - x - MCA-Needs: requirements&implementation

List of Figures

1 General structure of the problem analysis process. . . . . . . . . . . . . . 3

2 Hierarchy of criteria and indicators. . . . . . . . . . . . . . . . . . . . . 9
3 Main components of the analysis process of the future energy technologies. 19
4 Architecture of MCA (hardware view). . . . . . . . . . . . . . . . . . . . 21
5 Architecture of MCA (software view). . . . . . . . . . . . . . . . . . . . 22
6 The main user-interface screen. . . . . . . . . . . . . . . . . . . . . . . . 23
7 The JIRA screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
M. Makowski et al. - 1 - MCA-Needs: requirements&implementation

Requirement Analysis and Implementation

of Multicriteria Analysis in the NEEDS Project
Marek Makowski* ([email protected])
Janusz Granat** *** ([email protected])
Hongtao Ren* ([email protected])
Warren Schenler**** ([email protected])
Stefan Hirschberg**** ([email protected])

1 Introduction
The purpose of this report is to provide an analysis of requirements for a fairly complex
process of multicriteria analysis of European future energy technologies done by a large
number of diversified stakeholders, and to summarize how these requirements were actu-
ally met by implementing a dedicated Web site combined with a suite of supporting ap-
plications. The activities described in this report were done within the EU-funded project
NEEDS ”New Energy Externalities Developments for Sustainability”.1 The requirements
presented here show the complexity of the process, and the corresponding research and
technological challenges. Therefore the lessons from the approach to public participa-
tion described in this paper are of interest of researchers and practitioners involved in
science-based support for policy making.
The MCA-Needs has been developed within the Research Stream 2b (RS2b) of the
NEEDS project, and was thus a part of a fairly large research activity, which in turn was a
component of a very large integrated project in which over 70 institutions took part. The
general objectives of the RS2b were:

1. To identify, discuss and analyze the terms and conditions for an effective formula-
tion and implementation of long term energy strategies.

2. To broaden the basis for decision support beyond the assessment of external costs
by examining the robustness of results under various stake-holder perspectives.

3. To contribute to the integration of results by other analytical tasks with the NEEDS
project.
*
Integrated Modeling Environment Project, IIASA.
**
National Institute of Telecommunications, Warsaw, Poland.
***
Warsaw University of Technology, Warsaw, Poland.
****
Paul Scherrer Institut, Villigen, Switzerland.
1
The documentation of the NEEDS project is available at http://www.needs-project.org.
M. Makowski et al. - 2 - MCA-Needs: requirements&implementation

The specific objectives relevant to the development of MCA-Needs were:

1. To evaluate energy technologies and scenarios taking into account diverse prefer-
ences of a large group of stakeholders for trade-offs between economic, environ-
mental, and social criteria characterizing the technologies.

2. To investigate the sensitivity of the results of sustainability assessment to specific

patterns in stake-holder preferences.

Multi-criteria analysis was therefore a key element of the RS2b research and the re-
sulting policy recommendations. The original plan was to select a MCDA (Multi-Criteria
Decision Analysis) approach and software best suited for the purpose of NEEDS. When
making this choice, the arrangement of interactions with stakeholders needed to be taken
into account. The original report [13] provided a basis for the report [5], which in turn
justified the need for development of new methods for multicriteria analysis of the corre-
sponding class of discrete alternative problems, and a new Web-based tool enabling such
an analysis. An updated version of [5] is available as [6], which also provides information
about the new MCDA methods developed and implemented first for the NEEDS project,
and later extended for a multicriteria analysis of any problem of discrete choice.
Analyses of European future energy technologies implicitly defined a class of multi-
criteria analysis problems composed of large sets of alternatives, each characterized by
a large number of criteria organized in a hierarchical structure. The criteria are diversified
and conflicting, and are organized into three sets composed of economic, environmental,
and social criteria respectively. Each of these sets has the hierarchical structure of the
corresponding criteria. Moreover, the analysis has been done by different stakeholders
who not only have different preferences for trade-offs between such criteria, but also di-
versified backgrounds and thus typically very limited experience in analyzing problems
using formal multicriteria approaches, and especially in defining preferences. Therefore
suitable MCDA methods and corresponding modeling tools were necessary for reaching
the key objectives of the RS2b.
This report is composed of the selected (and updated) elements of the original Require-
ment Analysis [13] and new parts that summarize the implementation of the dedicated
Web-based application (here called MCA-Needs) developed for meeting the requirements
of multicriteria analysis of future energy technologies done by the stakeholders invited by
the EU-funded NEEDS project.
The report is composed of two main parts:
• a requirement analysis for the MCA-Needs, and
• a summary of the MCA-Needs implementation and lessons from its actual use.
We now summarize the structure of these two main parts.

Content and structure of the requirement analysis: This part of the report is orga-
nized as follows: Section 2 provides a detailed specification of the problem to
be subjected to multicriteria analysis; this includes the summary of the analysis
context, discussion of the sets of criteria and alternatives, and the participation of
the stakeholders. The implemented problem analysis process is summarized in Sec-
tion 3. Section 4 specifies the requirements for multicriteria analysis; it is composed
of two parts: (1) the user perspective, and (2) infrastructure and organization. In
addition, the Appendix contains the characteristics of three energy applications that
M. Makowski et al. - 3 - MCA-Needs: requirements&implementation

exploit different multicriteria analysis methods pertinent to the multicriteria analy-

sis in NEEDS RS2b.

Implementation and experience: Section 5 summarizes the basic assumptions adopted

for the implementation, and the key features of the dedicated Web-based application
for the multicriteria analysis called MCA-Needs, its architecture and description of
its implementation. The lessons learned from implementation and use of the MCA-
Needs are provided in Section 6.

Additionally, the appendix contains a review of the state-of-the-art of applying mul-

ticriteria analysis to energy problems, as well as characteristics of three applications that
exploit different multicriteria analysis methods for energy problems considered relevant
to the analysis reported in this paper.

2 Problem specification

Hierarchy of
the criteria
Stakeholders
Analyst
Preferences
Alternatives

MC Analysis

Clustering Ranking

Reports

Decision makers

Figure 1: General structure of the problem analysis process.

This Section starts with a top-level summary of the class of problems for which the MCA-
Needs has been developed.2 The general structure of the problem analysis process is
illustrated in Figure 1 and is characterized as follows:3
2
The context of such an analysis is summarized in Section 2.1.
3
The characteristics provided here differs slightly from those included in the original requirement anal-
ysis [13] because the latter had to be adjusted to the actual developments in the NEEDS project.
M. Makowski et al. - 4 - MCA-Needs: requirements&implementation

1. A set of approximately 60 criteria is given, organized in a hierarchical structure of

up to 4 levels. Two types of criteria are distinguished:
• Indicators (also called attributes), i.e., the criteria for which values were deter-
mined for each alternative. Indicators are therefore included into the hierarchical
criteria as the leaf-criteria.
• Higher4-level criteria. The higher-level criteria have no physical values; they only
serve for specifying multiplicative preference in the criteria tree. There are three
highest-level criteria, each corresponding to one of the three sustainability pillars,
i.e., environment, economy, and society.
There are about 40 attributes/indicators (lowest-level criteria), and about 20 higher-
level criteria. Moreover, the criteria are of two (mathematical) types: quantitative
and qualitative.

2. Four sets of discrete alternatives corresponding to energy technologies (each set

composed of about 20 alternatives) are given. Each set corresponds to the tech-
nologies and their characteristics developed for the corresponding country.5 Each
alternative is composed of an identifier and indicators’ values (either numerical or
qualitative).

3. The preferences of diversified stakeholders are elicited through the Web-based mul-
ticriteria analysis. Each stakeholder has a private data space for her/his preferences
and the corresponding analysis. The preferences are expressed in terms of relative
importance of the criteria.

4. Analysis of individual preferences and the corresponding solutions has been done
by experts in the energy domain, policy-makers and advisors, members of non-
governmental organizations, and researchers working in different areas.

5. The outcome of a multicriteria analysis of alternatives performed according to the

preferences of each stakeholder has been used by the experts for the second stage
of analysis.

6. At least two types of outcome from the multicriteria analyses done by the stake-
holders were desired as the input to the second stage analysis:
• Information about individual preferences for technologies analyzed indepen-
dently; if possible this information should include a ranking (full or partial, ordi-
nal or cardinal).
• Clusters of technologies, each matching a cluster (corresponding to a selected
similarity measure) of the preferences of stakeholders.

From an analytical perspective the class of problems is characterized by:

1. A medium-size set of alternatives, which was however clearly too large to consider
methods using pair-wise comparisons by the users.

2. A large set of criteria organized in a hierarchical structure.

4
Than the leaf/lowest level.
5
The study was done for France, Germany, Italy, and Switzerland.
M. Makowski et al. - 5 - MCA-Needs: requirements&implementation

3. Criteria having diverse characteristics, including:

• multimodal distributions of values,6
• different types of criteria, i.e., numerical and ordinal.

4. A large number of diversified stakeholders with substantially different preferences;

most stakeholders have rather limited skills in using formal analytical methods.

5. The need for analysis reflecting diverse preferences in a fair way.

The remaining part of this Section is organized as follows: We begin with a summary
of the context of analysis (Section 2.1) and follow with an outline of the elements of the
analysis in Section 2.2. The sets of criteria and alternatives are discussed in Sections 2.3
and 2.4, respectively. The problem specification is completed by the discussion of issues
pertinent to the stakeholders in Section 2.5.

2.1 Context
2.1.1 The NEEDS Project, relevance to energy/electricity sectors and importance
The NEEDS project was intended to address the sustainability of electricity generation
technologies and systems in a comprehensive, multi-criteria way, thus it focused on the
use of multicriteria analysis as an essential methodology to assist individual decision-
makers and groups in balancing the competing characteristics of different options in order
to reach an option ranking in accordance with their preferences.
The electric industry is an important part of the overall energy sector for many rea-
sons. Electricity serves as an energy carrier that transfers primary energy from many
diverse sources to provide customers with a very wide range of end-user services. It is
a uniquely flexible and high quality form of energy that is irreplaceable in many appli-
cations. Because of this, it has an important and increasing share of the end-use energy
market. The sheer scale of its use means that the electric supply industry has a very large
infrastructure with a wide range of significant impacts in all three areas that traditionally
comprise sustainability, i.e. the economy, the environment and society. Such impacts
include internal and external costs to customers and society, environmental burdens like
airborne emissions, toxic and nuclear waste and resource depletion, and an array of health,
risk and safety considerations. The size and life of the infrastructure also means that the
sector has a large inertia, so changes like the penetration of new primary energy sources
can take a long time to make a significant impact. As one example, electricity generation
is a primary contributor to CO2 emissions, but it is susceptible to reductions by switching
to low or zero carbon primary energy resources or carbon capture, more possible due to
the relatively low number of large, fixed (non-mobile) sources.
The NEEDS project intended to address the demand for improving sustainability in
the electricity sector by assessing a wide range of economic, environmental and societal
indicators for a range of generating technologies, and to extend this technology-specific
6
Roughly speaking, multimodal distributions are characterized by values split into several disjointed
subsets separated by empty subsets covering large ranges of values. Consider e.g., two subsets of values:
the first composed of positive values smaller than 100, and the second composed of values larger than
100000. Typical statistical characteristics of such data is usually not adequate. For example, the average
value is often far away from the closest value of a member of the set.
M. Makowski et al. - 6 - MCA-Needs: requirements&implementation

analysis to a limited number of scenarios for operating and expanding the electric sector
in the future.

2.1.2 The MCDA problem as it applies to NEEDS

We use in this document the widely used term MCDA (Multicriteria Decision Analysis)
because it covers a well developed field of Operational Research that provides methods
and tools pertinent to our problem. However, we need to stress that our problem (de-
scribed in detail below) substantially differs from typical MCDA problems (in which a
decision-maker analyzes a decision problem together with his/her preferences). We deal
with stakeholders and not with decision-makers, and these two groups are fundamen-
tally different and so are their goals. Thus the aim of the analysis of the problem under
consideration is to find relationships between the preferences of diversified groups of
stakeholders and corresponding solutions (technologies or scenarios). This analysis has
some similarities with decision-making therefore we also briefly discuss here multicriteria
analysis of decision-making.
Decision-making is difficult in the electric sector for many reasons. The scale of the
problem is very large, the range of impacts is very broad, and the time-scales involved
can range from seconds (or less) for operation to decades (or more) for planning. In
particular, the scale of the planning problem means that it is a very important one for
many people, and the broad range of impacts means that there are inherent trade-offs
between competing characteristics (e.g., cost versus emissions) - which in turn means
that stakeholders are unlikely to agree. Different input assumptions, uncertainties and
attitudes towards risk all further contribute to this disagreement. In particular, different
groups such as utilities, regulators, customers and environmentalists all have different
interests at stake (hence the term ’stakeholder’) and their different points of view lead
them to have significantly different opinions on how different planning options should be
ranked, or what may be the best strategy for system operation or expansion. To further
complicate matters, no single decision-maker exists. Instead, there are decision-makers
within each stakeholder group, and these groups interact within a public policy arena
where negotiation and political processes are the rule.
The NEEDS project contributed to support this complex decision-making process pri-
marily by supplying it with a common basis of trustworthy information. The MCDA task
in NEEDS particularly helped the decision-makers analyze technological alternatives and
strategies, consistent with their preferences expressed in terms of the diversified set of
criteria, in a clearly understandable and trustworthy way.

2.1.3 Why MCDA is needed

MCDA assists decision-makers in several different ways, according to the main problems
experienced in making decisions on complex systems. In particular, the goal is to help
make the decision-making process structured, explicit, clear and correct, so that not only
is the ranking of alternatives right for each decision-maker’s preferences, but the entire
process serves as a clear basis for debate with others. Some of the typical problems are
very briefly mentioned below:
Attainable goals: In order to make a good decision, it is necessary to specify preferences
that lead to attainable goals (i.e., feasible values of criteria). This means it is neces-
M. Makowski et al. - 7 - MCA-Needs: requirements&implementation

sary to clearly establish priorities and trade-offs between competing goals. MCDA
assists in this by using clear procedures to establish preferences, and identifying a
solution that best corresponds to the specified preferences.
Cognitive limits: Most people can intuitively select an alternative from a small set by
considering a small number of criteria. But for a large number of alternatives
and/or criteria intuition and/or experience need to be supported. This problem is
exacerbated by mixing quantitative and qualitative indicators and preferences that
are often discontinuous, non-linear, and have threshold values. MCDA provides
an analytic structure that can clearly indicate why a given set of preferences (ex-
pressed in terms of criteria) results in a certain efficient solution; in some cases a
certain ranking of alternatives can also be provided.
Preconceptions: It is typical for a decision-maker’s initial preferences (expressed in
terms of criteria) to result in a selection of alternatives that is inconsistent with
the stakeholder’s own preconceived characteristics of a solution. The stakeholder is
confronted with the choice of modifying his/her expectations about the solution, or
her/his preferences (or both) until a consistent result is achieved. Only multicriteria
analysis can really demonstrate such inconsistencies, and assist in resolving them
iteratively.
Group differences: It is rational for a heterogeneous stakeholder group to disagree, and
often necessary for them to reach some joint decision. MCDA can clarify the rea-
sons for disagreements and form a basis for discussions and negotiations. Some
MCDA methods are also more formally combined with joint-resolution methods
(although not in this project).

2.2 Elements of multicriteria analysis

Multicriteria analysis is composed of several interlinked elements, which do not form a
linear process, i.e., some of them are developed in parallel, and/or can be modified during
the analysis. We list these elements in the order in which they are discussed below:
1. Definition of criteria.
2. Definition of alternatives.
3. Preparation of the problem analysis.
4. Problem analysis composed of a sequence of two steps:
• specification of user/stakeholder preferences, and
• finding a solution (an alternative, or a ranking of alternatives) best corresponding to
the preferences.
5. Analysis of results.
The essence of multicriteria analysis is actually the sequence of modified preferences
based on the analysis of solutions corresponding to previously specified preferences. The
reason for such a sequence is the commonly known fact that typically, (especially in
an initial stage of problem analysis) solutions corresponding best to preferences differ
substantially from the expectations of the user. Therefore the user typically needs to
modify his/her preferences in order to find solutions that are close enough to the goals
(values of criteria) that are both attainable, and have trade-offs between criteria that fit the
user preferences.
M. Makowski et al. - 8 - MCA-Needs: requirements&implementation

2.3 Set of criteria

Criteria are used for measuring the performance of alternatives. Therefore the choice of a
set of criteria is of primary importance for the analysis of alternatives. The set of criteria
used in the MCA-Needs resulted from a comprehensive study made by the RS2b team that
is documented in [1, 9, 19, 20, 22]. Moreover, the survey on criteria [2] provided feedback
from the stakeholders. Therefore here we only summarize the main considerations taken
into account by this study.
The main considerations that were taken into account for the choice of specific indi-
cators included:
• Results of a literature survey on sustainability indicators in general (past experience).
• Social indicators were developed within a dedicated activity, which was a pioneering
effort that also included a Delphi exercise.
• Catching the essential characteristics of technologies and enabling differentiation be-
tween them.
• Assuring that indicators are representative (but not necessarily complete).
• Keeping the number of indicators at a reasonable level and striving for a certain balance
in terms of the number of indicators.
• Trying to avoid excessive overlapping.
• Aiming at limited aggregation of indicators provided that this involves no or minimum
subjectivity.
• Assuring practicability and feasibility; in particular having confidence that the values of
indicators would be available on-time to be used in the analysis.
Important features of the proposed set included:
• The selected indicators are distributed between the three sustainability dimensions, i.e.,
environmental, social and economic.
• The overall structure and selection made resulted from the stakeholder survey.
There are different types of indicators (e.g., quantitative and qualitative), some of them
having a multimodal distribution of values. Based on past work on similar projects consid-
ering sustainability indicators, the final set of indicators (for each country) is around 40.
Such indicators are often divided into the three pillars of sustainability, i.e., those relating
to the general areas of the economy, environment and society.
The hierarchical structure of the criteria and indicators is illustrated in Fig. 2. The
indicators are included into hierarchical criteria structure having up to four levels within
each of these three pillars. This structure is somewhat relevant to the ranking analysis,
but more important to preference elicitation, where the addition of such a structure can
give decision-makers a framework for specifying their preferences. Thus the indicators
are considered as the lowest-level criteria, and about 20 higher level criteria have been
defined. The higher-level criteria have no physical values, they only serve for specifying
multiplicative preference in the criteria tree. Note that preferences cannot be specified
effectively7 for criteria that have only one child criterion; the circles attached to the criteria
names in Fig. 2 indicate that preferences were specified for the corresponding criterion.
Therefore altogether about 60 criteria were defined. For issues of sustainability, 60
might also be considered a reasonable upper bound on the number of criteria, because
7
Because the normalized importance weight for such a criterion is equal to 1, irrespectively of the
relative criterion importance that would be specified by the user; see [7] for the definition of weights in the
criteria hierarchy.
M. Makowski et al. - 9 - MCA-Needs: requirements&implementation

Figure 2: Hierarchy of criteria and indicators.

adding further criteria dilutes the impact of those already present. For quantitative at-
tributes, the value represents an actual value of the indicator. For qualitative indicators,
the value corresponds to an order, i.e., a (real or integer) number inducing an order within
the set of admissible values of the corresponding indicator (e.g., very bad, bad, average,
good, very good). It is assumed that for maximized criteria the higher number corre-
sponds to a higher preference. Partial order is allowed (i.e., duplicate values of the order
attribute).

2.4 Sets of alternatives

A set of alternatives was developed for each of the four countries. The alternatives define
corresponding sets of electric generation technologies. The set of alternatives is broad
enough to be interesting to all stakeholders, and specific enough that results are calculable.
The definition of alternatives includes both common and technology-specific assumptions
(e.g., fuel prices and fuel efficiencies), system boundaries, etc. The number of generation
M. Makowski et al. - 10 - MCA-Needs: requirements&implementation

technologies is about8 20 for each country. This covers a range of technologies for the
different fossil fuels, nuclear plants, and a selection of renewable generation options.

2.4.1 Preparation of alternatives

Each alternative/object is described by values of criteria (often called indicators). Each
criterion is either numerical (quantitative or cardinal) or non-numerical (qualitative or
ordinal). The following table illustrates the data content for a set of m alternatives in
the form of a matrix, each identified by an identifier (here defined as alternativei) and
characterized by values of n criteria; in other words, the value of j-th criterion for i-th
alternative is defined by vi,j .

criteria criterion1 criterion2 . . . criterionn

alternative id
alternative1 v1,1 v1,2 ... v1,n
alternative2 v2,1 v2,2 ... v2,n
... ... ... ... ...
alternativem vm,1 vm,2 ... vm,n
Table 1: Illustration of specification of alternatives.

The values were defined in the units specified for each indicator. No scaling was ap-
plied to the indicators’ values of alternatives during the data preparation and verification.
The assigned values were transformed into achievement measures during the problem
analysis.

2.5 Stakeholders
2.5.1 Context
Planning and decision making in the electric power sector should consider stakeholder
preferences. Thus to achieve a reasonable quality of analysis (that could be a major factor
in a decision-making process) it is critically important to adequately represent the stake-
holder preferences. However, this is not only important but also a very difficult issue
because of two sets of problems.
Firstly:
• Preferences are substantially different for different groups of stakeholders.
• Stakeholders typically do not have experience in the processes of formal analysis.
• Stakeholders have diverse backgrounds, thus not many of them were able/willing to
specify preferences for all criteria (that are specified on the lowest level in a rather
detailed way).
• The rather short time period (between the set of alternatives being available and when
the results of analysis are due).
Secondly, it is known (from the properties of the mathematical programming problem
corresponding to any method of analysis of our problem) that:
8
Difference between countries reflect the actual resource availability and operating conditions.
M. Makowski et al. - 11 - MCA-Needs: requirements&implementation

• The relation between changes of preferences and the corresponding changes of solutions
is nonlinear and discontinuous, i.e., in many situations even large changes of preferences
do not result in (substantial) changes of solutions, while in other situations a very small
change of preferences results in a qualitative change of a solution.
• Even for problems that are easier from the mathematical programming viewpoint (e.g.,
continuous linear models), preferences specified by a stakeholder may result in solu-
tions inconsistent with his/her preconceived characteristics of the corresponding solu-
tion; such inconsistencies should be resolved and this is only possible by either changing
the preferences or expectations about a satisfactory solution. This is a typical situation,
and the main argument for interactive problem analysis.
It is commonly agreed that elicitation of stakeholders preferences must include com-
puterized interaction with each stakeholder during which she/he is supported in the analy-
sis of the correspondence between her/his desired goals and the corresponding attainable
outcomes/results. However, it is also commonly agreed that designing and implementing
an effective interaction is a challenging task. The challenge is due to the fact that elicita-
tion of preferences is based on learning about the problem through its analysis, and in the
case of many users/stakeholders this had to be unsupervised learning. Thus the interaction
has to be carefully designed to address the needs and expectations of stakeholders with a
wide spectrum of backgrounds and goals.

2.5.2 Information that was provided to the stakeholders

Each stakeholder was provided with two sets of information pertinent to specification of
preferences:
• General information common to all stakeholders, about the problem, elicitation of pref-
erences, etc.) and characteristics of the sets of alternatives. The latter should contain:
? a definition of each indicator,
? basic information regarding the value distribution of each indicator,
? basic information regarding the distribution of numbers of alternatives along values
of each indicator,
? the pay-off table (best and worst values of each indicator),
? information about clusters of solutions corresponding to ”selfish optimization” of each
indicator,9
? specification of the criteria and their hierarchical structure.
This information should be provided as part of a Web-site to be developed for on-line
elicitation of preferences.
• Individual information corresponding to various preferences specified by the stake-
holder. This information should enable a stakeholder to modify her/his preferences
until a satisfactory solution is found, and should include:
? providing a solution corresponding to a specification of preferences,
? various ad-hoc information, e.g., number of feasible alternatives for lower bounds
specified on values of a set of criteria,
? optional characteristics of classification/rankings of solutions corresponding to a set
of preferences.
9
The best value of each indicator has an associated set of alternatives. Some of these sets are likely to
overlap, and this might be interesting information.
M. Makowski et al. - 12 - MCA-Needs: requirements&implementation

2.5.3 Use of the preference information in a problem analysis

In order to justify the proposed requirements for the elicitation of stakeholder preferences
we need to summarize how preference information is used.
The preference information is used to compute parameters of a scalarizing function.
There is a class of formalized methods for model10 analysis that uses preference informa-
tion for the calculation of parameters of a scalarizing function, i.e., a function that asso-
ciates a number with each solution (an alternative in our case) that measures the quality
(goodness) of the solution. For a multicriteria analysis such a function maps a multi-
dimensional space of criteria into a one-dimensional real-number space that induces (at
least partial) order in the solution space. Therefore, various multicriteria methods differ
by:
• specification of scalarizing function,
• mapping of the preferences into parameters of the selected scalarizing function.
Thus, the properties of various methods of multicriteria analysis can be considered by
examination of the properties of:
• the corresponding scalarizing function,
• the properties of sets of criteria values.
The latter is especially important for the analysis of discrete alternatives with multi-
modal distribution of criteria values.
Typical users do not consider the mathematical properties of their problems. They
reason in terms of trade-offs between criteria. Such trade-offs alter with changes of cri-
teria values (e.g., a ”weight” for costs is much higher for ”expensive” alternatives than
for ”cheap” ones). Therefore specification of trade-offs is often done for a given solution
rather than for the whole range of criteria values (e.g., how much more am I willing to
pay for an alternative which has a lower emission of pollution). Moreover, users analyze
the quality of a solution in terms of the acceptability of the values of criteria (e.g., is the
cost within my budget?, is the emission level acceptable?). Hence, users focus on a sub-
set of criteria whose values the user considers unsatisfactory, and try to improve them by
changing preferences. Of course, by improving the value of even one criterion, the value
of at least one other criterion must worsen, and this may be not acceptable, which in turn
calls for another modification of preferences.
This short summary of the essence of multicriteria analysis shows that an interactive
procedure is practically required for a proper specification of preferences.

2.5.4 Preference information from stakeholders

Generally, the information provided by a stakeholder needs to be sufficient to represent
his/her preferences in terms of criteria (e.g., as trade-offs between criteria values). These
trade-offs are typically different for various ranges of criteria values at the lowest hierar-
chy level (i.e., corresponding to the attributes). Consider, for example, trade-offs between
changing values of two minimized criteria, cost and emission level. Such trade-offs are
typically different for:
• expensive, medium-cost, and cheap solutions, and/or
• large, medium, low emission levels.
10
A set of alternatives can also be considered/represented as a model.
M. Makowski et al. - 13 - MCA-Needs: requirements&implementation

For expensive solutions (and the corresponding low emission levels) a substantially
stronger preference is attached to the cost reduction than for cheap solutions. In other
words, the relative importance of costs (compared to the emission level) is typically much
higher for expensive than for cheap solutions. Similarly, the relative importance of the
emission level criterion decreases when one moves from high to low emission levels.
There are several ways of dealing with trade-off specification. For our problem (char-
acterized by large numbers of criteria and alternatives) approaches based on pairwise
comparisons are not practicable. This reduces the choice of methods for trade-off specifi-
cation to:
• Direct specification of weights (for criteria and for scalarizing functions).
• Indirect specification of weights, e.g., by specification of relative importance of criteria.
• Indirect specification of parameters of scalarizing functions by selection of:
? aspiration (the desired criterion value) and reservation (the worst criterion value the
stakeholder is willing to accept) values for each criterion, or
? aspiration or reservation values for each criterion, and information about trade-offs
between criteria at the selected aspiration (or reservation) point.
Additionally, the following preference information from stakeholders could be useful
for a better support of the preference elicitation process:
• Specification of an acceptability (threshold/veto) level for criteria (equivalent for reject-
ing alternatives having worse values of the corresponding criteria).11
• Optional specification of sets of compensatory criteria. Criteria are compensatory when
an increase of the value of one of them by a given value from a relative scale can be
rationally substantiated to compensate a deterioration of another criterion.
• Optional specification of trade-offs between a selected subset of criteria (e.g., answer-
ing questions like ”if you want to improve the value of this criterion then select crite-
rion/criteria you agree to worsen.”)
• Optional, based on intuition, selection of best and worst alternatives. This information
was not used for the representation of stakeholder preferences; it can be used in the final
analysis of the problem, including various characteristics of stakeholders.
Elicitation of stakeholder preferences was done through the Web-base interactive mul-
ticriteria analysis tool called MCA-Needs described in Section 5. Moreover, for the sec-
ond stage of analysis (done by analysts) some information about profile of each stake-
holder was required. Organization of this process is also discussed in Section 5.

3 Problem analysis
3.1 The purpose
The purpose of the NEEDS project was to support the EU decision-makers who can influ-
ence expansion planning for the electric generation sector. The decision-makers need to
make good quality decisions, consistent with their preferences, also taking into account
the preferences of stakeholders. NEEDS was intended to support decisions that enhance
sustainability in the electric sector, and ensure that a quality information base exists to
support these decisions.
11
This approach appears to be a much better way of eliminating some alternatives, than to attempt to do
so by playing with weights/reservations.
M. Makowski et al. - 14 - MCA-Needs: requirements&implementation

The report [21] summarizing multicriteria analysis was a major factor in such a
decision-making process. The purpose of the multicriteria analysis using MCA-Needs
was to provide a basis for analysis of future energy technologies, and to report on stake-
holder preferences on criteria characterizing them; these preferences were expected to
substantially differ amongst groups of stakeholders. Thus the analysis attempted to fairly
account for these differences and resulted in clusters of solutions corresponding to clusters
of preferences. It was however beyond the scope of the NEEDS project work to attempt
any type of analysis needed for supporting a group decision-making process, consensus
building, or negotiations. However, the authors stress that such an analysis would sub-
stantially enhance the quality of the decision-making process. Thus, the main target of
the MCA-Needs remained to enable a multi-criteria based analysis of a set of generation
technology alternatives.
The analysis done by the RS2b team was composed of two stages, each of them sum-
marized in the following subsections.

3.2 Individual stakeholder analysis

This analysis was done by each stakeholder individually through the Web-based applica-
tion supporting interactive multicriteria analysis of alternatives (using data provided by a
data server, see Section 4.2). The functionality of this application is documented in detail
in [12].
The basic result of each individual analysis was a Pareto efficient solution (a technol-
ogy) that corresponded best to the stakeholder preferences (expressed in terms of relative
importance of criteria), and the corresponding trade-offs between values of the underlying
attributes. Additionally, information about the corresponding ranking of alternatives was
provided although it was known that due to the nature of the problem a ranking may not
be stable, i.e., even small changes of preferences can result in rather different rankings.

3.3 Analysis of results corresponding to stakeholders’ preferences

This analysis was done by the PSI energy experts in consultation with IIASA modeling
experts, and is documented in [21]. The results of this analysis were used in the report
submitted to the decision-makers and made available to the stakeholders.
The following types of analysis were explored:
• Clustering of preferences (and possible correlations with categories of stakeholders) for
various similarity measures given by the analysts.
• Analysis of possible correlations between clusters of preferences and clusters of the
corresponding results.
• Analysis of distributions of solutions (technologies/scenarios).
• An attempt to find (possibly partial) rankings, if stable rankings are possible for the
given sets of alternatives and stakeholder preferences.
• Clusters of solutions (technologies/scenarios) corresponding to clusters of preferences
(the latter possibly correlated with clusters of stakeholders’ categories).
• A partial ranking of solutions (technologies/scenarios) within clusters of solutions.
• Identification of a subset of ”stable” solutions (those which are typically either ”very-
good” or ”bad” or ”in the middle”), and ”jumping” solutions (which for small changes
of preferences are either good or bad).
M. Makowski et al. - 15 - MCA-Needs: requirements&implementation

Given the characteristics of the problem, the following types of analysis were not
possible:
• Aggregation of stakeholders preferences, and using them as ”representative” prefer-
ences for multicriteria analysis of alternatives.
• Reliable rankings of solutions.

4 Requirements for MCA-Needs

The requirements are summarized here from two perspectives: functionality from the user
point of view, and requirements for the infrastructure and organization.

4.1 The user perspective

An appropriate elicitation of stakeholder preferences is typically difficult but – as dis-
cussed in Section 2.5 – it is especially challenging for the problem described in this report.
Therefore we provide here a much more detailed (than for other elements of the analysis
described in this Section) justification and description of the process.
Communication with stakeholders is extremely difficult because there is a gap be-
tween the information that is required by the analysis method and the language in which
the problem is communicated and understood by stakeholders. Therefore, the commu-
nication method is a key element in gathering proper information from the stakeholder,
using it in the decision process and communicating the results of the decision. Moreover,
a process of preference specification is not stationary, i.e., even very experienced users of
multicriteria analysis tools change their own preferences in a rather discontinuous/erratic
way. Therefore it is important to repeat here the arguments presented in Section 2.5 that
justify the need for an interactive (repetitive) process of elicitation of preferences. This is
a necessary condition to acquire a reasonably good representation of stakeholder prefer-
ences.
Given the large number of stakeholders (over 3,000 were invited), it was decided
that the elicitation of stakeholder preferences would be done via a Web-based interactive
multicriteria analysis, which could provide diversified characteristics of a solution cor-
responding to a current specification of preferences, and helped (by providing pertinent
information) to modify those preferences in a way that the next solution would better fit
the expectations/preferences of the stakeholder.
Both the number of stakeholders invited to make the MCA, and the tight time-table of
the whole NEEDS project (which left a rather short period of time between the alternative
description data being available and the results of the MCA being due) made a Web-based
survey directly linked with a data server to be the only solution acceptable from the project
management point of view. After a careful analysis, the following three key assumptions
were made for the MCA that served as the survey of the stakeholders’ preferences:
• Survey size - over 3,000 of stakeholders were invited to use the MCA-Needs for
multi-criteria analysis of the assigned problems (technologies in a selected country
from the four countries: France, Germany, Italy, and Switzerland).
• Survey form - Due to the large number of stakeholders invited to the survey it was
clear that it was impossible to perform individual preference elicitations, either in
M. Makowski et al. - 16 - MCA-Needs: requirements&implementation

person or by phone. Therefore it was decided to develop MCA-Needs, the Web-

based interactive multi-criteria analysis tool.

• Scope of survey - the survey length due to the response rates has two effects on
the choice of multicriteria analysis method. It would be desirable to use more than
one multicriteria method on the alternative and preference data, in order to compare
how well the different rankings corroborate each other. However, this would require
elicitation of preferences needed by each analysis method. Therefore it was decided
to examine several methods but to provide all the stakeholders with only one method
selected by the PSI team.

It was agreed that the following features of the multicriteria analysis method, and its
implementation were desirable:

Ease of use: The MCA-Needs was used by both stakeholders (who are typically not ex-
perienced in analytical tools) and experienced analysts. Therefore specification of
preferences had to be done in terms that were understood without knowledge of op-
erational research. Also explanations of all pertinent terms (used for specification
of preferences, and for the definition of criteria and alternatives) had to be easily ac-
cessible through hyperlinks in the Web-based MCA. Moreover, preferences needed
to be specified through a user-friendly interface. Finally, for the Web-based MCA,
at least a Pareto-efficient solution corresponding to the specified preferences had to
be easily available; preferably, assistance in assessing trade-offs between criteria
should also be provided.

Transparency: Transparency focuses on the two elements of clarity (easy to understand),

and trustworthiness. These both follow along the analytic chain, so it should be
easy to see and trust:
• the input assumptions for the analysis of alternatives,
• the analytic process (e.g. the modeling methodology),
• the multicriteria analysis method.

Treatment of preferences: A stakeholder should be confident that the analysis method

conforms to the form of his preferences, not the other way around. Preferences for
thresholds, vetoes, non-linear scalarizing functions, etc. should be addressed.12

Ease and speed of iteration: Using the method and the corresponding tool should be a
learning process, and the first specification of preferences should be the start of an
exploration process. The iteration process had to be quick and interactive in order
to satisfy the stakeholders and motivate them to spend more time in refining and
verifying their preferences.

The following, more specific features, were also considered:13

1. Is the considered method and the corresponding tool available and can be adapted
with a limited amount of resources, or does the method need to be developed and the
tool to be implemented ?
12
Note, that this requirement contradicts the requirement for the Ease of use, because informed specifi-
cation of such parameters requires analytical skills.
13
It is clear that there is no method which conforms to all of these criteria.
M. Makowski et al. - 17 - MCA-Needs: requirements&implementation

2. Availability (free or license, restrictions, price).

3. Has method/tool been successfully used for energy applications relevant for NEEDS?
4. Simplicity, transparency, easy to use, interactivity.
5. Mathematical correctness.
6. Internal consistency checks.
7. Suitable for large amount of applications.
8. Processing, analysis and presentation of results.
9. Sensitivity analysis capability.
10. Compatibility with the intended elicitation of preferences.
11. Possibility to use “simulated” typical preference profiles.
12. Expandability in the future.
13. Can minority views be considered?
14. Non-discriminatory treatment of technologies.
Moreover, it was necessary to decide on whether to provide the stakeholders with one
or more multicriteria analysis methods. There are advantages and disadvantages in both
cases. Stakeholders with analytical skills would most likely prefer to make analysis using
several methods while others would likely be confused when confronted with several
methods even if they shared the same way of specifying preferences.

4.2 Infrastructure and organization

The many participants of the analysis process (stakeholders), the serious time constraints
(short time between availability of data and required output), and the many diversified
data flows involving various teams implied the necessity for an efficient computing in-
frastructure.
Depending on the final selected method, the required computing resources might have
been substantial (especially, if many stakeholders would perform interactive analysis si-
multaneously); therefore it was desired to have a possibility for the easy use of a computa-
tional grid when a peak of computations occurs. Such a computational grid was prepared,
based on SGE (Sun Grid Engine), and a network of unix workstations.
Before the MCA-Needs was ready for extensive testing it was not clear how many
computing resources were actually needed for the on-line evaluation of preferences. How-
ever, it was clear that data handling posed a serious challenge unless an effective data
server was provided. The most rational solution appeared to be a Web-based data server
handling all data necessary for the analysis, including:
• Definition of criteria and alternatives, including all necessary dictionaries, see Sec-
tion 2.3.
• Data needed for Web-based MCA,
• Representations of stakeholder preferences (updated through the Web either by directly
linking the survey forms with the DB, or by a dedicated interface to be used by staff
processing paper surveys); versioning and automated documentation needed.
• Providing data for MC tools through either a direct link to the DB or upload/export of
data from/to CSV files.
Such a data-server was implemented using modern technology for the development
of Web-based and distributed applications, and was based on a transactional professional
database.
M. Makowski et al. - 18 - MCA-Needs: requirements&implementation

5 Implementation
It should be stressed that the reported activities have been a pioneering work in the field
of integrating public participation with science-based support for policy-making. While
there is a lot of experience in various forms of public participation in policy-making,
there was no attempt to involve a large group of stakeholders in interactive multicriteria
analysis. Moreover, the analyzed problem was complex by itself, i.e., there has been no
suitable method for its analysis. Therefore the team that implemented the analysis had to
cope with several interlinked challenges, including:
• development of new methods for multicriteria analysis of the underlying class of prob-
lems; the methods had to use a simple way of specification of preferences that were also
suitable for users having no experience in mathematical programming,
• designing and implementing an interface to these methods suitable both for researchers
and for stakeholders with almost no analytical skills typically used in model-based prob-
lem analysis,
• design and implementation of a Web-site for multicriteria analysis by a large number of
inexperienced stakeholders using advanced methods of multicriteria analysis.
This Section summarizes the main elements of the implementation, and discusses in
more detail those elements which are likely to be of interest of both research communities
and practitioners involved in science-based support of policy making.
The requirements for multicriteria analysis specified during the first stage of the
project, and summarized in Section 4 had to be met within the available time and re-
sources, including availability of data for specification of the underlying problem, as well
as with the state-of-the-art in both methodology of the multicriteria analysis and software
tools supporting such analysis. This in turn has determined sets of feasible decisions
regarding the actual implementation of the analysis.

5.1 Overview of the problem analysis

The structure of the analysis of future energy technologies (here called alternatives) can
be considered as a process composed of three stages:
1. Preparation of alternatives, including:
• Selection of the set of attributes characterizing each alternative, and evaluation of
the values of the attributes.
• Selection of the hierarchical criteria structure to be used for the multicriteria anal-
ysis (where the lowest level criteria were the attributes, and the three highest level
criteria were the three pillars of sustainability).
Four sets of alternatives and criteria were prepared for four countries, namely
France, Germany, Italy, and Switzerland. Preparation of the alternatives was done
by a concerted set of activities documented in detail elsewhere, therefore we don’t
comment on this part of the process here.
2. Individual multicriteria analysis of alternatives, the part of the process which is the
main focus of this report.
3. Analysis of the results obtained from the multicriteria algorithms applied to prefer-
ences of individual stakeholders. This analysis was done by the analysts to provide
M. Makowski et al. - 19 - MCA-Needs: requirements&implementation

a basis for the final report that was submitted to decision-makers and to stakehold-
ers. Clustering algorithms were applied to identification of groups of stakeholders
with similar preferences, and for clusters of the corresponding solutions. Finally,
an analysis of the characteristics of clusters of solutions has been made to detect if
rankings can be established for at least subsets of solutions (technologies or scenar-
ios).

Hierarchy of
the criteria Stakeholders
Analyst
Preferences
Alternatives
Design

CSV Data Exchange

Questionnaires WEB Interface
MC Analysis

Database

Clustering Ranking

Reports

Decision makers

Figure 3: Main components of the analysis process of the future energy technologies.

Here we focus on the multicriteria analysis of alternatives performed by the stake-

holders. The structure of this analysis is illustrated in Figure 3. The presentation of the
implementation of this analysis is composed of the following two parts discussed in the
corresponding sections:
• basic assumptions, and
• architecture of the MC implementation.

5.2 Basic assumptions

5.2.1 General assumptions
The analysts actively participated in the process of problem definition (definition of alter-
natives and criteria) as well as in the process of defining the MCA-Needs functionality.
The analyst team also provided comments and feedback for the design of the MCA-Needs,
and in particular for the specification of the way in which the stakeholders specified their
preferences.
Another requirement for the MCA-Needs was to design it in such a way that the
stakeholders were able to observe in real time the influence of his/her preferences on the
M. Makowski et al. - 20 - MCA-Needs: requirements&implementation

corresponding solutions, and then change his/her preferences until a satisfactory solution
was found. Such a process supports learning about the problem during the specification
of preferences. This approach has significant advantages over static questionnaires, and
stakeholders should be more motivated to use the Web interface. It should be stressed that
most of the multicriteria analysis methods assume interaction with the decision makers
or stakeholders. Therefore, the use of static questionnaires to elicit the preferences of the
stakeholders has limited value in comparison to an interactive tool accessible by the Web
interface, which in turn provides real-time access to a multicriteria tool operating on the
data provided by the data server.
Moreover, for a Web-based elicitation of preferences the results can be stored directly
in the database, and thus allow the stakeholder to optionally continue the analysis later.
The Web-interface also provides efficient ways of designing user-friendly surveys, includ-
ing context sensitive help and tutorials.
The data for criteria and alternatives for each of the four analyzed countries were
uploaded to the data-server. While doing this the analysts performed a consistency check
of the data loaded to the data-server, and assured that the final sets of data were suitable
for the analysis.
For the second stage of the analysis, it was necessary to collect information about
the profiles of the stakeholders. This information was collected in a separate part of the
survey (implemented by another team and on another hardware facility) in order to not
keep personal preferences together (i.e., in one database) with personal profiles of the
stakeholders.

5.2.2 Specification of the user preferences

The summary of various approaches to preference specification contained in Section 2.5.4
reflects the state-of-the-art of methodology for multicriteria analysis. However, the key
factor for selecting a way of preference specification for the MCA-Needs was the require-
ment that such a specification had to be simple enough to effectively support also the
stakeholders without analytical skills in expressing their preferences. One should also
recall that the analysis involved about 60 criteria, which obviously eliminates many of the
above outlined methods of preference specification.
After a number of discussions, the team responsible for the implementation of the
MCA for the NEEDS projects has decided to use a very simple method, namely specifi-
cation of relative importance of each criterion by selecting one of the following discrete
importance levels, each having an intuitive characteristics, namely:
• vastly less important than average,
• much less important than average,
• less important than average,
• average importance,
• more important than average,
• much more important than average,
• vastly more important than average.
It was also possible to entirely ignore a criterion. This simple approach proved to
be effective in the sense that it was not only easy to be understood and used, but also
supported analysis of the whole Pareto-set (i.e., it was possible to select each efficient
alternative), and it was relatively easy to find an alternative with a better value of a selected
M. Makowski et al. - 21 - MCA-Needs: requirements&implementation

criterion. It is interesting to note that this way of specification of preferences effectively

supports also analysis of other problems, see [7, 11] for more details.
Finally, we stress that over thirty new methods for multicriteria analysis of the under-
lying class of problems (characterized by a large number of alternatives, and of criteria)
were developed, and extensively tested. After the testing, one of these methods was made
available for the actual analysis by stakeholders. It is interesting to note that the com-
parative study of the methods (made after the survey using its results) has shown [7] that
in most cases (i.e., for distinct preferences) several different methods provided the same
Pareto-optimal alternative.

5.3 Architecture of the Web-based MCA

Figure 4: Architecture of MCA (hardware view).

MCA-Needs was developed by applying the internet and database technologies. The
general architecture of the MCA is characterized from two perspectives, namely, hardware
and software.
The hardware perspective is shown in Figures 4. The users can access the MCA
from any computer connected to the Internet and running a web browser. Most pop-
ular browsers are supported, and no browser plug-ins are required. The only require-
ment is to allow the opening of pop-up windows generated by applications run on the
www.ime.iiasa.ac.at domain. Therefore users that do not routinely allow pop-up windows
needed to change their permissions to allow opening pop-up windows by the IME-IIASA
site.
The MCA software runs on the IIASA Sun-Solaris servers. The software has modular
structure, and the modules are designed to work in the distributed environment illustrated
in Figure 4. In particular, there are:
• a www server,
• a servlet container,
• a database server, and
• a computing cluster.
In the near future there are plans to extend the MCA functionality to handle (pos-
sibly large) linear models. Optimization of such models requires substantial computing
M. Makowski et al. - 22 - MCA-Needs: requirements&implementation

resources. Therefore, a computing cluster might be configured using e.g., SGE (the Sun
Grid Engine).

Figure 5: Architecture of MCA (software view).

The software perspective is illustrated in Figure 5. The following components were

implemented14:
• web server (Apache),
• servlet container (Tomcat),
• database (Oracle),
• web clients for graphical user interface (Java, Java Server Pages),
• solvers for the multicriteria optimization (dedicated application in C++),
• Sun Grid Engine, and
• various utilities for maintaining contact with the users, reports for the organizers, etc.
(implemented in C++, Java, JavaScript, Perl, csh, and API to Jira).
The software provides the desired functionality, including handling of data, specifica-
tion of the user preferences, running the solvers of the underlying optimization tasks,15
displaying the results of analysis to the users, providing reports for the analysis adminis-
trators, etc.
Most parts of the MCA software were developed using Java technology and JSP (Java
Server Pages). The graphics are based on the jfreechart (www.jfree.org) library. The
14
In parenthesis the currently used software is noted.
15
Used for selecting Pareto-optimal alternatives corresponding to the specified preferences.
M. Makowski et al. - 23 - MCA-Needs: requirements&implementation

solvers have been implemented in C++. All MCA applications communicate through the
Oracle data base, which stores all pertinent data, including:
• configuration of the software, in particular interface between the GUI and solvers,
• data about users, including their roles, privileges, contacts,
• specification of the problems to be analyzed,
• specification of instances of each problem,
• private data space for individual analyses.
Such a solution is not only very efficient from the point of view of the performance of
the user interaction, but also at solving the underlying optimization problem; the modular
structure of the MCA components is also effective for the process of software develop-
ment and maintenance.

Figure 6: The main user-interface screen.

An example of the the main user-interface screen is shown in Figure 6. This screen
contains all information needed for a basic analysis of the Pareto-optimal alternative cor-
responding to the specified preferences. The implemented way of preference specification
is summarized in Section 5.2.2, and the selection is done by selecting a corresponding but-
ton from the panel located on the right side of the screen.16 The trade-offs between the
criteria values are shown in the parallel coordinate graph on the left side of the screen.
The controls available to the user are organized into two panels: the bottom one pro-
vides controls for additional functions for analysis of the solution related to a selected
specification of preferences, while the upper panel provides controls to access other func-
tions of the application, such as on-line documentation, contact to the developers, as well
as switching analysis to other problems and/or problem instances. Detailed description
16
The figure shows an example of a small problem. For the actual analysis the panel contains about
60 criteria organized in a hierarchical structure. The meaning of a button is explained as a hint (not shown
in the Figure) displayed when the mouse points to a selected button.
M. Makowski et al. - 24 - MCA-Needs: requirements&implementation

of the functionality of the MCA illustrated by tutorial examples is available in the user
guide [12].

Figure 7: The JIRA screen.

MCA provides the Contact utility, an easy to use and effective for users to make sug-
gestions, ask questions and report problems to developers. For organizing the process
of problem solving by the MCA developers the JIRA system is used. JIRA is a highly
popular and effective bug-tracking, issue tracking and agile project management software
application. Each submitted problem has an unique key, short summary, assignments to
the person responsible to solve the problem, reporter name, priority, status, and dates
of creation, update, due. All submitted problems can be classified in various way, e.g.,
according to the priorities, due date, types, affected software version, etc., which signifi-
cantly helps in management of software development.
JIRA also provides an API, therefore it was relatively easy to develop a simple form
through which the MCA users can submit a problem description directly to the JIRA. An
example of the developer’s view on selected problems is shown in Figure 7. Moreover, the
application handling the user input also stores all information (like the browser and OS
specifications, screen id, and the current status of the analysis) necessary for replicating
the situation that was at the time the problem was reported. The latter is of course very
helpful in solving the problem and providing the user with an appropriate feedback.

5.4 Summary of the MCA implementation

5.4.1 Versions of the software
There are three versions of the MCA software:
• The version dedicated to the multicriteria analysis of stakeholder preferences done
within the NEEDS project. This version had a limited functionality, tuned to the needs
of stakeholders who analyzed an assigned problem using a selected method. Therefore,
M. Makowski et al. - 25 - MCA-Needs: requirements&implementation

only one (out of four specified for the whole analysis) predefined problem was available
for each stakeholder, and one solver was available to all stakeholders. Stakeholders tak-
ing part in the analysis were asked to fill-in another survey to provide their profile for
the second stage analysis. This version is no longer available.
• The version used for the survey has been enhanced, in particular the user interface and
documentation were substantially improved; also access to all four problems is now
provided for all users. This enhanced version is now publicly available as the MCA-
Needs. It has a dedicated documentation which provides the background and various
details on the underlying problems, i.e., future energy technologies in the four European
countries: France, Germany, Italy, and Switzerland.
• A general purpose MCA Web-site, which supports analysis of problems defined by the
users, as well as several predefined problems provided for testing, and for the tutorial.
The user guide to the MCA is available as [12]. There is an automated self-registration
to the MCA and MCA-Needs, which eases their use by anybody interested either in mul-
tiple criteria analysis of discrete alternatives, or in such an analysis of future energy tech-
nologies. Since the features of the two versions currently available are practically the
same, henceforth we use only the MCA name.

5.4.2 Summary of the basic features of the MCA

MCA is Web-based thus providing users with anytime, anywhere access. The basic char-
acteristics of the MCA are as follows:
• It has a server-client architecture:
? The server side runs on the Solaris-based computers operated by the IME17 team. The
servers provide all resource-demanding functionality, including handling of data, and
running the solvers of the underlying optimization tasks used for selecting Pareto-
alternatives corresponding to the specified preferences.
? The thin-client side runs on a Web browser selected by the user. Most popular
browsers are supported, and no plug-ins are required. The only requirement for the
browser is to allow the pop-up windows generated by the IME server.18
• The user specifies his/her preferences in a very simple, yet effective way (in the sense
of easy analysis of all Pareto-efficient alternatives). After the preferences are specified
the solver is called and various characteristics of the solutions (current and previously
obtained) are available to the user.
• Each user has his/her own private data space, therefore he/she can specify own prob-
lems, their instances, and the corresponding analysis. She/he can interrupt the analysis
process, and continue it later.
• There is an on-line tutorial for the MCA, with several pre-loaded problems and their
analysis. In this way, the users can become familiar with the MCA without having to
start by preparing data for their own problems.
• MCA provides the Contact utility, an easy to use and effective tool for users to make
suggestions, ask questions, and report problems to the developers.
• There is automated self-registration to the MCA, which allows easy start to use it for
anybody interested in multiple criteria analysis of discrete alternatives.
17
Integrated Modeling Environment Project of IIASA.
18
Such windows are used only for providing the necessary functionality of the analysis. Of course, they
are not used for any type of advertisements.
M. Makowski et al. - 26 - MCA-Needs: requirements&implementation

5.4.3 Summary of the issues specific to the MCA-NEEDS

The research that provided the basis for the development of the MCA-NEEDS focused on
developing methods and a Web-based tool for multiple analysis of problems characterized
by:
• Two-stage analysis:
? large number of invited stakeholders (about 3000) having diversified backgrounds and
interests, as well as typically very limited analytical skills,
? experts analyzing the stakeholder preferences in order to prepare recommendations
for future energy technologies.
• Four similar problems (one for each of the four countries), each characterized by:
? a large set of discrete alternatives (over 20),
? a large number of criteria (over 60) organized in hierarchical structure represented by
an unbalanced tree,
? a number of criteria having multimodal value distribution and/or either large or very
small range of values.
• Easy and effective access through the Web by a large number of users. Most of the users
had limited computing and analytical skills; however, some of them had knowledge and
experience in multiple criteria analysis.
• An easy way for specification of preferences, suitable for users without analytical skills.
The set of criteria defined was developed by a dedicated research activity. The four
sets of alternatives described in Section 2.4 were developed by the NEEDS project partic-
ipants, then prepared in CSV format files, and uploaded to a data-server. The verified data
was prepared for the stakeholder’s analysis in the a CSV19 text format, which was up-
loaded to a data-server (see Section 4.2). This approach guaranteed that all stakeholders
were analyzing the problems defined by the same data set.
The analysis was done independently for four countries (see Section 2 for details), i.e.,
each of the stakeholders invited to the survey was assigned a specific country. The user
had only the choice of the language used for the interface, i.e., either English or German.
Moreover, the software version used for the survey had several additional features com-
pared with the two versions currently available, including: two languages for the interface
(English and German), and a dedicated introductory video explaining the whole analysis
process step-by-step.

6 Experience
6.1 Beyond the state-of-the-art
According to the author’s best knowledge, the MCA is the first application for interactive
multicriteria analysis that has introduced the following novelties to this type of analysis:
• It is Web-based, i.e., provides anytime anywhere access to the analysis. The client-
server architecture results in fast interaction; this is achieved by the design in which the
thin client handles only the data related to specification of preferences and displaying
19
The abbreviation comes from: Comma Separated Values. However, another character can be declared
as the separators.
M. Makowski et al. - 27 - MCA-Needs: requirements&implementation

the results; the resource-demanding computations and data management are done by
servers.
• It uses the database technology for all elements of the modeling process that consists of
several stages: the problem definition, the definition of the problem instances, and all
elements of analysis of each instance. Data for the whole process is managed by a data
server thus allowing subsequent analyses from different physical locations.
• It provides several functions that simplify definition of new problems, their instances,
and analyses. The users have a choice of defining new problems by either preparing
a csv-format file or by interactive specification of the problem. New problems can
also be specified as modifications of the previously defined problems. Several problem
instances of a problem (that differ by selection of alternatives and/or attributes used as
criteria) can be defined from a problem. Moreover, several analyses (each composed of
iterations associated with a specification of preferences) can be done for each instance.
Such a hierarchical structure of data and analysis greatly simplifies specification and
analysis of non-trivial problems.

6.2 Lessons learned

This multicriteria analysis of a large number of alternatives resulting from multidisci-
plinary research and done by a large number of stakeholders was the first activity of its
kind. Moreover, the MCA is the first Web-based client-server implementation of interac-
tive multicriteria analysis. Therefore it is worthwhile to summarize the lessons that are
likely to be of interest for researchers and practitioners.
Here we point out the selected issues we suggest considering when developing appli-
cations having requirements similar to those specified in this report:
• Requirement analysis is a critically important (but in practice often forgotten) element
of any non-trivial application. It should be done early enough to allow for develop-
ment of appropriate methods in situations when no appropriate methods and/or tools
are available.
• All persistent data of the whole analysis process should be managed by a DBMS.
• The server-client architecture is typical for Web-based applications. A thin client is also
typically used; however, one should consider efficient data caching to achieve reason-
able performance in situations when the client needs access to even a moderate amount
of data.
• Modular structure of the GUI and solvers eases not only software development and
maintenance but also experimentation with various ways of preference specification
and the corresponding solvers.
• The DBMS used for handling all persistent data (including the problems, preferences,
users, and results) should also handle data defining the configuration of interfaces be-
tween the application components.
• For two-stage applications (composed of both individual analyses and the analysis of
the resulting preferences) it is recommended that user profiles be handled on a different
server than the one used for individual analyses. In this way one can more easily assure
the privacy of individual preferences, and yet provide aggregated data for the second-
stage analysis.
• Contacts with the users can be maintained through an API to bug tracking software
M. Makowski et al. - 28 - MCA-Needs: requirements&implementation

(e.g., JIRA). This is effective in handling comments and problem reports, the latter can
be effectively used in software maintenance by linking the reported problems assigned
to a developer with a version control system (such as svn) used for software develop-
ment. A properly configured API should enable easy replication of the reported problem
(including information about the client computing environment).
• On-line documentation, including dedicated documentation for specific cases is very
useful. Although its development requires substantial resources, it is strongly recom-
mended.
• Automated self-registration and generation of access codes appears to be necessary for
efficiently handling a large number of application users.
Finally, we mention two issues that are rather commonly known but still remain typ-
ical troublemakers in the development of non-trivial applications. First, the amount of
needed resources (especially time) is underestimated. This problem is even more difficult
to handle in applications that require input from other activities as delays in providing
inputs cause delays in testing the application on actual data. In order to cope with this
problem it is recommended to plan in advance the development of realistic approxima-
tions of the missed input, and use it for initial testing. Second, the broad version range
of external components such as browsers, client operating systems, or DBMS used (or
affecting) the developed application result in many unexpected problems; some of them
require a substantial amount of resources to detect the problem and its source, and then to
fix it.
We close the summary of the experience by stressing the value of collaboration with
experts in domains pertinent to the problem, including the substantive area, stakeholder
involvement, methods of multicriteria analysis, development of the Web-based GUI’s,
and DBMS. Collaboration with experienced users willing to test the pre-release version
of the application is also essential. A successful development of a complex application
is practically impossible without such a wide and diversified network of experienced and
dedicated collaborators.

7 Conclusions
The report provides a comprehensive analysis of the requirements for the multicriteria
analysis of future energy technologies to be performed by a large number of stakeholders,
and summarizes the actual implementation of this analysis.
In the planning stage of the NEEDS project it was assumed that it would be possible
to select one of many existing methods and tools for multicriteria analysis of sets of dis-
crete alternatives, and to implement them for a survey of stakeholder preferences. The
comparison of the requirements documented in this report with the analysis of features of
the available methods documented in [6] clearly shows that new methods for multicriteria
analysis had to be developed. Moreover, the requirement analysis has also shown that de-
veloping a Web-based interactive application was the only rational way to support actual
multicriteria analysis by a large number of stakeholders.
The development of new methods and their pioneering implementation as a Web-site
supporting interactive multicriteria analysis required solutions of several methodological
and technological problems. These problems and their solutions are also relevant to other
applications of science-based support of policy-making. Advances in technology also
M. Makowski et al. - 29 - MCA-Needs: requirements&implementation

makes the internet widely available for public participation by citizens in policy-making.
Actual participation often involves analysis of conflicting objectives and attainable goals,
which is the essence of multicriteria analysis. However, true multicriteria analysis is
an interactive process, and its effective support (especially for participants with limited
analytical skills) still remains a challenge.
The implementation described in this report shows that meeting the requirements of
an effective multicriteria analysis of a complex problem by a large number of diversified
stakeholders is possible, but it requires multidisciplinary work throughout the entire pro-
cess, starting from the requirement analysis, through development of suitable methods
and modular tools, and their integration into an application that needs to be supported
by skilled staff during its use. Thus the authors believe that the approach and experi-
ence described in this report will be useful for researchers and practitioners involved in
science-based support of policy-making in various areas.
M. Makowski et al. - 30 - MCA-Needs: requirements&implementation

References
[1] B URGHERR , P., H IRSCHBERG , S., AND C AZZOLI , E. Quantification of risk in-
dicators. Deliverable report RS2b D-7.1, NEEDS project ”New Energy External-
ities Developments for Sustainability”, Brussels, Belgium, 2008. Available from
http://www.needs-project.org/2009/.

[2] B URGHERR , P., H IRSCHBERG , S., AND S CHENLER , W. Implementation, evalua-

tion and reporting on the survey on criteria and indicators for assessment of future
electricity supply options. Deliverable report RS2b D-12.3, NEEDS project ”New
Energy Externalities Developments for Sustainability”, Brussels, Belgium, 2008.
Available from http://www.needs-project.org/2009/.

[3] E LIASSON , B., AND LEE , Y. Y. Integrated Assessment of Sustainable Energy Sys-
tems. Kluwer Academic Publisher, 2003.

[4] G OLDSTEIN , G., H OBBS , B., AND LAITNER , J. MARKAL Goal Programming
Formulation. MARKAL-GP Version 5.1. International Resource Group, 2003.

[5] G RANAT, J., AND M AKOWSKI , M. Multicriteria methodology for the NEEDS
project. Technical report, International Institute for Applied Systems Analysis, Lax-
enburg, Austria, 2006. (report for the EU Project NEEDS; restricted distribution).

[6] G RANAT, J., AND M AKOWSKI , M. Multicriteria methodology for the NEEDS
project. Interim Report IR-09-10, International Institute for Applied Systems Anal-
ysis, Laxenburg, Austria, 2009.

[7] G RANAT, J., M AKOWSKI , M., AND O GRYCZAK , W. Multiple criteria anal-
ysis of discrete alternatives with a simple preference specification: Pairwise-
outperformance approaches. Interim Report IR-09-23, International Institute for
Applied Systems Analysis, Laxenburg, Austria, 2009.

[8] G REENING , L., AND B ERNOW, S. Design of coordinated energy and environmental
policies: use of multi-criteria decision-making. Energy Policy 32 (2004), 721–735.

[9] H IRSCHBERG , S., BAUER , C., B URGHERR , P., D ONES , R., S CHENLER , W.,
BACHMANN , T., AND C ARRERA , D. G. Sustainability criteria and indicators for
assessment of electricity supply options. Deliverable report RS2b D-3.2, NEEDS
project ”New Energy Externalities Developments for Sustainability”, Brussels, Bel-
gium, 2008. Available from http://www.needs-project.org/2009/.

[10] H OBBS , B., AND M EIER , P. Energy Decisions and the Environment. A Guide to
the Use of Multicriteria Methods. Kluwer Academic Publisher, 2000.

[11] M AKOWSKI , M., G RANAT, J., AND O GRYCZAK , W. Overview of methods im-
plemented in MCA: Multiple criteria analysis of discrete alternatives with a simple
preference specification. Interim Report IR-09-24, International Institute for Ap-
plied Systems Analysis, Laxenburg, Austria, 2009.
M. Makowski et al. - 31 - MCA-Needs: requirements&implementation

[12] M AKOWSKI , M., G RANAT, J., AND R EN , H. User guide to MCA: Multi-criteria
analysis of discrete alternatives with a simple preference specification. Interim
Report IR-09-22, International Institute for Applied Systems Analysis, Laxenburg,
Austria, 2009.

[13] M AKOWSKI , M., G RANAT, J., S CHENLER , W., AND H IRSCHBERG , S. Require-
ment analysis for WP9 of NEEDS RS2b. Technical report, International Institute
for Applied Systems Analysis, Laxenburg, Austria, 2006. (report for the EU Project
NEEDS; restricted distribution).

[14] M ESSNER , S., AND S TRUBBEGER , M. User’s Guide for MESSAGE III, WP-95-69.
IIASA, Laxenburg, Austria, 2001.

[15] M ESSNER , S., S TRUBEGGER , M., AND W IERZBICKI , A. Energy planning.

In Model-Based Decision Support Methodology with Environmental Applications,
J. Wessels, M. Makowski, and A. Wierzbicki, Eds., Mathematical Modeling and
Applications. Kluwer Academic Publishers, Dordrecht, 2000.

[16] M ESSNER , S., S TRUBEGGER , M., AND W IERZBICKI , A. Energy planning.

In Model-Based Decision Support Methodology with Environmental Applications,
A. Wierzbicki, M. Makowski, and J. Wessels, Eds., Series: Mathematical Model-
ing and Applications. Kluwer Academic Publishers, Dordrecht, 2000, pp. 399–414.
ISBN 0-7923-6327-2.

[17] N IGIM, K., M UNIER , N., AND G REEN , J. Pre-feasibility mcdm tools to aid com-
munities in prioritizing local viable renewable energy sources. Renewable Energy
29 (2004), 1775–1791.

[18] P OHEKAR , S., AND M.R AMACHANDRAN. Application of multi-criteria decision

making to sustainable energy planning, a review. Renewable and Sustainable Energy
8 (2004), 365–381.

[19] S CHENLER , W., AND BACHMANN , T. Quantification of economic indicators.

Deliverable report RS2b D-5.2, NEEDS project ”New Energy Externalities De-
velopments for Sustainability”, Brussels, Belgium, 2008. Available from http:
//www.needs-project.org/2009/.

[20] S CHENLER , W., BAUER , C., B URGHERR , P., AND H IRSCHBERG , S. Final report
on indicator database for sustainability assessment of advanced electricity supply
options. Deliverable report RS2b D-10.1, NEEDS project ”New Energy External-
ities Developments for Sustainability”, Brussels, Belgium, 2008. Available from
http://www.needs-project.org/2009/.

[21] S CHENLER , W., H IRSCHBERG , S., B URGHERR , P., M AKOWSKI , M., AND
G RANAT, J. Final report on sustainability assessment of advanced electricity supply
options. Deliverable report RS2b D-10.2, NEEDS project ”New Energy External-
ities Developments for Sustainability”, Brussels, Belgium, 2009. Available from
http://www.needs-project.org/2009/.
M. Makowski et al. - 32 - MCA-Needs: requirements&implementation

[22] S IMONS , A., BAUER , C., AND H ECK , T. Quantification of environmental indi-
cators. Deliverable report RS2b D-6.1, NEEDS project ”New Energy Externali-
ties Developments for Sustainability”, Brussels, Belgium, 2008. Available from
http://www.needs-project.org/2009/.

[23] U LUTAS , B. Determination of the appropriate energy policy for Turkey. Energy 30
(2005), 1146–1161.
M. Makowski et al. - 33 - MCA-Needs: requirements&implementation

A Selected applications of MCDA to energy planning

This appendix contains a review of the state-of-the-art of applying multicriteria analysis
to energy problems, as well as characteristics of three applications that exploit different
multicriteria analysis methods for energy problems considered relevant to the analysis
reported in this paper. Section A.1 provides here a brief overview of applications in the
energy planning area, including a brief classification, literature review and discussion
and Sections A.2 through A.4 present three selected case examples of prior multicriteria
analysis of energy problems.

A.1 Prior MCDA applications in the electric and energy sectors

MCDA has a broad history of use in both the electricity and energy sectors. Both fields
offer rich opportunities for a variety of modeling and optimization methodologies. The
problems in these fields are complex and multi-dimensional, meaning that modeling or
optimizing based on any one criterion (including least cost) is readily seen as overly sim-
plistic. Therefore MCDA has often been applied either as an endogenous optimization
function or as a post-analysis method of ranking results. MCDA has typically been used
on an ascending scale with three main levels, including the following.

Technology choice: This is typically in the area of generation technology choice for
either a generic or specific site. A broad range of criteria apply, but interaction with
the rest of the electric system is ignored.

Electricity sector models: Modeling the electricity system means an integrated analysis
of how different technologies are combined to meet shifting demand over time. The
classic use is system dispatch modeling, which optimizes plant operation based on
the variable dispatch cost using either a load duration curve or hourly operation ap-
proach. Dispatch modeling may be used tactically over the short term to model op-
eration of an existing system, or for strategic planning over the long term to model
system expansion. Some related, subsidiary electricity-related uses of MCDA em-
phasize the modeling of competitive markets and deregulation, emissions control
policies and costs and transmission and distribution. Many electricity models ig-
nore demand-side issues and price feedback on demand.

Energy sector models: Energy models include electricity as one sub-sector of the wider
energy sector. They are generally based on a broader, higher level, including
• Substitution between different primary energy resources.
• Specification of end-use technologies with competition/substitution.
• Incorporation of price feedback on demand.
• Aggregation on a national, regional or global level.

Such models may use simulation or optimization over successive time periods, and
again the MCDA may be either endogenous to the modeling or exogenously performed
upon results.
M. Makowski et al. - 34 - MCA-Needs: requirements&implementation

A.1.1 Literature review

A literature review of MCDA applications was conducted, focusing largely upon electric-
ity and energy related uses within 1) the Swiss research library search service NEBIS,
and 2) an online MCDA bibliography database maintained at the University of Auckland
in New Zealand.20 Other reviews of MCDA applications in energy planning were also
surveyed, see e.g., [3, 8, 10, 15, 17, 18, 23].
These various sources show that within three main levels of technology, electricity and
energy indicated above, MCDA methods have been applied to a wide range of specific
topics, including the following areas (the numbers in brackets indicate the number of
specific papers surveyed).
Energy sector topics:
• General Energy Planning & Policy (24)
• Project, location or technology specific planning (12)
• Expansion planning (6)
• Emissions related (3)
• Transmission & Distribution (2)
• Deregulation/competition (1)

Electric sector topics:

• Strategic expansion planning (10)
• Dispatch/scheduling (9)
• Transmission & Distribution (7)
• Competition & deregulation (2)
• Emissions reduction (2)
The NEEDS project falls within the area of expansion planning, specifically the choice
between many different generation technologies and several different system expansion
scenarios, based upon sustainability considerations. As can be seen, papers dealing with
expansion planning are a minority of the topics surveyed. The following discussion fo-
cuses on describing methods used in expansion planning, or used in related areas and
suitable for such use.
The literature review reveals that a primary dichotomy within the area of expansion
planning is between those methods that endogenously optimize the choice of technology
or scenario, and those methods that choose between discrete or predefined alternatives.
The optimizing approach appears more dominant in the literature, but this may be less
true in real-world applications as publications may tend to over-represent more theoretical
approaches.
A range of mathematical programming techniques has been used for endogenous op-
timization. Linear programming (LP) and mixed integer-linear programming (MILP) ap-
pear to dominate this area, but other methods include various goal-seeking approaches,
including distance minimization or aspiration/reservation techniques. It is important to
distinguish between the optimization technique involved and the way that the multicriteria
approach is formulated into the objective function of the technique. For example, both LP
and MILP techniques have a linear formulation of the objective function, which implies a
linear tradeoff between the different criteria (the integer restriction of some variables for
20
http://www.esc.auckland.ac.nz/people/staff/mehr002/References.
M. Makowski et al. - 35 - MCA-Needs: requirements&implementation

MILP is part of the constraints and not the objective function). This linear formulation
may imply21 a weighted sum approach to the MCDA; this is often cost minimization with
pricing of non-economic criteria, or can be formulated as the weighted sum of expected
values or expected utility functions.
The mixed-integer linear programming approach is often chosen because of the
integer-number nature of building a new generation facility (i.e., it is impossible, or at
least undesirable, to build only part of a new plant).
While it is not generally explicit in the literature, it does appear that for some optimiz-
ing techniques (e.g. LP and MILP) the application of the tool (or model) to the energy
or electricity sector came first, and the MCDA elements were added later as a way of
expanding the tool to incorporate other additional criteria. This is an evolutionary ap-
proach to using MCDA. The use of MCDA in the energy and electricity sectors also has
trends in development, which include combining different MCDA tools or methods (e.g.,
Promethee and AHP or the use of fuzzy logic in many different approaches).
Applying multicriteria analysis to a set of discrete options typically aims not only at
finding an optimum but also at providing various characteristics of pre-defined alterna-
tives, typically ranking or classification or clustering of alternatives. The literature shows
that the weighting approach is very popularly used in the electric sector for discrete rank-
ings, as well as for the optimizing approaches described above. Overall, the weighting
approach is fast, easy to understand, and flexible, allowing the incorporation of utility and
risk elements. It does have drawbacks however (e.g., eliminating one option may cause
the ranking of the remaining options to change). More detailed discussion of the features
of the weighting approach is provided in [5].
The other main school of ranking evident in energy applications is the French school,
including the family of Electre models (Electre I, II, III and IV, Electre IS and TRI) and
Promethee. These models use the twin elements of concordance and discordance (or
conjunction and disjunction). The concordance procedure allows a ranking of alternatives
based on their positive elements, and the discordance procedure down-rates alternatives
that are particularly bad on some (one or more) criteria. The 2D graph produced on the
concordance/discordance axes gives a visual representation of which alternatives do well
on many criteria and poorly on few, but a definite and unique ranking is not produced. The
literature indicates that these models are more frequently used for screening alternatives
as acceptable or unacceptable within a hierarchical framework of needs than for a cardinal
ranking.
In addition to the concordance/discordance method, several other MCDA screening
methods are available, including dominance comparisons, maximin/minimax comparison
(risk averse), maximax comparison (risk positive), and lexicographic elimination. These
methods are not used to produce cardinal rankings, so while their presence in the literature
is noted, they are not suitable to the present NEEDS needs in the electric sector.
21
This depends on the type of the scalarizing function. For example, reference point approaches use a
non-linear scalarizing function, which however can be converted to a linear problem. Therefore an LP/MIP
approach does not imply a weighted sum approach.
M. Makowski et al. - 36 - MCA-Needs: requirements&implementation

A.2 CETP (China Energy Technology Program)

Members of the NEEDS Task RS2b team have applied MCDA methods in a number of
past projects, but one in particular illustrates the way that several screening and rank-
ing approaches can be used together to provide a richer MCDA perspective. The China
Energy Technology Program was a large project that used state-of-the-art techniques to
apply sustainability criteria to electricity sector planning in Shandong province, China.
From the electric system modeling perspective, the CETP project consisted of au-
tomated, repetitive simulation of electric power system dispatch using a model called
EGEAS (Electric Generation Expansion Analysis System). A large number of scenarios
(18,144) were defined using stakeholder inputs. Specific alternatives were combined to
produce 1008 strategies that were modeled under a range of specific uncertainties produc-
ing 18 different futures. Strategies specifying a mix of different new generation technolo-
gies were implemented using a model called PSP (Pre-Specified Pathway) to schedule
the construction of individual plants. EGEAS was used to simulate least-cost system dis-
patch to produce plant generation, fuel consumption, plant and system costs and pollution
emissions. These results were combined with results from other tasks that modeled life
cycle burdens, emissions transport and damages, and risk and safety factors to produce a
wide range of sustainability indicators for each scenario. MCDA methods were applied
to these results in three main different ways.

A.2.1 Multi-scenario, multi-attribute tradeoff analysis

The repetitive simulation approach applied in CETP does not optimize for the ”best” so-
lution, but rather ”maps out” the option space available. The results are compared using
interactive computer graphics to datamine the mass of results produced. In particular,
tradeoffs between different pairs of criteria are examined to find patterns due to the ef-
fects of specific scenario options and uncertainties. These patterns showed the interactions
between old and new capacity, fuel choices and operation alternatives (like early retire-
ment or pollution control retrofits). The same software used for interactive statistics was
also used to present the results to stakeholders, emphasizing strategies that were robust
and flexible. This was a MCDA screening application that allowed stakeholders to reject
dominated strategies and focus upon the tradeoff set of Pareto optimal strategies for key
tradeoff pairs, in particular cost v. SO2 emissions.

A.2.2 Concordance/Discordance MCDA analysis using stakeholder preferences

To assist the stakeholder decision-makers participating in these projects, a second phase
of MCDA was used for deeper analysis of a subset of 12 selected scenarios. The MCDA
concordance/discordance ELECTRE III model was selected for this analysis, and the
stakeholder preferences were gathered using individual interviews. The concordance/dis-
cordance method did not produce a simple ranking, but rather a 2D mapping of which
strategies performed consistently well versus occasionally poorly.

A.2.3 Interactive weighted average MCDA for DVD presentation

The results of the CETP project were published in a book that included an accompanying
DVD presenting the project’s structure, assumptions and results. This DVD included
M. Makowski et al. - 37 - MCA-Needs: requirements&implementation

presentation of both the screening and Electre methods described above. However for this
purpose, a simple method of MCDA was also desired that could be interactively used to
elicit preferences and present the user with rankings of individual generation technologies
and simple combined strategies. The weighted sum approach was chosen for this purpose,
and programmed in Macromedia Flash. This approach required some simplifications of
system dispatch, emissions transport and other factors, but it also allowed individual users
to experiment and learn about the implications of their own choices.
The experience gained in CETP indicates the strong value of using and comparing
more than one method of MCDA analysis. This definitely requires a commitment of time
and effort, but it also prevents undue confidence in a single method requiring subjective
inputs (as all MCDA methods do).

A.3 MARKAL goal programming formulation

MARKAL is a large energy-economy model that covers the entire energy sector, includ-
ing the electricity sector. Many different versions of the model exist, with a large user
community that employs them for national, regional and global studies involving sce-
nario analysis of energy economics, the environment, etc. MARKAL itself is used to
formulate or structure energy questions, and its output passes to a solver engine that ac-
tually computes the solution to the stated problem. MARKAL models the full energy
sector, including competing primary fuels (coal, oil, gas, uranium, wind, etc.), energy
transport and conversion (i.e. primary energy to an energy carrier like synthetic fuels or
electricity), and competing end-use technologies that supply energy service demands (for
heat, light, transportation, etc.). Although MARKAL models represent the entire energy
system, some versions focus in more detail on the electricity sector or other areas like
synthetic fuels and transportation. While originally a linear programming model, some
formulations of the MARKAL family add non-linear elements and/or demand elasticity.
MARKAL’s objective function was originally simple cost minimization, and the cost
of emissions constraints were given by shadow prices. MARKAL has been expanded in
many cases to consider multiple criteria by adding them to the objective function, implic-
itly monetizing them (e.g., carbon taxes). However MARKAL can also use an objective
function formulation that is goal seeking. This means that rather than putting monetary
weights on non-economic factors, the model attempts to minimize the distance in vector
space from a given multi-criterion goal state.

A.3.1 The substantive model

The model specification of MARKAL [4] can be summarized by the following set of
equations:
X
q1 = ((1/cape,t ) ∗ escale ∗ (ewt− − + +
e d1e,t + ewte d1e,t ))
t≥gpstart,e∈GP EN V

X
q2 = ((1/least coste ) ∗ cscale ∗ (cwt− d− + +
2t + cwt d2t ))
t≥gpstart
X
T SC = ci,txi,t 
i,t
M. Makowski et al. - 38 - MCA-Needs: requirements&implementation

X
ai,txi,t ≤ bt ∀t
i
X
ee,f x − f, t + d−
1e,t − d1et = cape,t ∗ (1 − cappcte )
+

e∈GP EN V,j∈F
X
ci,t xi,t + d−
2t − d2t = least costt
+

xi , bi , d− + − +
1 , d1 , d2 , d2 ≥ 0

where:
• ci,t - cost associated with each component or technology i of the energy system for time
period t
• ai,t - matrix coefficient associated with each variable and row in LP representation of
the energy system
• bt - the right hand side of the equations of the LP for period t
• ee,f - emission coefficient associated with the technologies/fuel types in the energy sys-
tem
• xi,t - variables associated with each component of the energy system
• q1 - shows the total emissions over time,
• q2 - denotes the total discounted costs over time.
• T SC - defines that the total investment over time must sum to the total cost TSC,
and the remaining equations reflect system structure, emissions and costs.

A.3.2 The preference structure

The preference structure for a single criterion minimization is represented by the objective
function s defined as:

s = q1(cape,t , escale, ewt− + − +

e , ewte ) + q2 (least coste , escal, cwt , cwt )

with the following given parameters:

escale - scaling factor for each emission
cscal - scaling factor for total cost
ewte - weighting factor, above/below, for each emission
cwt- weighting factor, above/below, for total cost
cape,t - emission levels from the reference case
least costt - cost of least cost solution from the reference case
gpstart - year from which the GP emissions & costs limits are applied
Note that a scaling factor is used to bring the different variables (in different units) to
a common scale (0-1 or 0-100), and any variation from the target value for each criterion
is penalized by an individual weighting factor, which can be different above and below
the target value.
M. Makowski et al. - 39 - MCA-Needs: requirements&implementation

A.4 Reference point method - an energy planning model for Vienna

Energy planning is a general term that is applied to a variety of issues. It can address,
for example, the design of energy supply and utilization in new buildings; it can also
address municipal planning of district heat supply and the structure of heating systems.
In national energy planning, the focus is on political targets such as a diversification of
energy sources or environmental targets such as a reduction in acidification of soil and
lakes.
The public utility of Vienna used the dynamic linear programming MESSAGE [14]
model for its energy plan to evaluate the future development of the municipal energy
system, especially with respect to the coordinated expansion of the gas and district heat
grids. The organizational structure is such that electricity, district heat, and natural gas
are handled by three players in the energy market of Vienna.
A multi-objective approach was natural in the more long-term planning problems of
energy systems. The model for Vienna has been set up as a multi-objective optimization
model ensuring that three objectives are minimized: system costs; energy imports; and
pollutant emissions (using an aggregated index for various pollutants, which is generally
applied in Vienna). A simple aggregation of these objectives into one is not possible; for
example, although energy imports can be counted as costs, the dependence on imported
energy is a separate issue and it is difficult to convert this objective into monetary units;
the same concerns pollutant emissions.

A.4.1 The substantive model

The dynamic linear programming model MESSAGE [14]. The variables of MESSAGE
were grouped into three categories:

1. Energy flow variables representing an annual energy flow quantity. The unit is
usually MWyr for small regions and GWyr for bigger areas

2. Power variables representing the production capacity of a technology (usual unit:

MW or GW)

3. Stock-piles representing the quantity of a fuel being cumulated at a certain point in

time (usual unit: MWyr or GWyr).

The constraints generated by MESSAGE were grouped into the following categories:

1. Energy flow balances modeling the flow of energy in the energy chain from resource
extraction via conversion, transport, distribution up to final utilization

2. sum or relational constraints limiting aggregate activities on an annual or overall

basis, either absolute or in relation to other activities,

3. dynamic constraints setting a relation between the activities of two consecutive pe-
riods

4. counters that are only used for accounting purposes.

M. Makowski et al. - 40 - MCA-Needs: requirements&implementation

A.4.2 The preference specification

The targets of such investigations differ depending on the scope of the problem and the
decision makers involved. Industrial bodies and energy utilities strive for strategies with
minimal costs. In this case other objectives, such as environmental or social aspects,
are typically viewed as constraints. As an example, fuel use in a power plant can be
constrained due to site-specific regulatory emission limits. However, such constraints are
actually soft; and their classical representation as hard constraints often results in various
difficulties. This was the reason for applying the reference point approach that proved to
be a very suitable analysis method for the energy planning in Vienna.
The Reference Trajectory Optimization Method is an approach to optimize more than
one objective function for a problem in a way that circumvents the necessity of defin-
ing weights for converting a multicriteria problem into the corresponding single criterion
problem. The reference point approach allows the definition of reference trajectories for
all objectives; the solution of the auxiliary parametric optimization problem lies on the
Pareto-optimal border of the feasible region and is as close as possible to all reference
trajectories.
From a methodological point of view, the results of the study show the usefulness of
a multi-objective modeling approach. Especially, the applied reference point approach
supports multi-objective problem analysis which proved to be much better than the clas-
sical single-criterion optimization-based approaches, including the most popular method
of treating multicriteria problems by assigning weights to criteria. More details on this
study can be found in [16].