Systems Research and Behavioral Science

Syst. Res 35, 791–810 (2018)

Published online 17 July 2018 in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/sres.2519

■ Research Paper

Transparency and Reproducibility in

Participatory Systems Modelling: the Case
of Fuzzy Cognitive Mapping
Marta Olazabal1* , Marc B. Neumann1,2 , Sébastien Foudi1 and
Aline Chiabai1
1
Basque Centre for Climate Change (BC3), Leioa, Spain
2
IKERBASQUE, Basque Foundation for Science, Bilbao, Spain

By aggregating semi-quantitative mind maps from multiple agents, fuzzy cognitive map-
ping (FCM) allows developing an integrated, cross-sectoral understanding of complex
systems. However, and especially for FCM based on individual interviews, the map-
building process presents potential pitfalls. These are mainly related to the different
understandings of the interviewees about the FCM semantics as well as the biases of the
analyst during the elicitation and treatment of data. This paper introduces a set of good
practice measures to increase transparency and reproducibility of map-building processes
in order to improve credibility of results from FCM applications. The case study used to
illustrate the proposed good practices assesses heatwave impacts and adaptation options
in an urban environment. Agents from different urban sectors were interviewed to obtain
individual cognitive maps. Using this set of data, we suggest good practices to collect,
digitalize, interpret, pre-process and aggregate the individual maps in a traceable and co-
herent way. © 2018 The Authors Systems Research and Behavioral Science published by
International Federation for Systems Research and John Wiley & Sons Ltd
Keywords fuzzy cognitive mapping; reproducibility; transparency; open science; participatory
modelling

INTRODUCTION scientific readership is increasingly demanding

the full disclosure of original data and details
Transparency and reproducibility are two about the methods and the analysis itself. It is
essential characteristics of scientific studies. The not only the replicability of the results but also
the growing demand of evidence-based policy
* Correspondence to: Marta Olazabal, Basque Centre for Climate that reinforces the importance of transparency.
Change, BC3, Edificio sede, Planta 1, Parque Científico UPV/EHU,
Barrio Sarriena s/n, 48940 Leioa (Bizkaia), Spain. Disclosure, pre-analysis plans and open data are
E-mail: [email protected] three elements critical to strengthen open science

© 2018 The Authors Systems Research and Behavioral Science published by Received 11 May 2017
International Federation for Systems Research and John Wiley & Sons Ltd Accepted 6 May 2018
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits
use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial
purposes.
 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

for credible decision-making (Miguel et al., 2014). and Bousquet, 2010). Among the variety of
Disclosure consists of systematically reporting methods used in participatory systems modelling
key details about the data collection and the analy- to formalize knowledge, in this paper, we partic-
sis and allows other researchers to replicate results ularly focus on fuzzy cognitive mapping (FCM)
and build on current studies, therefore improving for its increasing application as a system map-
scientific progress (Dafoe, 2014). The pre-analysis ping method (Papageorgiou and Salmeron,
plan elaborates the standards for the study design 2013; Jetter and Kok, 2014; Olazabal and Pascual,
with regards to the sample size, data handling or 2016).
inclusion rules. Open data and materials provide FCM is a systems mapping method applied to
the means to test, reproduce or extend the a wealth of disciplines dealing with complex
research, so that results are refutable and credible decision environments from socio-ecological
(Krugman, 2013; Dafoe, 2014). (e.g. Gray et al., 2015) to financial (e.g. Mezei
Providing transparency and reproducibility and Sarlin, 2016). Likewise, FCM is increasingly
not only benefits the readers who want to used to deal with transdisciplinary problems
compare findings, learn and build on research such as, e.g. climate change (Reckien, 2014;
outcomes but also benefits the authors in Kok et al., 2015; Olazabal and Pascual, 2016;
reducing their research biases (MacCoun and Olazabal et al., 2018) as a means to integrate
Perlmutter, 2015). Aiming at transparency and different kinds of knowledge to get a better
reproducibility helps to avoid self-biases such understanding of phenomena and alternative
as pursuing mainly supporting evidences, action pathways. FCM captures expert
extracting signals from random patterns, taking knowledge developing a holistic view of the
for granted expected results or focusing on post- system that allows identifying interrelations
hoc ‘stories’ to justify non-significant results among elements which would have been
(Nuzzo, 2015). For this reason, transparency difficult to recognize otherwise (Mezei and
and reproducibility turn out not to be just the Sarlin, 2016). FCM has important contributions
responsibility of scientists but also of the to decision-making environments: It is a method
institutions and journals that publish their work that allows the integration of multiple expert
(Russell, 2013; McNutt, 2014). perspectives (Olazabal and Pascual, 2016), it
The issue of transparency and reproducibility is allows characterizing systems in data scarce
more problematic in social and integrated environments (Reckien, 2014), it helps to under-
sciences where the evaluation of the reliability of stand the complex structure of a real system
findings might be challenging and thus characterized by quantitative and qualitative
jeopardize the credibility of the studies for elements (Mehryar et al., 2017), and it allows
decision makers. However, many complex and scenario building (Kok, 2009).
dynamic real-life problems cannot be solved with The FCM community has touched upon issues
quantitative approaches, either because there is a related to traceability in FCM building (e.g.
lack of data or because the data is qualitative Vanwindekens et al., 2013) or started to collect
(Obiedat and Samarasinghe, 2016). This challenge common difficulties and pitfalls in FCM practices
often calls for the use of methods based on expert (Jetter and Kok, 2014), and there have been pro-
elicitation or participatory processes that help un- posals to solve technical difficulties for individ-
derstand systems that are characterized by data ual map aggregation and condensation
scarcity, scattered knowledge among multiple (Mourhir et al., 2016; Obiedat and Samarasinghe,
agents or high complexity (Olazabal and Reckien, 2016). However, these authors also detect space
2015; Obiedat and Samarasinghe, 2016). for improvement at least in the reporting of
In contrast to quantitative data-driven source data, collection and treatment methods.
methods (Hewitt and Escobar, 2011), a method More formal approaches are definitively required
based on a participatory process and expert to make these studies more reliable. This is par-
elicitation is more likely to be questioned for ticularly true in cases where the final FCM is ob-
transparency and reproducibility issues (Voinov tained from maps elicited in individual

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

792 Marta Olazabal et al.

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Syst. Res RESEARCH PAPER

interviews with experts and stakeholders (hereaf- directed, signed and weighted arrows
ter, agents) (Jetter and Kok, 2014). First, because representing causal relationships, thus forming
elicited data require an extensive treatment pro- a cause-and-effect diagram. Their graph structure
cess (maps need to be aggregated) and, second, allows systematic causal propagation and the
because the final output is used to simulate sce- addition of complementary knowledge by
narios to be used for decision-making (Penn merging different maps (Kosko, 1986). FCM can
et al., 2013). thus be used to provide an all-inclusive and
The aim of the paper is thus to perform a revi- integrated lens on the ‘perceived’ mechanisms
sion of the FCM map-building process based on of a system.
individual interviews so as to identify the poten- Aiming at improving decision-making pro-
tial pitfalls that may hinder transparency and re- cesses, FCM may be used to capture individual
producibility of FCM studies and suggest a set of or shared knowledge environments (Langfield-
good practices to overcome them. Smith, 1992). Common practice in knowledge
We start the paper by analysing caveats in cur- elicitation for FCM building expands on three
rent FCM building practice particularly focusing options (combined approaches are also possi-
on an example on socio-ecological applications ble, refer to Jetter and Kok, 2014): (i) The ana-
where the FCM community is exponentially lyst team (i.e. group of analysts responsible for
growing (Papageorgiou and Salmeron, 2013; the FCM study development) builds the map
Olazabal and Pascual, 2016). We then identify directly based upon their own technical or sci-
pitfalls and guide the reader through a set of entific knowledge, (ii) the analyst team builds
suggestions to improve the traceability and the map based on knowledge elicited from
coherency in map building as a means of agents (system experts or stakeholders that par-
guaranteeing transparency and reproducibility ticipate in the study providing their knowl-
(section on “Proposal to increase transparency edge) or (iii) the analyst team reviews existing
and reproducibility when combining fuzzy cog- documents or datasets to build the map (i.e.
nitive 3 maps”). data-driven FCM).
We discuss the proposed measures through a Following the second option (ii) where partici-
case study on the impacts of heatwaves in an ur- pation from agents is required, there are two
ban context (section on “Illustrative FCM study main modes in which an FCM can be built (refer
on urban climate change adaptation”) and pro- to discussion in Gray et al., 2014): either as a
pose a good practice approach on how to disclose group exercise (social or collective FCMs
all data and tools. The user of the final model is resulting from a group modelling exercise) or by
able to visualize the entire process that leads to combining information obtained from individual
the final model and is able to access all of the in- interviews.
termediary outputs using Supporting Informa- The first mode (group exercise) aims to obtain
tion (SM1–12). We discuss the findings and offer consensus on how the system under study func-
conclusions on the relevance of this contribution tions and to identify the most influential ele-
in the “Conclusions” section. ments of such system. Maps are jointly built
by a selected group of agents through a series
of workshops or focus groups (refer to, e.g.
FUZZY COGNITIVE MAPPING: COMMON Gray et al., 2015). This can lead to a reasonable
PRACTICE AND CAVEATS level of reproducibility and interpretability be-
ing based on intersubjective knowledge and
Previous scientific literature offers the interested consensus building. This type of group-based
reader detailed insights into the structure of building of FCM reduces misunderstanding, in-
FCM, its construction methods and performance creases coherency and facilitates knowledge ex-
indicators (e.g. Kosko, 1986; Özesmi and Özesmi, change (Hobbs et al., 2002; Jetter and Kok,
2004; Penn et al., 2013; Jetter and Kok, 2014). In 2014). However, it may reduce the potential
FCM, concepts relate to each other through richness, diversity and complexity that could

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

Transparency and reproducibility in Fuzzy Cognitive Mapping 793

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

be elicited, given that participants are focused In step 1, the problem is defined by the ana-
on reaching consensus instead of expressing lyst who formulates the question to be asked
the intricacies of their individual understand- to the participants according to the scope and
ings. The group-based mode also includes the objectives of the study. Step 2 entails the elicita-
risk of powerful agents dominating the process tion process. Participants are identified and in-
and therefore introducing bias. vited to take part in the study. Then, the
The second mode (individual interviews) is interview process is designed. Interviews are
used when the analyst is interested in acquiring conducted, either face to face or virtually, and
wider and deeper knowledge about how a spe- individual maps are drawn. Step 3 includes the
cific system works (refer to, e.g. Olazabal and digitalization, interpretation and pre-processing
Pascual, 2016) rather than on searching consen- of individual maps by the analyst. Step 4 deals
sus among stakeholders. However, this method with the homogenization across the set of indi-
requires the analyst to interpret, pre-process and vidual maps. Homogenization in this case refers
aggregate individual maps after these have been to the definition of a common terminology
collected. The high level of heterogeneity makes across maps and the choice of a common level
this option especially exposed to questions re- of detail to which the problem will be modelled.
lated to transparency and reproducibility. Two Later on, individual maps are combined to pro-
main issues may alter results or threaten coher- duce the final map (step 5).
ency in FCM that are based on individual inter- The final aggregated map is usually used in
views: (i) the different understandings of the two ways. It can be analysed as a complex net-
interviewees about the FCM methodology itself work (so-called static analysis), or it can be used
(mainly related to the semantics of ‘concepts’ to test different scenarios (so-called dynamic
and ‘relations’) and (ii) the biases of the analyst analysis) (for further reading, refer to Özesmi
when interpreting, analysing and aggregating and Özesmi, 2004; Jetter and Kok, 2014) which al-
maps that may involve (un)intended manipula- lows responding to what if questions (Carvalho,
tions of maps to support pre-conceived theories 2013; Mezei and Sarlin, 2016). In this paper, we
about the system under analysis. Because of the focus on the so-called static analysis, steps 1 to 5
intrinsic nature of participatory processes, nei- (map building process depicted in Figure 1),
ther of these two issues can be completely which is a preliminary step before running a sce-
avoided, but through adequate documentation, nario analysis (for further reading, refer to Kok,
they can be made as explicit as possible. We pro- 2009). Steps 1 to 5 are the steps most vulnerable
pose ways to do this in our paper. to self-biases and where risks for the credibility
In the next section, we extend the approach by of final results are most likely to occur. To fill this
Özesmi and Özesmi (2004) and suggest explicit gap, we list potential pitfalls and make sugges-
ways to improve transparency and reproducibil- tions to overcome them (Table 1). We highlight
ity for the most critical steps in FCM building the most critical methodological steps (column
processes for studies based on individual A) and sub-steps (column B) in relation to im-
interviews. proving transparency and reproducibility. We
identify potential pitfalls (column C) and sug-
gest ways to address them (column D). We also
suggest materials and documentation of
PROPOSAL TO INCREASE TRANSPARENCY methods to be provided (column E). To illus-
AND REPRODUCIBILITY WHEN COMBINING trate the potential of our proposals we build
FUZZY COGNITIVE MAPS and show these materials in form of Supporting
Information (SM1–SM12) for an illustrative case
As an extension of Özesmi and Özesmi (2004), study (column F) whose results and detailed
we illustrate the step-by-step FCM building pro- analyses are described in the section on
cess detailing intermediate processes and outputs “Illustrative FCM study on urban climate
(Figure 1). change adaptation”.

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

794 Marta Olazabal et al.

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Syst. Res RESEARCH PAPER

Figure 1 Detailed process and intermediate products of a fuzzy cognitive map (FCM) building exercise

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

Transparency and reproducibility in Fuzzy Cognitive Mapping 795

 Table 1 Strengthening reproducibility and transparency in FCM

796
A B C D E F

Approaches to
strengthen
Critical reproducibility and Material and methods
RESEARCH PAPER

Steps sub-steps Potential pitfalls transparency suggested Examples

Step 1: problem Definition of - Setting a too narrow - Define the boundaries - Workshop with Section entitled “Step 1:
definition research scope that restricts the of the system (temporal, stakeholders involved in Problem definition”
objective input from stakeholders spatial, sectoral) the management of the
(Jetter and Kok, 2014). according to the problem system
- Unstructured and identified.
incoherent definition of - ‘Model boundary chart’
the system under (Sterman, 2000 cited in
analysis leading to Jetter and Kok, 2014)
ambiguous final map.
- Description of the case
study context and the
system boundaries.
Step 2: elicitation Selection of - Ignoring important - Identify required - List with characteristics SM1: list of participating
relevant agents cause–effect relations, knowledge domains. of participating agents.
jeopardizing objectivity - Identify types of stakeholders.
and validity of the results stakeholders (civil
(Papageorgiou and society, decision-makers,
Salmeron, 2013). researchers and

International Federation for Systems Research and John Wiley & Sons Ltd
academia, private

© 2018 The Authors Systems Research and Behavioral Science published by

sector…).
- Ensure a minimum
number of participants
(Özesmi and Özesmi,
2004 suggest a minimum
of 20).
Interview - Maps do not represent - Define clear and - Interview guideline or SM2: interview
cause–effects relations of unambiguous questions. clear description on how guidelines.
the system under - Prepare an interview the interview was
analysis. guideline and follow it conducted.
- Participants use with each participant
different scales to without deviations.
evaluate problems - Help participants to - Original maps as SM3: original data
(Papageorgiou and clearly understand the drafted by the collected.
Salmeron, 2013). FCM methodology interviewees.
(Continues)

Marta Olazabal et al.

Syst. Res 35, 791–810 (2018)
Syst. Res

DOI: 10.1002/sres.2519
10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
 Table 1 (Continued)
A B C D E F
Syst. Res

Approaches to
strengthen
Critical reproducibility and Material and methods
Steps sub-steps Potential pitfalls transparency suggested Examples

- Incoherencies within (semantics of concepts,

individual maps. arrows and weights).
- Check and confirm the - Notes of the interviews. SM4: notes taken during
meaning of each new individual interviews.
addition to the map.
- Record the interview
(audio/video); take notes
during the interview.
Step 3: data Pre-processing - Distorting original - Maintain original - List of both original and SM7: FCM workbench
treatment knowledge, perceptions meaning and of translated conceptsa for manipulation of
or experiences translated connotations to original concepts
by participants, by guarantee a coherent (column C and D)
introducing bias by the interpretation.
analyst. - Track every change or
modification during the
pre-processing.
- Interpret maps using SM6: adjacency matrices
notes and recordings of - Original digitalized of individual maps

International Federation for Systems Research and John Wiley & Sons Ltd

Transparency and reproducibility in Fuzzy Cognitive Mapping

the interviews. matrices

© 2018 The Authors Systems Research and Behavioral Science published by

- Validate changes with
notes/recording and/or SM7: workbench for
with participants. manipulation of original
- Make sure that the concepts (columns C to H)
interviewer leads the - Evidence about any
data treatment. change or modification
affecting any of the
original matrices
Step 4: Selection of - Unstructured - Track every change - Material and/or SM7: workbench for
homogenization common homogenization can lead made in the process of methods used to define manipulation of original
terminology to double counting, homogenization. how concepts have been concepts (columns F
irrelevant or groundless - Select a consistent renamed and groups and G)
relations in the final map. terminology across maps. created (e.g. using
- Ambiguous or erroneous - Reverse signs where “Grounded Theory” as
interpretation of the final needed. proposed by Reckien,
aggregated map. 2014).
RESEARCH PAPER

Syst. Res 35, 791–810 (2018)

797
DOI: 10.1002/sres.2519
(Continues)

10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
 Table 1 (Continued)

798
A B C D E F

Approaches to
strengthen
Critical reproducibility and Material and methods
Steps sub-steps Potential pitfalls transparency suggested Examples
RESEARCH PAPER

- Interviewer(s)
participate(s) in the - Information about
homogenization process. which weight signs have
- Develop accumulation been reversed because of
curves of new concepts homogenization.
to validate the sample Refer to Figure 3
size.
- Accumulation curve
(Özesmi and Özesmi,
2004).
Choice of level - Track every change - Information about how SM7: workbench for
of detail: made in the process. concepts have been manipulation of original
grouping and - Specify the level of renamed and about the concepts (columns F and
ungrouping of model detail ungrouping groups created. G)
concepts or grouping of concepts
depending on the
objectives of the research. SM8: lists of concepts in
- Define the meaning and the aggregated map and

International Federation for Systems Research and John Wiley & Sons Ltd
context for each final - Interpretation/ their interpretation

© 2018 The Authors Systems Research and Behavioral Science published by

concept to facilitate definition of each of the
interpretation. concepts in the final map.
- Interviewer(s)
participate(s) in the
homogenization process.
Step 5: Collapsing of - Incoherent connections - Define how the weights - Information about the SM9: individual
aggregation matrices in the final aggregated are averaged when collapsing technique and adjacency matrices after
map leading to grouping concepts. This its program code. step 4
irrelevant, inaccurate or applies to individual
groundless results when matrices and to the And
using the final model for augmented matrix.
scenario analysis. - Identify potential SM10: R code for
- Incoherencies between incoherencies: two augmenting and
the information collected concepts connected with collapsing matrices and
and the final output. different directions or for quality and
(Continues)

Marta Olazabal et al.

Syst. Res 35, 791–810 (2018)
Syst. Res

DOI: 10.1002/sres.2519
10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
 Syst. Res

Table 1 (Continued)
A B C D E F

Approaches to
strengthen
Critical reproducibility and Material and methods
Steps sub-steps Potential pitfalls transparency suggested Examples

with different weight redundancy checks for

signing. final matrix/map
- Select an action to be
taken for each identified
incoherency. Document
the action.
- Interviewer(s)
participate(s) in the
aggregation process.

SM11: aggregated matrix;

- Final aggregated map in matrices for mean,
an easy-to-handle standard deviations and

International Federation for Systems Research and John Wiley & Sons Ltd

Transparency and reproducibility in Fuzzy Cognitive Mapping

format, e.g. in a sign changes of weights.
spreadsheet with graph

© 2018 The Authors Systems Research and Behavioral Science published by

visualizing capacities.
And

SM12: final aggregated

map in NodeXL format
(Social Media Research
Foundation, 2014)
a
Step required depending on the language in which results would be disseminated.
RESEARCH PAPER

Syst. Res 35, 791–810 (2018)

799
DOI: 10.1002/sres.2519
10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

ILLUSTRATIVE FCM STUDY ON URBAN (driving force, pressure, state, impact and re-
CLIMATE CHANGE ADAPTATION sponse), workshops and literature reviews to de-
fine the problem and research questions. The
The aim of this section is to illustrate Table 1 problem for our case study was defined through
through a complete case study process. The case a workshop in the framework of the European
study had the objective of exploring direct and project Bottom-up Climate Adaptation Strategies
indirect impacts of heatwaves and potential cli- for a Sustainable Europe (BASE). Eleven agents
mate change adaptation strategies in an urban from different sectors (water, energy and agricul-
context. The goal was to elicit transdisciplinary ture) participated including private and public
knowledge (different research disciplines and agents at different action levels (national, re-
different policy sectors) on impacts of heatwaves gional and local). The workshop aimed at identi-
and adaptation options in a specific urban envi- fying vulnerabilities and potential adaptation
ronment (city of Madrid, Spain). The output of pathways for heatwaves and droughts in the re-
this FCM building exercise is used in a second gional area of Madrid. Among other strategic ac-
stage to build scenarios of urban policy options tions, results pointed out the importance of the
(adaptation measures) considering cascading impact of heatwaves on health at city level and
cross-sectoral effects. Providing credibility to the the deployment of green infrastructures as a po-
full process of data elicitation and treatment is tential adaptation option. This workshop set the
therefore critical to rely on the results of the sce- scene to develop a specific case study in the city
narios. The experiment is reported following the of Madrid. In order to capture higher amounts
consolidated criteria for reporting qualitative re- and more diverse information, the approach se-
search (COREQ) (Tong et al., 2007). For the inter- lected was to elicit individual FCM maps from
pretation and discussion of the results in the participants with relevant knowledge and
specific context of climate change adaptation, experience on the impacts of heatwaves at urban
we refer the reader to Olazabal et al. (2018). level and specifically, in Madrid. The initial
workshop pointed out the importance of
considering not only urban agents but also
Step 1: Problem Definition agents at higher decision-making levels as their
policy options can impact the urban systems.
Defining the problem both from a scientific and The study therefore also contemplates multi-
stakeholder point of view is critical to under- level interactions elicited from stakeholders at
stand the boundaries and the scope of the exer- regional or national level.
cise and to understand the interests and
objectives of each one of the final users of the out-
puts (Lang et al., 2012). Failing to define the prob- Step 2: Elicitation
lem under study in an unambiguous way can
lead to “all-encompassing, overly complex Identification of Relevant Agents
models” (Jetter and Kok, 2014, p. 49) that are In FCM studies, participants are selected based
not useful to address stakeholder needs. Impor- on their expertise and knowledge about a specific
tantly, the definition of the problem will greatly issue. The objective here is not to obtain a repre-
influence stakeholders’ selection (section on sentative sample of a population but to represent
“Identification of relevant agents” below) and different knowledge areas. There is a variety of
the framing of the question which will guide the tools that can be used to identify relevant agents
FCM interviews with participants (refer to sec- such as snow ball sampling, stakeholder analysis
tion on “Interview design”). or organizational network analysis (Jetter, 2006;
Different approaches and methods can be used Reed, 2008).
for problem definition in FCM. For example, We identify two different but complementary
Mourhir et al. (2016) and Mehryar et al. (2017) strategies that could be used for this purpose:
use a mix of methods including a DPSIR model (1) selecting agents with similar expertise at the

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

800 Marta Olazabal et al.

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Syst. Res RESEARCH PAPER

same scale so as to obtain robust information on respond in two different ways: (i) adding new
the functioning of a sub-system and identify concepts to the map or (ii) identifying concepts
incoherencies and disagreements or (2) selecting already existing in their map as adaptation
agents with expertise in different areas of knowl- measures.
edge and/or at different scales so as to collect the
most diverse and complete information about
how a system functions.
Because, in our case study, the underlying ob- Conducting Individual Interviews
jective is to map the widest range of interdepen- Meaningful definitions of map concepts (Jetter
dencies between urban sectors in order to and Kok, 2014) as well as a full understanding
identify direct and indirect impacts of heatwaves, of FCM semantics (Carvalho, 2013) are required.
participants were identified according to the sec- We used a free-association technique rather than
ond strategy. Participating agents were selected pre-defined concepts (refer to Gray et al., 2014;
according to (i) multiple sectors (or knowledge Jetter and Kok, 2014). The latter mode may acti-
areas) that potentially exhibit interactions during vate memory (Jetter and Kok, 2014) and facilitate
heatwaves (health (9 agents), urban planning homogenization (refer to step 4); however, it may
and design (4 agents), green and blue infrastruc- also inhibit stakeholders’ reasoning and intro-
tures (4 agents) and climate change (5 agents); duce bias because of the pre-conceptions of the
(ii) professional perspectives (scientists, decision analyst. For this reason, we used the first ap-
makers and technical experts) and (iii) levels of proach of free association where participants are
decision-making (local, regional, national). In free to choose any concept and express it in their
some cases, there are limits to providing detailed own words, in contrast to a process where con-
information on the selected agents because of cepts are pre-established by the analyst (e.g. using
confidentiality (SM1). structured questions with pre-selected options).
We found that providing instructions that help
participants to understand FCM semantics
Interview Design through the available components (concepts,
The interview needs to be designed in alignment links and weights) along with an example is help-
with the research objective (refer to examples in ful (refer to guidelines in SM2 where we included
Isak et al., 2009; Reckien, 2014): (i) The objective an example on the impact of urban development
must be translated into understandable and un- on water reservoirs). Such instructions should in-
ambiguous interview questions and (ii) if multi- clude guidance to guarantee that both concepts
ple objectives are pursued in the same and links represent variables that can take on dif-
interview, this needs to be incorporated in an ex- ferent values along a gradient (Carvalho, 2013).
plicit way. The interview designed for our illus- In Table 2, we suggest some ways to improve
trative case study consisted of two stages (refer coherence within and between individual maps
to interview guidelines in SM2) (other examples which are specifically related to how concepts,
of guidelines in, e.g. Reckien, 2014). links and weights are explained and understood
First, all participants meet separately in person during interviews. When requested by the inter-
with the interviewee and are asked to develop a viewee, the interviewer may take the lead and
map, that according to their perception, experi- translate the discussion into the map (e.g.
ence and knowledge, responds to the question Reckien, 2014). This was often the case in our
“what are the impacts of heat waves in the city of Ma- study where participants generally showed little
drid?”. In a second round, they are asked to iden- confidence in their own initiative. To increase
tify the adaptation measures that would perform the traceability of this mapping process, the orig-
best to reduce the identified impacts: “Which mea- inal maps should be made available. SM3 con-
sures could help to compensate or mitigate (reduce) tains maps as drafted by participants together
the impacts of heatwaves at short, medium or long with the analyst’s notes taken during the inter-
term?” In this second round, stakeholders may views. SM4 includes any annotations that the

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

Transparency and reproducibility in Fuzzy Cognitive Mapping 801

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

Table 2 Suggestions to strengthen coherency in the elicitation phase

Definition Suggestions to strengthen coherency

Concepts Concepts (also known as nodes, factors or - Participants are asked to include only concepts
variables) are elements or entities of the that may reflect a gradient. This prevents from
system. having concepts in the map that cannot “integrate
the effects of the causal changes modelled by the
relations” (Carvalho, 2013, p. 2459).
Links Links (also known as arrows, edges, arcs or - Participants are asked to consider each link
connections) indicate the presence, sign and separately, i.e. effects of previously connected
direction of cause–effect relationships. concepts should not be taken into account.
- It is also advisable to offer positive or negative
linear causalities to avoid the cognitive strain on
participants (Jetter and Kok, 2014)
Weights A weight reflects the strength of the - Participants are reminded that the assigned
connection between two concepts, and it is weight does not refer to percentages, correlations
represented by a value on a linguistic or or probabilities.
numerical scale. - Participants need to assign weights based on
what they believe the strength of the cause–effect
relationship is. However, it is not always possible
to fully separate such a belief estimate from the
confidence about the estimate.
- Participants are reminded to contextualize
weights for the case study.

interviewer deemed useful for future interpreta- involve model adjustments to enable proper
tion and analysis. FCM computation and meaningful model inter-
pretations. They list the following cases for
model adjustment (ibid.):
Step 3: Data Treatment • Disregard for model boundaries as defined in
step 1 (problem definition)
Many of the non-transparent and non- • Overdetailed causal links, i.e. when a partici-
reproducible aspects of FCM are hidden in this pant describes a concept in too much detail.
step. Decisions taken here are often poorly docu- • Inclusion of ‘receiver’ concepts, i.e. when a
mented, and clear arguments are missing. The participant includes a concept that has no out-
data treatment process starts once all individual going arrows. According to Jetter and Kok
maps are collected. First, the analyst digitalizes (2014), this is a sign of incomplete knowledge
the individual original maps by converting them although these concepts can be used as diag-
into adjacency matrices (refer to didactic exam- nostic variables for the calibration of FCMs.
ples in Özesmi and Özesmi, 2004; Kok, 2009) (ex- • Inclusion of conditional causality, i.e. when a
amples from the case study in SM5). The analyst participant includes a relation that is
carries out a pre-processing of each of the indi- conditional.
vidual maps which can be later digitalized (SM6). • The presence of time lags, i.e. when a partici-
Pre-processing activities include all manipula- pant introduces relations that span different
tions on original individual maps as drafted by timeframes (Park and Kim, 1995).
the agents, as a result of a process of interpreta-
In our case study, pre-processing of individual
tion by the analyst and validation with the
maps have entailed the following cases:
agents. Jetter and Kok (2014) refer to these activ-
ities as post-processing activities (activities fol- • Redundant information: Concepts and rela-
lowing knowledge elicitation processes) that tions which are basically conveying the same

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

802 Marta Olazabal et al.

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Syst. Res RESEARCH PAPER

information should be removed. For example, • Minor adjustments such as typos can be vali-
in map #20 (refer to Figure 2a, SM3), the inter- dated through the notes (SM4) and recordings
viewee connects various concepts (‘data/fore- taken by the interviewer or with the participants.
casts’, ‘regional climate projections’ and • Consideration must be given to anonymity.
‘information/knowledge/tools’) and assigns Participants may prefer not to be recorded. In
a weight of 1 to the connections. This is a sign this regard, making information available and
of redundancy as one (information/knowl- building a transparent process are important
edge/tools) includes the other two. to create a trustful environment.
• Deviations from FCM method: e.g. when the
analyst suspects that concepts or relations can- As mentioned earlier, many of the caveats for
not be represented by variables of when rela- transparency and reproducibility of FCM
tions do not represent cause–effect studies emerge from not performing an explicit
relationships (as described in Table 2). For ex- and traceable data treatment process in this
ample, participant #15 links ‘residential water step. Not identifying adequate pre-processing
demand’ and ‘municipal services water de- needs in individual maps may lead to inaccu-
mand’ with ‘urban water demand’ (weight rate, uncoherent final outputs and increase the
0.6 and 0.4 respectively). This does not reflect potentiality of misleading decisions based on
a cause–effect relationship but two fractions those outputs. Some freeware tools such as
(refer to Figure 2b, SM3). FCMapper (Bachhofer and Wildenberg, 2009)
• If the information is collected in a language allow tracking part of the information, but
different from the language in which scientific not in a structured manner, and it is almost
results will be communicated, this requires never reported.
translation (Spanish to English in our case). Our suggestion here is to develop a unique
Here, the most critical aspect is not losing ‘workbench’ listing the entirety of concepts en-
any implicit meaning or cultural connotations countered in all the interviews and collecting
embedded in the terminology. the information on how these have been pre-

Figure 2 Examples of maps requiring pre-processing: redundant information (a, participant #20) and deviations from FCM
method (b, participant #15). Solid blue arrows denote positive connections; dashed red arrows denote negative connections;
the thickness of the arrows denotes the strength (weight) of the connection. Adaptation measures identified by the participants
are indicated with green nodes and text. Produced with NodeXL software

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

Transparency and reproducibility in Fuzzy Cognitive Mapping 803

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

processed. In this way, every decision taken by by the analyst during the interview and vali-
the analyst can be tracked back to the original dated with participant #6). Importantly, when
maps. This material can easily be shared and the analyst renames a concept using an antonym
compared. Our ‘workbench’ (refer to SM7 and (e.g. ‘health’ is renamed by ‘morbidity’), the signs
examples in Table 3) provides information on of connections need to be reversed. For this pro-
the 380 original concepts from the 22 original cess, using the notes taken during the interview
maps and allows to report on data treatment (refer to SM4), the analyst can assess how the ex-
(step 3, this step) and step 4 of homogenization pertise and background of the participants (refer
(developed in the section entitled “Step 4: Ho- to SM1) influences their use of terminology.
mogenisation”) (Table 3, columns F and G). The The choice of a consistent level of detail (stage
workbench, exemplified in SM7, stands as the 2) requires the analyst—in agreement with the fi-
central logbook for tracking the entire map build- nal user(s)—to decide the level of generalization
ing process which is further expanded in step 4. or specialization that the final map will convey.
Based on that, this will require to group or
ungroup concepts. In the case study, it was
Step 4: Homogenization mostly necessary to ‘zoom out’ (reduce the level
of detail) to solve unbalanced terminology (e.g.
Homogenization—also referred to as standardi- include ‘trees’ within ‘green infrastructures’). In
zation (Gray et al., 2014) or, partially, as conden- another example, morbidity was used as a term
sation or grouping (Özesmi and Özesmi, 2004) to group ‘allergies’, ‘waterborne outbreak’,
—relates to both the definition of a common ter- ‘foodborne outbreaks’, ‘legionellosis’, ‘cold’,
minology across maps (stage 1) and to the choice ‘thermal stress’, ‘dehydration’, ‘anuria’, ‘disorien-
of a common level of detail to which the problem tation’, ‘heatstrokes’, ‘burns’, ‘physical activity
will be modelled (stage 2). Both stages aim at in- health improvement’,1 ‘asthma/cold propensity’
creasing the consistency of the terminology and and ‘tuberculosis/legionella propagation’. One
the coherency of the structure in the final map. can envisage the case where the analyst wishes
Selecting a common terminology (stage 1) is re- to ‘zoom in’, i.e. to get a higher resolution of a
quired because participants can express the same specific phenomenon. For instance, the concept
idea in different ways or different ideas in the ‘morbi-mortality’ was mentioned by several par-
same way. It is important that a single analyst ticipants in our example (participants #2, #4 and
conducts all interviews and that the same analyst #21). If the analyst decides to analyse ‘morbidity’
leads the processes of data treatment, homogeni- and ‘mortality’ separately, as it is the case of our
zation, aggregation and analysis. If different ana- example, morbi-mortality should be ungrouped.
lysts conduct interviews but do not participate In this case, the incoming and outgoing connec-
across the full FCM building process, the risk of tions must be duplicated. Other examples can
losing important connotations or misinterpreting be found in the workbench SM7.
concepts and/or connections increases. In gen- The workbench SM7 is now expanded during
eral, the process of homogenization in stage 1 im- step 4 (treatment and homogenization) with
plies a grouping of concepts that have the same renaming and (un)grouping into the final 87 con-
meaning and selecting a common and consistent cepts. It also informs whether those changes im-
wording across maps (in the case study, e.g. ply a change of the sign of the incoming and
‘green areas’, ‘green spaces’, ‘urban green areas’, outgoing connections.
‘parks’… all renamed as ‘urban parks’) provided It should be noted that it is the analyst who de-
that is what they are referring to in all in each of cides which degree of complexity to reveal. Based
the maps. In other cases, it may also imply a re- on our experience, there exists a trade-off be-
wording of a concept to reflect the original idea tween the desire of adequately capturing com-
(e.g. participant #6 included ‘social services’ in plexity and the need of simplicity to facilitate
his/her map but he/she meant ‘social impact
prevention measures and policies’ as interpreted 1
Sign of connection is reversed. See expanded information in SM7.

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

804 Marta Olazabal et al.

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Syst. Res RESEARCH PAPER

green infrastructure and

duplications, change of
the interpretation of the final aggregated map. In

Comments (deletions,

urban parks and water

one blue infrastructure

bodies. Ungroup: one
that sense, the homogenization step is critical as

Participant refers to
some concepts and connections between concepts

Change of sign
sign)
might be removed or combined as a result of it.
H

Using a technique where the concepts are pre-

defined by the interviewer would avoid the need
of homogenization and thus reduce potential mis-
interpretations, but as already stated above, it
would also reduce diversity and broadness of col-

Green infrastructures
lected knowledge. Traceability in this step is re-
planning and design
Homogenization

quired in order to be able to replicate the study

stage 2 (scale)

Climate-sensitive

under different terminology and scaling choices.

Heat-warning

It is also helpful to provide a glossary for the con-

G

Morbidity

cepts to be included in the final aggregated map

systems

(step 5). This avoids ambiguities and facilitates an

adequate interpretation of the results by end-users
Table 3 Structure of the suggested workbench and examples

or by researchers aiming to replicate the study.

The use of the glossary is illustrated in SM8.
Homogenization

Effectiveness of
(terminology)

Eventually, once all individual maps have been

heat-warning

insulation of

Urban parks
and gardens
stage 1

interpreted and homogenized, the adequacy of

Morbidity

buildings
F

the sample size should be assessed. We suggest

Thermal

to follow the approach of Özesmi and Özesmi

plans

(2004) who analyse the incremental addition of

information as new maps are added. A stabiliza-
tion of the curve indicates that adding new par-
adaptation)
(impact or

ticipants to a sample would not add substantial

I or A
E

new knowledge to the problem representation.

For our example, we compare this so-called accu-
A

A
I

mulation curve (Figure 3, darker area) against the

total number of concepts (lighter area, Figure 3).
Heat-warning

insulation of
Translation

Formal approaches to homogenization are im-

Green/blue
dwellings

portant to increase transparency and reproduc-

Thermal
D

Health

ibility of FCM studies. They are also important

plans

areas

to strengthen internal coherency of the FCM

building process ensuring that only concepts that
reflect the same ideas are combined (Jetter and
Aislamiento viviendas

Kok, 2014). In an example from our study, ‘air

conditioning (AC)’ is positively linked to morbid-
sociales (planes
Original
concept

ity. However, some participants argued that AC

vegetal/agua
C

decreases indoor temperature and thus decreases

Superficie
de alerta)
Servicios

morbidity related to heat stress (i.e. negative rela-

Salud

tion). Others argued that AC increases the sus-

ceptibility towards infections (common cold),
implying a positive relationship with morbidity.
Both are initially valid and meaningful but be-
Map
ID

#16
B

#1
#2

#9

cause of the homogenization process (refer to ex-

ample of morbidity earlier where all potential
health problems are grouped), they might end
#155

#268
Var

#41
ID

#9

up being ignored (intentionally or not).

A

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

Transparency and reproducibility in Fuzzy Cognitive Mapping 805

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

Figure 3 Accumulation curve: number of maps (accumulated) vs. number of concepts mentioned

Step 5: Aggregation this process and adequately interpret the results,

we suggest to provide some additional metrics
The process of aggregating the individual maps for interpretation of the final map: ‘mean’ is an
into one large map involves three main processes m × m matrix (with m being the number of con-
(refer to step 5 in Figure 1): (i) in each individual cepts, m = 87 in our example) with the average
adjacency matrix, internal collapsing of concepts value of the weight for each of the existing con-
that belong to the same group, (ii) building of nections in the final aggregated map (here, we
the augmented matrix from individual matrices work with credibility factors of one, i.e. each
and (iii) collapsing of concepts belonging to the agent has the same weight). If a connection is ac-
same group in the final augmented matrix. tive in multiple maps (i.e. it has been mentioned
First, the process of collapsing needs to be by multiple agents), the standard deviation ‘sd’
done for each individual map (internal collaps- and the coefficient of variation ‘cv’ are computed
ing) as some connections may appear multiple to give an indication of spread between the
times after step 4 (section entitled “Step 4: Ho- weights given by different agents to that same
mogenisation”). If required, weights need to be connection. Values with high spreads should be
assigned to existing duplicated connections checked by the analyst as they may point to-
resulting from the process of ungrouping (refer wards disagreements between experts or inter-
to the previous morbi-mortality example). In pretation errors by the analyst. ‘Count’ provides
the collapsing process of individual and the number of agents mentioning the same
augmented matrices, weights are generally aver- connection. ‘Sign change’ checks if an active
aged. The analyst may also decide to weight connection has the same sign (positive or nega-
agents or groups of agents differently, to in- tive) in all individual maps. Sign changes are
clude, for example, a credibility factor flagged, indicating, again, a potential inconsis-
(Groumpos, 2010). tency that should be checked. SM11 contains
In the example, we reviewed and collapsed these m × m matrices for ‘mean’, ‘sd’, ‘cv’ and
each individual matrix using spreadsheets (refer ‘sign change’ values.
to result in SM9). Then, to build the augmented Generally, sign changes between concepts can
matrix and collapse it, we have developed a code occur after the process of homogenization when
in R (SM10). It uses as source data SM9 (individ- concepts are grouped. For example, when (i) con-
ual maps as separate worksheets) and computes cepts are renamed (e.g. ‘health’ is renamed to
the final map (refer to result in SM11). To track ‘morbidity’) or (ii) participants follow a different

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

806 Marta Olazabal et al.

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Syst. Res RESEARCH PAPER

Figure 4 Visualization of final output: aggregated map (SM12). Solid blue arrows denote positive connections; dashed red ar-
rows denote negative connections; the thickness of the arrows denotes the strength (weight) of the connection; the size of the
nodes denotes their centrality (importance) in the map. Adaptation measures identified by the participants are indicated with
green nodes and text. Produced with NodeXL software

reasoning when linking the same two concepts Figure 4 shows the final output. Depending on
(e.g. air conditioning (AC) linked to morbidity the level of homogenization, systems can achieve
described in the section entitled “Step 4: Homog- more simple representations. However, when de-
enisation”, in which case, the analyst may decide veloping FCMs, there must be a balance between
to keep both connections to stress these different the understandability of the system and the ro-
cause–effect relations or to average the weights bustness of the output for scenario building
and track the information). (Penn et al., 2013). If the objective is to capture
Positive or negative self-loops, that appear as the complexity and consider hidden cascading ef-
non-zero diagonal values in the adjacency ma- fects in decision-making processes, a certain level
trix, are of special interest. In the final aggregated of complexity is required. Typically, it is impossi-
map, 13 self-loops were found (refer to overall ble to capture this complexity through human
metrics in SM12). In our case study, we found, perception. We strongly encourage to provide
for example, that increasing green infrastructures the final map as both a matrix and in an accessi-
promotes further actions to create more green ble format (such as an open source viewer). The
infrastructures. Some other examples are matrix of connections and weights is shown in
‘autonomous adaptation by individuals’, SM11 and visualized in SM12 (in NodeXL for-
‘climate-sensitive planning and design’, and mat). Through this visualization, the output can
‘health services use’. In all these cases, the self- be more easily examined and discussed by re-
loop represents a positive feedback of the concept searchers, practitioners or stakeholders whose
on itself and is a result of the process of feedback may be solicited for revisions.
homogenization. Although this was not the case
for us, participants could be allowed to include
self-loops in their original maps. For scenario CONCLUSIONS
building, self-loops in the final aggregated map
can be kept (Buruzs et al., 2014) or removed Based on a step-by-step FCM building process,
(Kosko, 1986). we suggest good practice measures to improve

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

Transparency and reproducibility in Fuzzy Cognitive Mapping 807

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

transparency and reproducibility of FCM results. contribution is taken as a first step and opens
These suggestions ultimately intend to increase up a reflection to advance towards good practice
credibility of FCM results and comparability approaches for FCM building in particular and
among different studies. To illustrate the useful- for participatory system modelling in general.
ness of our proposals, we used a real case study
that analyses the impacts of heatwaves and po-
tential adaptation options in an urban setting.
We propose which documents to provide, and FUNDING
we suggest that a single lead analyst directs the
process and conducts all interviews to maintain This study is part of the project Bottom-up Cli-
coherency and reduce deviations from the origi- mate Adaptation Strategies for a Sustainable
nal data. We also highlight the process of homog- Europe (BASE) funded by the European Union’s
enization as a central and challenging step in Seventh Framework Programme for research,
FCM studies. technological development and demonstration
In general, disclosing more information in under Grant Agreement No. 308337. MO (FPDI-
FCM would allow to better understand the diver- 2013-16631 and IJCI-2016-28835) and MBN
sity and broadness of collected knowledge and to (RYC-2013-13628) acknowledge co-funding from
adequately interpret the final output, both critical the Spanish Ministry of Economy, Industry and
for reproducibility and effective applicability. For Competitiveness (MINECO).
traceability, it turns out imperative to track all
manipulations of the source data such as transla-
tion, renaming or adjustments in maps as the
central piece of documentation of the FCM model ACKNOWLEDGEMENTS
building. Because FCM aggregation methods are
not well agreed upon, it is also recommended to We would like to thank Ana Iglesias, Luis
provide a full disclosure of any techniques used Garrote and Pedro Iglesias from the Universidad
for this purpose. Politécnica de Madrid (UPM) for their support in
We conclude that good practices for FCM map establishing contacts to the participants of this
building should at least look at the following: study. We are grateful to the 24 participants that
devoted their time and shared their knowledge.
1 The provision of clear guidelines for building
an FCM from individual maps (guarantees
comparability among individual maps and re-
duces data treatment needs)
AUTHORS’ CONTRIBUTION
2 The use of a structured and understandable
workbench for data treatment (guarantees MO, MBN and AC conceptualized the study.
traceability of potential changes)
MO, MBN, SF and AC jointly designed the meth-
3 A homogenization method aligned to scientific odological approach. MO led the study, con-
requirements and stakeholder needs (guaran- ducted the interviews, led the data analysis,
tees coherency and usability of the final
prepared the supporting material (digitalized
output) maps, spreadsheets…) and developed the
4 A replicable aggregation method built on clear displayed items shown in the manuscript (figures
aggregation rules (guarantees reproducibility)
and tables). MBN developed the R code with the
5 Provision of metrics (e.g. mean, standard devi- support of MO. AC coordinated various experi-
ation) for concepts and connections (guaran- ments in the regional area of Madrid where this
tees interpretability of final map).
FCM exercise is framed. MO prepared the origi-
Given the increasing use of FCM across disci- nal draft of the manuscript. All authors reviewed
plines and the growing call for transparency and edited the manuscript and the supplemen-
and reproducibility in science, we hope that our tary material provided with it.

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

808 Marta Olazabal et al.

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Syst. Res RESEARCH PAPER

REFERENCES Wijnhoven PDF (eds.); 65–76 https://doi.org/

10.1007/3-7908-1681-7_5
Jetter AJ, Kok K. 2014. Fuzzy cognitive maps for fu-
Bachhofer M, Wildenberg M. 2009. FCmapper—Fuzzy
Cognitive Mapping Software Solution [WWW Doc- tures studies—a methodological assessment of con-
ument]. http://fcmappers.net (accessed 2.17.17). cepts and methods. Futures 61: 45–57. https://doi.
org/10.1016/j.futures.2014.05.002
Buruzs A, Kóczy LT, Hatwágner MF. 2014. Using
fuzzy cognitive maps approach to identify inte- Kok K. 2009. The potential of fuzzy cognitive maps for
grated waste management system characteristics, semi-quantitative scenario development, with an ex-
ample from Brazil. Global Environmental Change 19:
in: 2014 5th IEEE Conference on Cognitive
Infocommunications (CogInfoCom). Presented at 122–133.
the 2014 5th IEEE Conference on Cognitive Kok K, Bärlund I, Flörke M et al. 2015. European partic-
Infocommunications (CogInfoCom), pp. 141–147. ipatory scenario development: strengthening the
doi:https://doi.org/10.1109/ link between stories and models. Climatic Change
CogInfoCom.2014.7020435 128: 187–200. https://doi.org/10.1007/s10584-014-
Carvalho JP. 2013. On the semantics and the use of 1143-y
fuzzy cognitive maps and dynamic cognitive maps Kosko B. 1986. Fuzzy cognitive maps. International
Journal of Man-Machine Studies 24: 65–75. https://
in social sciences. Fuzzy Sets and Systems 214: 6–19.
https://doi.org/10.1016/j.fss.2011.12.009 doi.org/10.1016/S0020-7373(86)80040-2
Krugman P. 2013. The Excel depression. NewYork
Dafoe A. 2014. Science deserves better: the imperative
Times. http://www.nytimes.com/2013/04/19/
to share complete replication files. PS: Political Sci-
opinion/krugman-the-excel-depression.html
ence & Politics 47: 60–66. https://doi.org/10.1017/
Lang DJ, Wiek A, Bergmann M et al. 2012. Transdisci-
S104909651300173X
plinary research in sustainability science: practice,
Gray SA, Zanre E, Gray SRJ. 2014. Fuzzy cognitive
principles, and challenges. Sustainability Science 7:
maps as representations of mental models and
25–43. https://doi.org/10.1007/s11625-011-0149-x
group beliefs. In Fuzzy Cognitive Maps for Applied Sci-
ences and Engineering, Papageorgiou EI (ed.). Langfield-Smith K. 1992. Exploring the need for a
Springer: Berlin Heidelberg, Berlin, Heidelberg; shared cognitive map. Journal of Management Studies
29: 349–368. https://doi.org/10.1111/j.1467-6486.
29–48.
Gray SA, Gray S, De Kok JL et al. 2015. Using fuzzy 1992.tb00669.x
cognitive mapping as a participatory approach to MacCoun R, Perlmutter S. 2015. Blind analysis: hide re-
analyze change, preferred states, and perceived resil- sults to seek the truth. Nature 526: 187–189. https://
ience of social-ecological systems. Ecology and Society doi.org/10.1038/526187a
20. https://doi.org/10.5751/ES-07396-200211 McNutt M. 2014. Reproducibility. Science 343: 229–229.
Groumpos PP. 2010. Fuzzy cognitive maps: basic theo- https://doi.org/10.1126/science.1250475
ries and their application to complex systems. In Mehryar S, Sliuzas R, Sharifi A, Reckien D, van
Fuzzy Cognitive Maps: Advances in Theory, Methodolo- Maarseveen M. 2017. A structured participatory
gies, Tools and Applications, Studies in Fuzziness and method to support policy option analysis in a
Soft Computing, Glykas M (ed.). Springer: Berlin social-ecological system. Journal of Environmental
Heidelberg; 17–38. Management 197: 360–372. https://doi.org/
Hewitt R, Escobar F. 2011. The territorial dynamics of 10.1016/j.jenvman.2017.04.017
fast-growing regions: unsustainable land use change Mezei J, Sarlin P. 2016. Aggregating expert knowledge
and future policy challenges in Madrid, Spain. Ap- for the measurement of systemic risk. Decision Sup-
plied Geography 31: 650–667. https://doi.org/ port Systems 88: 38–50. https://doi.org/10.1016/j.
10.1016/j.apgeog.2010.11.002 dss.2016.05.007
Hobbs BF, Ludsin SA, Knight RL, Ryan PA, Biberhofer Miguel E, Camerer C, Casey K et al. 2014. Promoting
J, Ciborowski JJH. 2002. Fuzzy cognitive mapping as transparency in social science research. Science 343:
a tool to define management objectives for complex 30–31. https://doi.org/10.1126/science.1245317
ecosystems. Ecological Applications 12: 1548–1565. Mourhir A, Rachidi T, Papageorgiou EI, Karim M,
https://doi.org/10.1890/1051-0761(2002)012% Alaoui FS. 2016. A cognitive map framework to sup-
5B1548:FCMAAT%5D2.0.CO;2 port integrated environmental assessment. Environ-
Isak KGQ, Wildenberg M, Adamescu CM, Skov F, De mental Modelling & Software 77: 81–94. https://doi.
Blust G. 2009. Manual for applying fuzzy cognitive org/10.1016/j.envsoft.2015.11.018
mapping—experiences from ALTER-Net. ALTER- Nuzzo R. 2015. How scientists fool themselves—and
Net: A Long-Term Biodiversity, Ecosystem and how they can stop. Nature 526: 182–185. https://
Awareness Research Network. doi.org/10.1038/526182a
Jetter A. 2006. Elicitation—extracting knowledge from Obiedat M, Samarasinghe S. 2016. A novel semi-
experts. In Knowledge Integration. Physica-Verlag HD, quantitative fuzzy cognitive map model for
Jetter DA, Schröder PDH-H, Kraaijenbrink J, complexsystems for addressing challenging

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

Transparency and reproducibility in Fuzzy Cognitive Mapping 809

 10991743a, 2018, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/sres.2519 by Readcube (Labtiva Inc.), Wiley Online Library on [08/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
RESEARCH PAPER Syst. Res

participatory real life problems. Applied Soft Computing groups. International Journal for Quality in Health
48: 91–110. https://doi.org/10.1016/j.asoc.2016.06.001 Care 19: 349–357. https://doi.org/10.1093/intqhc/
Olazabal M, Pascual U. 2016. Use of fuzzy cognitive mzm042
maps to study urban resilience and transformation. Vanwindekens FM, Stilmant D, Baret PV. 2013. Devel-
Environmental Innovation and Societal Transitions 18: opment of a broadened cognitive mapping approach
18–40. https://doi.org/10.1016/j.eist.2015.06.006 for analysing systems of practices in social-
Olazabal M, Reckien D. 2015. Fuzzy cognitive map- ecological systems. Ecological Modelling 250:
ping: applications to urban environmental decision- 352–362. https://doi.org/10.1016/j.ecolmodel.2012.
making. In Handbook of Research Methods and 11.023
Applications in Environmental Studies, Ruth M (ed.). Voinov A, Bousquet F. 2010. Modelling with stake-
Edward Elgar Publishing; 576. holders. Environmental Modelling & Software 25:
Olazabal M, Chiabai A, Foudi S, Neumann MB. 2018. 1268–1281. https://doi.org/10.1016/j.envsoft.2010.
Emergence of new knowledge for climate change 03.007
adaptation. Environmental Science & Policy 83:
46–53. https://doi.org/10.1016/j.envsci.2018.01.017
Özesmi U, Özesmi SL. 2004. Ecological models based SUPPORTING INFORMATION
on people’s knowledge: a multi-step fuzzy cognitive
mapping approach. Ecological Modelling 176: 43–64. Additional supporting information may be found
https://doi.org/10.1016/j.ecolmodel.2003.10.027 online in the Supporting Information section at
Papageorgiou EI, Salmeron JL. 2013. A review of fuzzy
cognitive maps research during the last decade. IEEE
the end of the article.
Transactions on Fuzzy Systems. 21: 66–79. https://doi.
org/10.1109/TFUZZ.2012.2201727 DATA SM0: List of supplementary materials and
Park KS, Kim SH. 1995. Fuzzy cognitive maps consid- brief description
ering time relationships. International Journal of DATA SM1: List of participating agents
Human-Computer Studies 42: 157–168. https://doi.
org/10.1006/ijhc.1995.1007
DATA SM2: Interview guidelines
Penn AS, Knight CJK, Lloyd DJB et al. 2013. Participa- DATA SM3: Original data collected
tory development and analysis of a fuzzy cognitive DATA SM4: Notes taken during individual
map of the establishment of a bio-based economy interviews
in the Humber Region. PLoS One 8: e78319. DATA SM5: Individual digitalised maps
https://doi.org/10.1371/journal.pone.0078319
Reckien D. 2014. Weather extremes and street life in In-
DATA SM6: Adjacency matrices of individual
dia—implications of fuzzy cognitive mapping as a maps
new tool for semi-quantitative impact assessment DATA SM7: Workbench for manipulation of
and ranking of adaptation measures. Global Environ- original concepts
mental Change 26. DATA SM8: Lists of concepts in the aggregated
Reed MS. 2008. Stakeholder participation for environ-
map and their meaning
mental management: a literature review. Biological
Conservation 141: 2417–2431. https://doi.org/ DATA SM9: Individual adjacency matrices after
10.1016/j.biocon.2008.07.014 Step 4
Russell JF. 2013. If a job is worth doing, it is worth do- DATA SM10: R code for building the augmented
ing twice. Nature 496: 7–7. https://doi.org/10.1038/ matrix and collapsing
496007a
DATA SM11: Aggregated matrix
Tong A, Sainsbury P, Craig J. 2007. Consolidated
criteria for reporting qualitative research (COREQ): DATA SM12: Final aggregated map in NodeXL
a 32-item checklist for interviews and focus format

© 2018 The Authors Systems Research and Behavioral Science published by Syst. Res 35, 791–810 (2018)
International Federation for Systems Research and John Wiley & Sons Ltd DOI: 10.1002/sres.2519

810 Marta Olazabal et al.