Renewable and Sustainable Energy Reviews 186 (2023) 113632

Contents lists available at ScienceDirect

Renewable and Sustainable Energy Reviews

journal homepage: www.elsevier.com/locate/rser

Internal consistency and diversity scenario development: A comparative

framework to evaluate energy model scenarios
Noah Sandoval a, b, *, Janet L. Reyna a, b, Amy E. Landis b
a
Colorado School of Mines, 1500 Illinois St, Golden, CO, 80401, USA
b
National Renewable Energy Laboratory, 15013 Denver West Parkway, Golden, CO, 80401, USA

A R T I C L E I N F O A B S T R A C T

Keywords: Energy modeling frameworks and scenario analysis help us explore the potential impact of our actions and
Energy modeling uncertainties in future energy systems. Despite their importance, there is no systematic procedure for evaluating
Scenario development the scenario development process. In a literature review, we identify two core elements of the scenario devel­
Comparative framework
opment process: internal consistency and diversity which are oftentimes missing from scenarios. To address this
Annual energy outlook
World energy outlook
gap, we create the Internal consistency and Diversity Scenario Development (IDSD) comparative framework
Scout model which aims to assess the feasibility and diversity of scenarios for a given energy model. With this framework, we
LA100 study review commonly used energy models and demonstrate our framework on their scenarios.
Shared socioeconomic pathways The IDSD comparative framework can serve several purposes absent from previous scenario development
Asia-Pacific integrated model work by aiding energy modelers and report writers in crafting high-quality scenarios. First, the IDSD is a
reflective tool which can improve the quality of the scenario development process, enabling a comparative
assessment of energy models and scenarios. Second, the IDSD can provide guidance to modeling frameworks with
existing scenarios and those still in development; this feedback will enable modelers to improve the development
and the communication of the limitations of their scenarios. Third, this study has highlighted areas for
improvement in the scenario development of some commonly used energy model frameworks. Finally, there is a
complete lack of explanation regarding the stakeholder selection process. Addressing these identified items could
increase opportunities for advanced energy technology uptake and improve our options for achieving a more
resilient energy system.

results from these modeling frameworks can help key energy system
stakeholders make informed decisions. Energy modeling frameworks
1. Introduction
can offer valuable insight to future uncertainty and seek ways to build
towards a more resilient, sustainable, and just energy system.
The global energy system is currently situated along several inflec­
A fundamental component of energy modeling frameworks is the
tion points – environmentally, technologically, politically, and
implementation of scenario analysis. Scenario analysis allows us to
economically – where the decisions and actions taken now could dras­
compare many possible futures side-by-side. Along with this, we can
tically alter the trajectory of the future. Environmentally, anthropogenic
leverage scenario analysis to understand the implications of current and
climate change is beginning to significantly impact temperature and
future actions. It is important to note that given the high level of
weather patterns [1], and much of this is a direct result of energy sector
complexity and uncertainty found in the energy system, scenario anal­
emissions [2]. Technologically, the development of many advanced
ysis is not a tool to predict the future, but rather a method to find a range
energy technologies can both reduce the carbon intensity of energy
of possible futures [7]. Scenario analysis is a powerful tool for variable
production and increase efficiency while remaining cost effective [3]
exploration, but how can we assess quality and practicality of developed
and reliable [4]. However, the deployment of these technologies to
scenarios?
combat climate change faces significant political and economic barriers
The process of creating a set of scenarios is called scenario devel­
[5]. Energy modeling frameworks can explore the tradeoffs of different
opment. Based on the literature, the most fundamental aspect of sce­
technology deployment cases to provide an understanding of what
nario development is project scoping. The first part of project scoping is
future energy supply and demand could look like [6]. Furthermore,

* Corresponding author. Colorado School of Mines, 1500 Illinois St., Golden, CO, 80401, USA.
E-mail address: [email protected] (N. Sandoval).

https://doi.org/10.1016/j.rser.2023.113632
Received 17 October 2022; Received in revised form 2 August 2023; Accepted 6 August 2023
Available online 24 August 2023
1364-0321/© 2023 Elsevier Ltd. All rights reserved.
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

simply as an auxiliary guide rather than comprehensive comparative

List of abbreviations framework [25,26].
These previous studies evaluating scenarios from energy modeling
AEO Annual Energy Outlook frameworks lack several fundamental elements for comprehensively
AIM Asia-Pacific Integrated Model assessing scenario development. First, existing frameworks are unable to
BTO Buildings Technology Office ensure the internal consistency of each scenario. We define internally
DOE Department of Energy consistent scenarios as coherent [27], plausible [6], focused descriptions
IDSD Internal consistency and Diversity Scenario [7] of possible futures [28]. The internal consistency of a scenario is
Development closely connected with the scenario development process; only through
LADWP Los Angeles Department of Water and Power the clear articulation of the purpose, key questions, and characteristics
NEMS National Energy Modeling System can a scenario development team construct a scenario which has all the
NREL National Renewable Energy Laboratory attributes of internal consistency listed above. Ensuring internal con­
SSP Shared Socio-economic Pathways sistency is a key part of creating high-quality scenarios yet there exists
WEM World Energy Model no evaluation which focuses on this aspect of the scenario development
WEO World Energy Outlook process. While individual model reviews and model inventories could
hypothetically gauge the internal consistency of a scenario, none of the
studies we identified did so. At the same time, state-of-the-field reviews
have no ability to do this type of analysis, because they provide a
to articulate the purpose of the scenario and the key questions to be macro-level overview of the trends and gaps in the scenario analysis
answered. These questions can pertain to a wide variety of situations field rather than a detailed analysis of each scenario set (Table 1).
including: policy decisions [8], an organization’s response to new The second way current studies fall short is in how they define and
development [9], the achievement of specific objectives [10], the in­ evaluate diversity. For this work, we define diversity in two ways, as
crease in economic development of a country [11], or the changes in the both the complexity [29] and multi-dimensionality [30] of each sce­
climate processes [12]. Once the purpose and key questions have been nario, as well as the range of diversity found across a set of scenarios
solidified, the second part of project scoping is to outline the basic [30]. Both types of diversity, like internal consistency, are inextricably
characteristics of each scenario - What variables to use? Their range and tied to the scenario development process. It is critical when identifying
interdependence? What type of scenario outlook and progression to the purpose of the scenario and articulating the key questions to be
utilize? These two elements of project scoping are critical in ensuring answered that the entire spectrum of variable and scenario character­
high-quality scenarios. istics be considered. Given the complexity of the energy system, it is
In a review of the literature, we identified three types of studies important to consider the wide array of factors that impact any indi­
which examine scenario development – individual model reviews, state- vidual piece. It is crucial to explore all available variables to ensure that
of-the-field reviews, and model inventories (Table 1). The first type of the goals of the scenario analysis are met. While individual model re­
analyses are detailed reviews of individual energy modeling frameworks views and model inventories could analyze diversity, they typically do
and their respective reports which focus on the model methodology, not. Just as with internal consistency, state-of-the-field reviews do not
scenario contents, and an analysis of past projections [13–17]. The have the ability to report on the diversity of scenario sets due to their
state-of-the-field reviews survey large numbers of energy modeling macro-level analysis of the field (Table 1).
frameworks to identify specific shortcomings of the field as a whole The third gap in the previous work is the inability of existing ap­
[18–22]. Lastly, model inventories are reflective tools with the intention proaches to assess their wider impact in the scenario development field.
to be used by energy modelers to iterate and improve upon the devel­ Of the types of studies mentioned, none can: 1) compare scenario sets
opment of both their modeling frameworks and scenarios [23,24]. While side-by-side, 2) provide targeted feedback to each scenario set, and 3)
these model inventories have been adapted to other analyses, they act offer general guidance to the scenario development field through the
identification of trends and gaps (Table 1). Current studies are either too
focused and cannot generalize their results (e.g., individual model re­
Table 1 views), or are too general and do not offer specific feedback and guid­
Analytic capabilities of scenario development review types. ance to individual models (e.g., state-of-the-field reviews, model
Analytic Capabilities Individual State of the Model inventories). To ensure the development of high-quality scenarios, it is
Model Reviews Field Reviews Inventories critical that we have an ability to learn from the strengths of other
Analyze the internal scenario development processes, get targeted feedback on our own
consistency of scenarios processes, and learn from the greater energy modeling field.
In summary, there is currently no existing comprehensive compar­
Analyze the diversity of
scenarios ative framework that evaluates the internal consistency and diversity of
the scenarios used in energy models. Furthermore, none of the work
Provide targeted feedback for currently available can compare the scenarios of energy modeling
scenario of a single model
frameworks side-by-side, provide targeted recommendations for the
framework
Compare the scenarios of scenario development for individual models, and identify trends and
different models gaps across the field. Without a comparative framework to evaluate the
scenario development process, the projections of the future supply and
Offer general scenario
demand of energy from these scenarios could prove unreliable. These
development guidance
unrealistic views of the future could provide decision makers and
Key:
stakeholders with faulty information, spurring them to take inappro­
priate action. The consequences of this include missing opportunities in
Symbol
advanced energy technologies, a lack of understanding of the impacts of
future developments, and on the whole, a less resilient energy system.
Meaning Can analyze and Can analyze but typically Cannot do this Any of one of these could lead to significant social, economic, political,
typically does this does not do this analysis type of analysis and environmental ramifications.
analysis
To bridge these gaps, in this study we propose the Internal

2
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

consistency and Diversity Scenario Development (IDSD) comparative possible. Although scenarios, by their definition, are often exploring
framework which has all the analytical capabilities outlined in Table 1. unlikely or edge-case futures, any scenario should be one that under
The IDSD can critique the quality of the scenario development process some combination of events could exist. In practice evaluating internal
specifically in terms of the internal consistency and diversity of the sce­ consistency requires two steps. The first step to understand the process
narios. It is important to note that we are not critiquing the energy through which each scenario was developed. Therefore, the first Inter­
modeling frameworks themselves, but rather their scenario develop­ nal Consistency evaluation is the Development Evaluation. The second
ment and the reporting of those scenarios. In this way, we can compare step is to analyze the accessibility of the relevant documentation and
the scenarios of all different types of energy modeling frameworks side- data surrounding the development process of the scenario. The process
by-side. Beyond these comparisons, the IDSD can provide guidance to of collecting this information makes up our second evaluation, the Ac­
modeling teams working on both existing model frameworks and those cess Evaluation. Through the implementation of both evaluations, we can
still in development. The conclusions drawn from this evaluation will judge the Internal Consistency of any energy modeling framework
give energy modelers guidance to improve the development of their scenario. The process of crafting these two evaluations is outlined in the
scenarios along with the communication of the limitations of their sce­ next two sub-sections.
narios. At the same time, it will give insight to decision makers into the
types of scenarios which are possible and the limitations of certain 2.1.1. Development process evaluation
scenario types. This study includes an overview of the methods used to The driving question “Is there a rigorous development process?” sum­
create the IDSD, the results of the comparative framework on five case marizes the Development Process Evaluation. To answer this question, we
studies, a discussion of the results and recommendations for the energy define three metrics: stakeholder involvement, narrative complexity, and
modeling field, along with a conclusion which summarizes the IDSD data utilization. We define internally consistent scenarios as coherent
comparative framework and its implementation. [27], plausible [6], focused descriptions [7] of possible futures [28]. In
this definition, we use the words ‘plausible’ and ‘possible’, to
2. Methods acknowledge that some scenarios represent boundary conditions and are
highly unlikely but are viable to some degree. Next, we utilize the term
The following section outlines the methodology used in the IDSD ‘coherent’ to convey the idea that the variables which define any sce­
comparative framework. The IDSD is composed of two distinct sections– nario must be congruous with the scenario aim. Finally, we employ the
Internal Consistency and Diversity. Internal Consistency is assessed phrase ‘focused description’ to communicate that each scenario must be
through the Development Process and Access evaluations; Diversity is purposeful both as an individual scenario, but also within the scenario
assessed by evaluating Variable Diversity and Scenario Diversity. To orient set. Creating scenarios with these attributes requires a variety of per­
the reader of the purpose of each evaluation, we present a driving spectives, an understanding of the interactions and dynamics of the
question, along with a series of evaluation metrics which are then used system, and thoughtful discussion regarding the future progression of
to answer that driving question (Table 2). the system as a whole. Thus, the Internal Consistency of a scenario
Within each metric, there are several different classifications. For the cannot be confirmed unless an extensive development process is un­
Internal Consistency metrics, classifications were highlighted in green, dertaken. Table 3 outlines these three metrics along with their classifi­
yellow, and red to represent best, acceptable, and unacceptable prac­ cations, criteria, and reflection questions.
tices, respectively. For the Diversity metrics, there is not a single pref­
erable practice. Instead, Diversity is evaluated by whether the diversity 2.1.1.1. Stakeholder Involvement. Stakeholder involvement is key for
found in the variables and scenarios used are adequate to fulfill the the effective development of each of the other characteristics. The range
purpose of the scenario or the overall analysis. Each evaluation will of the stakeholder group could include any of the actors within the en­
include a set of reflection questions which can be used by scenario ergy system [7,9,28]. Those participating in scenario development can
development teams to reflect on the results of the evaluations and give be central to the modeling group, or they can be less involved and
them guidance on how best to improve their scenarios. participate via interviews, charrettes, consultations, or focus groups [6,
9,30]. Stakeholder involvement can come in a variety of different forms
2.1. Internal consistency and the reason(s) for this involvement is variable as well. It is crucial
that scenario development teams clearly communicate the purpose of
When we evaluate the Internal Consistency of scenarios from energy this involvement as well as the process for selecting and working with
modeling frameworks, we want to ensure that the futures posited are these stakeholders.

Table 2
Comparative framework outline.
Section Evaluation Driving Question Metric

Internal Consistency Development Process Is there a rigorous development process? Stakeholder

Involvement
Narrative
Complexity
Data
Utilization
Access Is the documentation accessible? Public
Availability
Ease of
Access
Diversity Variable Diversity Is the complexity of the scenario sufficient for its purpose? Variable
Type
Variable
Control
Scenario Diversity Is the range of possible futures found in the set of scenarios sufficient to meet the goal of the scenario analysis Scenario
Outlook
Scenario
Progression

3
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

Table 3
Scenario development evaluation metrics along with associated classifications, criteria, and reflection
questions.

2.1.1.2. Narrative Complexity. The narrative contextualizes the scenario speculations.

by providing the outlook or subject of the scenario [8,30–32], the trend
(s) which define the scenario [7], and/or the construction of the scenario 2.1.2. Access Evaluation
storyline [23]. For this study, we define variables as the specific pa­ The driving question “Is the documentation accessible?” summarizes
rameters which are used to differentiate the non-baseline scenarios from the Access Evaluation. To answer this question, we define two metrics:
the baseline. A detailed narrative is one which thoroughly explores the public availability and ease of access. (Table 3). If the applicable infor­
implications of specific future developments, considers all the variables mation outlining the scenario development process is not available, it is
which could impact this narrative, selects specific variables to define the impossible for a third party to judge the validity of the development
storyline of the scenario, and then provides justification for this process, and the Internal Consistency of the scenario cannot be
selection. confirmed. Table 4 outlines these two metrics along with their classifi­
cations, criteria, and reflection questions.
2.1.1.3. Data Utilization. Once a narrative has been developed, the next
step is to quantify the major trends. This is accomplished by identifying 2.1.2.1. Public availability. Availability of the main model report or
the major variables which will be used to define the scenario and to model documentation could be restricted by a subscription fee or
quantify uncertainty. This can be done through the statistical analysis of membership within a particular organization. Open access to all model
historical trends [8,23,31], data from the literature [9], or input from documentation, reports, and data without any type of restriction is ideal
subject matter experts [6]. Utilization of specific data sets along with a [18,33].
justification of why this specific data was used shows careful consider­
ation of the implications of the impact the data will have on the scenario 2.1.2.2. Ease of access. When seeking information about the scenario
outcomes. Projections not founded in specific data sets are perceived as development process, two factors lead to a high ease of access. First, the
simple conjectures without thought to the repercussions of these location of the information; it is easier to find information in the major

4
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

Table 4
Access evaluation metrics along with associated classifications, criteria, and reflection questions.

report or main documentation as compared to the appendices or sup­ 2.2.1. Variable Diversity evaluation
porting material. Second, the guidance to the information. Information The driving question “Is the complexity of the scenario sufficient for its
that can be found easily with precise guidance is preferrable to imprecise purpose?” summarizes the Variable Diversity Evaluation. To answer this
guidance where reading through many sections or documents to find the question, we define two metrics: variable type, and variable control. The
pertinent information is required [21,34]. energy system is highly complex, thus the number and combination of
variables we can perturb allow us to explore innumerable scenarios. In
2.2. Diversity this evaluation, we want to ensure that the variables used in each sce­
nario fulfill the purpose of the analysis. Table 5 outlines these metrics
Diversity, for the purpose of this paper, encapsulates two different along with their classifications, criteria, and reflection questions to be
ideas; the diversity found within each scenario and the diversity across used for all scenarios.
the set of scenarios. It is important to note that when quantifying both
types of diversity, we compare all the non-baseline scenarios to the 2.2.1.1. Variable Type. From an economic point of view, this metric
baseline scenario. The diversity found within a non-baseline scenario is evaluates whether the variable in question will alter the supply curve,
created through the different types of variables used to differentiate it the demand curve, or a combination of the two. For example, a decrease
from the baseline scenario. Therefore, the first diversity evaluation is the in the capital cost to build wind turbines will produce a horizontal shift
Variable Diversity Evaluation. Beyond this, Diversity can also be found in the supply curve, altering the price of electricity and thus the quantity
across a set of scenarios in terms of the types of scenarios and variables demanded [35].
used to differentiate the non-baseline scenarios from the baseline sce­
nario. Thus, the second diversity evaluation is the Scenario Diversity 2.2.1.2. Variable Control. This metric evaluates the level of control or
Evaluation. The process of crafting these evaluations is outlined in the influence that energy system stakeholders (e.g., policymakers, utilities,
next two sub-sections. energy consumers) have over the variable in question. By level of con­
trol, we mean the ability to dictate or influence a given variable, and

Table 5
Variable type and control evaluation metrics along with associated classifications, criteria, and reflection questions.
Metric Classification Criteria

Variable Type Supply Side A variable whose primary impact is on the supply curve of an energy commodity.
Mixed A variable whose impacts cannot be ascribed to either the supply or demand of an energy commodity.
Demand Side A variable whose primary impact is on the demand curve of an energy commodity.
Reflection • How many types of variables do you have in each scenario and across the scenario set?
Questions • Why did you choose these types of variables and not others for each scenario?
• Do the types of variables used for each scenario help in answering the key questions?
Variable Control External A variable where energy system stakeholders have no direct control or influence over due to the large number of dependencies, the
complexity of the phenomena, and/or the large number of parties involved.
Limited A variable where energy system stakeholders have limited but not direct control over due to the ability to somewhat influence the
dependencies and/or other parties involved.
Internal A variable where energy system stakeholders have no control or influence over.
Reflection • How many control levels do you have in each scenario and across the scenario set?
Questions • Why did you choose these control levels and not others for each scenario?
• Do the control levels of the variables used for each scenario help in answering the key questions?

5
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

Table 6
Scenario diversity evaluation metrics along with associated classifications, criteria, and reflection questions.
Metric Classification Criteria

Scenario Outlook Descriptive Scenarios with myopic or no foresight which do not resolve each time step to reach a predetermined final state.
Normative Scenarios which take specific variables and values into account to attempt to reach some final, predetermined state.
Reflection • Does the goal of the scenario analysis include both exploratory and goal-based objectives? Why or why not?
Questions • Is the purpose of the scenario to determine the impact of an action or trend or is it to reach a goal?
Scenario Progression Trend Scenarios whose projections are based on the extrapolation of historical data and which could represent any plausible or exploratory
trends.
Disruptive Scenarios whose projections have marked disruptions which cannot be attributed to or predicted by historical data.
Reflection • Does the goal of the scenario analysis only include the projection of historical trends? If so, why?
Questions • Is the purpose of the scenario to model a significant disruption to the status quo or an exploratory future? Does this match the
scenario progression?
• For scenarios which model a significant disruption, what is the purpose of this disruption? How was this disruption identified and
parameterized?
• For scenarios which model an exploratory future, what trend(s) is/are driving this scenario? Why is/are this/these trend(s) valuable
to monitor?

thus its impact on the energy system. A variable with internal control is Diversity (Table 8) results for the case studies. Next is an overview of
something that energy system stakeholders have complete control over, each case study which includes a summary of the report and the energy
such as the ability to create a carbon tax [11]. A variable with limited modeling framework(s) used, a recap of the IDSD Internal Consistency
control is a variable which can be indirectly influenced by energy system and Diversity evaluations, and a brief analysis of the results. We present
stakeholders. An example of limited control is the ability to use subsidies a comparative analysis of the trends across the case studies and major
or fuel reserves to influence fuel prices, while other market pressures findings in the Discussion section.
impact these prices as well. Whereas a variable where energy system
stakeholders have little to no control over, such as population growth, 3.1. IDSD internal consistency and diversity evaluation results
would be classified as external control [11]. We acknowledge that the
differentiation between limited and external control can be highly Table 7 and 8 compare side-by-side the IDSD Internal Consistency
subjective. For this analysis we took into consideration the scope of the
and Diversity results for each of the five case studies conducted in this
energy modeling framework being analyzed and the context of the study, respectively. For Internal Consistency (Table 7), classifications
variable in the scenario in question to make these distinctions.
which are highlighted in green, yellow, and red to represent best,
acceptable, and unacceptable practices, respectively. Along with this,
2.2.2. Scenario Diversity evaluation
baseline and non-baseline scenarios were evaluated separately because
The driving question “Is the range of possible future found in the set of of difference in the documentation level in some of the reports. For
scenarios sufficient to meeting the goal of the scenario analysis?” summa­ Diversity (Table 8), there is not a single, preferable practice. Instead,
rizes the Scenario Diversity Evaluation. To answer this question, we define Diversity is evaluated by the number of different variable or scenario
two metrics: scenario outlook and scenario progression. Within scenario classifications provide sufficient diversity to answer the key questions of
outlook, a scenario’s projection methodology is described as either the analysis. It is important to also note that we only analyze the non-
descriptive or normative [28]. Scenario progression classifies the evo­ baseline scenarios in how they compare to the baseline scenario. We
lution of the modeled future as either trend or disruptive [28]. The believe that the greatest insight can come from reviewing all metrics
purpose of a set of scenarios is to contrast the results of different scenario from each evaluation to understand how each scenario plays a part in
narratives [6,36]. The range created by these different scenarios should addressing the purpose of the analysis.
be large enough to include a wide variety of futures, while at the same
time being focused to effectively answer the key questions posed by the
modelers. Table 6 outlines these two metrics along with their classifi­ 3.2. Case study 1: World Energy Outlook
cations, criteria, and reflection questions.
The World Energy Outlook (WEO) is published in the fall of each year
2.2.2.1. Scenario Outlook. This metric evaluates whether a scenario is by the International Energy Agency with the purpose to provide “critical
descriptive, which simulates futures by resolving each time step irre­ analysis of and insights on trends in energy demand and supply, and
spective of the evolution of the projection pathway, or normative, which what they mean for energy security, environmental protection and
begins with a final, stated goal and resolves each time step in an attempt economic development” [37]. The modeling framework used to
to best meet that goal [28,29]. generate the WEO is the World Energy Model (WEM) which is a
large-scale, general equilibrium, simulation-based model which uses
yearly time-steps to the year 2050 across the entire globe which is
2.2.2.2. Scenario Progression. This metric evaluates the evolution of the
divided into 26 different regions/nations [38]. The baseline scenario is
scenario. Trend scenarios use projections extrapolated from historical
called the stated policies scenario, and it includes three other scenarios –
data, which represent plausible or even exploratory development.
announced pledges, sustainable development, and net zero emissions by
Disruptive scenarios include events that shock the energy system, which
2050 scenarios [39].
serve as boundary conditions for future energy trends [29].
What follows is a summary of IDSD evaluation of the 2021 WEO. For
Internal Consistency, the report itself, all model framework documen­
3. Case studies
tation, and support materials were completely available online [38,39].
All the scenarios were outlined with a basic narrative in the model
In this section we present the results of the IDSD comparative
documentation and main report [38,39]. The data used to define both
framework for five case studies. Each case study is comprised of a
the baseline and non-baseline scenarios was found in the model docu­
commonly used energy modeling framework with an associated report.
mentation and main report [38,39]. The WEM documentation refers to
These are used together to understand the scenario development pro­
the collaboration with various other modeling groups, academic in­
cess. We begin with the IDSD Internal Consistency (Table 7) and
stitutions, and expert reviewers, but does not go into the detail about the

6
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

Table 7
The case study results for the IDSD internal consistency evaluations.

Table 8
The case study results for the IDSD diversity evaluations.

selection process of these stakeholders, nor their purpose. In terms of being said, it is clear that many different expert stakeholders are
Diversity, the variables used to differentiate the non-baseline scenarios involved, representing a variety of institutions. However, the purpose of
include the fuel cost and carbon tax which are both supply side vari­ their involvement and their selection process remains unclear. Second,
ables. The carbon tax is classified as internal control, whereas fuel cost is while the 2021 WEO does feature both normative and descriptive sce­
classified as limited control [38,39]. narios, there are only trend-based scenarios. Finally, the 2021 WEO only
Based on the analysis of the 2021 WEO using the IDSD comparative considers supply-side variables. We therefore recommend that the WEO
framework (Tables 7 and 8), we offer the following recommendations to development team consider the reflection questions for the following
the WEO development team. First, while the 2021 WEO has low access metrics: stakeholder involvement, narrative complexity, data utilization,
barriers, this report lacks some elements of a robust development pro­ variable type, and scenario progression to ensure the Internal Consistency
cess with basic narrative complexity and data utilization ratings. That and Diversity of their scenarios and scenario set.

7
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

3.3. Case study 2: Annual Energy Outlook use and fuel type outputs from the 2020 AEO and then applies energy
conservation measures to these technologies resulting in energy savings
The Annual Energy Outlook (AEO) is published in the spring of each [44]. Therefore, this study has a similar temporal and spatial resolution
year by the US Energy Information Administration [40]. While an and scope as the 2020 AEO. This study uses scenarios built upon com­
explicit purpose for AEO was not stated, the documentation does state binations of three variables: increasing building technology efficiency
that the 2022 AEO “explores long-term energy trends in the United (1T through 1T-2T–3T), carbon intensity of the electric power supply
States” [40]. The modeling framework used to generate the results for (RB – reference, HR – high renewable), and an incentive for fuel
the AEO is the National Energy Modeling System (NEMS) which is a switching (FS0 – no incentive, FS20 – 20% incentive) [46]. See Table 8
general equilibrium, simulation-based model which uses yearly for a full list of scenarios.
time-steps to the year 2050 across the entire contiguous United States What follows is a summary of IDSD evaluation of the Scout Core
with a resolution of the nine US census divisions [41]. The purpose of Measure Scenario Analysis. For Internal Consistency, the analysis itself,
NEMS is to “project the impact that energy, economic, environmental, along with all Scout model documentation and datasets used in the
and security factors can have on the U.S. energy system” [41]. The scenarios were fully available online [45]. Information about the model
baseline scenario is called the reference scenario, with eight other sce­ and its scenarios was prominent and easy to find in all the documenta­
narios – high oil price, low oil price, high oil and gas supply, low oil and tion [44–46]. While the number of scenarios and variable combinations
gas supply, high economic growth, low economic growth, high renew­ make for a wide range of possible futures, the model documentation
able energy cost, and low renewable energy cost [40]. simply explained the difference between the different variable levels and
What follows is a summary of IDSD evaluation of the 2022 AEO. For did not provide a narrative for any of its scenarios. Rather these sce­
Internal Consistency, the report itself, all model framework documen­ narios are permutations of the three variables outlined previously
tation, and supporting materials were completely available online [44–46]. Conversely, the model documentation went into great detail
[40–43]. For all scenarios, the AEO team conducted multiple working about the data used by the different variables with all the datasets
group meetings in the year leading up to the report for each of the major available online [45]. Lastly, while the Scout model was developed by a
subsectors in their model (Industrial, Macroeconomic, Residential & group of researchers from the Department of Energy’s Energy Efficiency
Commercial Buildings, Petroleum & Natural Gas, Electricity, Coal, Re­ & Renewable Energy Office, Lawrence Berkeley National Laboratory,
newables, and Nuclear). The purpose of these working groups is clearly the National Renewable Energy Laboratory, and several energy-related
articulated and there is ample documentation of the work done by each research consultants. However, there was no available documentation
group [42]. That being said, the stakeholder selection process is unclear. of the selection process nor the reason for their inclusion in this
A basic description of the development of the baseline scenario was modeling effort [44,47]. In terms of Diversity, the variables used to
available in the report narrative along with the data used to generate it differentiate the non-baseline scenarios include fuel supply as a supply
[40]. Unfortunately, only a small amount of information was found side variable, along with fuel switching and energy efficiency im­
about the development of the non-baseline scenarios in the report provements for demand side variables [44–46]. The incentivized fuel
narrative. The creation of these scenarios appears to be a simple percent switching variable would be classified as internal control level, carbon
change in certain variables from the baseline scenario [43]. No infor­ intensity of the grid would be classified as limited control, and building
mation pertaining to the specific narratives nor the reasoning for the technologies would fall under the external control classification
changes in variables was available for these scenarios. In terms of Di­ [44–46].
versity, the variables used to differentiate the non-baseline scenarios Based on the analysis of the Scout Core Measure Scenario Analysis
include fuel cost, fuel supply, and asset capital cost which are classified using the IDSD comparative framework (Tables 7 and 8), we offer the
as supply side and limited control variables, along with population following recommendations to the Scout team. First, while this study
growth and gross domestic product growth which are classified as mixed involves researchers from a variety of different research institutions, the
type and external control variables [40]. Scout team could reflect more on their stakeholder involvement. Sec­
Based on the analysis of the 2022 AEO using the IDSD comparative ond, the narrative complexity of this study is limited and it only con­
framework (Table 7 and 8), we offer the following recommendations to siders descriptive, trend-based scenarios. It is important to note that this
the AEO development team. First, while the 2022 AEO does have low is the only scenario analysis to include disruptive scenarios. These sce­
access requirements and pursues extensive stakeholder involvement, the narios (HR 3T FS0 and HR 3T FS20) feature high renewable electricity
narrative complexity and data utilization of this report is lacking. Sec­ generation, upgrading the existing building stock to the highest per­
ond, the 2022 AEO only considers descriptive and trend-based scenarios. forming building technologies, and fuel switching.
Third, while the 2022 AEO does consider supply and mixed-type vari­ Lastly, this report does not consider mixed-type variables. We
ables, no scenario features a scenario with multiple Variable Types. therefore recommend that the Scout development team consider the
Lastly, the 2022 AEO does not consider internal control variables. We reflection questions for the following metrics: stakeholder involvement,
therefore recommend that the AEO development team consider the narrative complexity, scenario progression, and variable type to ensure the
reflection questions for the following metrics: narrative complexity, data Internal Consistency and Diversity of their scenarios and scenario set.
utilization, scenario outlook, scenario progression, variable type, and vari­
able control to ensure the Internal Consistency and Diversity of their 3.5. Case study 4: LA100 Study
scenarios and scenario set.
The LA100 Study was conducted by the National Renewable Energy
3.4. Case study 3: Scout Core Measure Scenario Analysis Laboratory (NREL) for the Los Angeles Department of Water and Power
(LADWP) with the purpose of informing “the city of Los Angeles,
The Scout model is a building energy model developed primarily by LADWP, and other stakeholders of the possible pathways to 100%
the U.S. Department of Energy’s Building Technologies Office (BTO) renewable energy, and the implications of these pathways for the people
[44]. While the Scout model is used for a variety of applications by BTO, who live and work in Los Angeles” [48]. The LA100 Study was created
for this study we are evaluating the Scout Core Measure Scenario Analysis using a combination of demand models and historical demand data. The
[45]. The purpose of the Scout model is to “project the reductions in residential, commercial, and transportation demand sectors were
building operation-phase CO2 emissions and primary energy use modeled through a suite of tools developed primarily by the NREL:
through 2050 and compare these reductions against targets in the [U.S. ResStock [49], ComStock [50], and EVI-Pro [51], respectively. The in­
Mid-Century Strategy]” [46]. Scout is a bottom-up, stock inflow model dustrial demand sector along with the water system and outdoor lighting
which uses the residential and commercial building sector energy end sectors were modeled using historical demand data from LADWP [52,

8
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

53]. The city is divided into receiving stations, census tracts, and indi­ modeling methodology was described in the AIM implementation of
vidual properties [48]. The baseline scenario is called the moderate Shared Socioeconomic Pathways article [58]. This same journal article
scenario, with two other scenarios, high and stress, which each include that introduced the SSP Framework also directed readers to where they
different technology electrification levels (e.g., electric heating systems, could find the data used to project the variables in their scenarios
electric vehicles, etc.) and energy efficiency improvements [48]. [59–66]. Lastly, another journal article for the SSP Framework described
What follows is a summary of IDSD evaluation of the LA100 Study. the origins of the framework along with the expert teams, interdisci­
For Internal Consistency, the study itself was fully available online [48] plinary teams, and modeling teams involved in its development [31].
and had precise guidance to the scenario development process in Unfortunately, we found no information regarding the selection process
Chapter 3 of the study [54]. Chapter 3 of this study began with a general of these different stakeholder groups. In terms of Diversity, the variables
outline of each scenario and then it went into great detail as to how the used to differentiate the non-baseline scenarios include energy intensity
different scenarios were modeled along with the data used delineate changes and technology adoption for demand-side variables, population
these scenarios [54]. Chapter 2 of the report outlined the study approach growth, gross domestic product growth, land use change, and climate for
and gave an overview of the proceedings of the LA100 Advisory Group. mixed type variables, with fuel cost and asset capital cost for supply-side
The purpose of the Advisory Group was to provide feedback and variables [59–66]. Of these variables, technology adoption, fuel cost,
comment on the work done by the project groups. Along with this, the and asset capital cost would be classified as mixed control with the rest
Advisory Group provided insights to help improve the project given falling under the external control level [59–66].
their unique perspectives [55]. While the purpose of the Advisory Group Based on the analysis of the AIM implementation of Shared Socioeco­
was fully transparent, the selection process of the Advisory Group was nomic Pathways using the IDSD comparative framework (Tables 7 and 8),
unclear. Documentation of all these meetings along with attendees were we offer the following recommendations to the SSP team. First, the AIM
available on the LADWP’s website [56]. In terms of Diversity, the var­ implementation of Shared Socioeconomic Pathways only considers trend-
iables used to differentiate the non-baseline scenarios include gross based scenarios. Second, the AIM implementation of Shared Socioeco­
domestic product growth for mixed type variables along with energy nomic Pathways does not consider internal control variables. We there­
efficiency improvements, technology adoption, and demand response fore recommend that the SSP Framework and AIM development teams
for demand side variables [54]. Gross domestic product growth is clas­ consider the reflection questions for the following metrics: scenario
sified as an external control variable, whereas the remaining variables progression and variable control to ensure the Internal Consistency and
are classified as limited control [54]. Diversity of their scenarios and scenario set.
Based on the analysis of the LA100 Study using the IDSD comparative
framework (Tables 7 and 8), we offer the following recommendations to 4. Discussion
the LA100 team. First, the development process of the LA100 Study had
basic narrative complexity, and the study team could review the Based on the case study results, we have identified four major find­
reflection questions in this area. Second, the LA100 Study only featured ings. First, the scenarios in the 2021 WEO, and to some extent the sce­
descriptive and trend-based scenarios. Lastly, the LA100 Study did not narios of the 2022 AEO and the Scout Core Measure Scenario Analysis lack
consider supply-type and internal control variables. We therefore the documentation or elements of the Development Process required to
recommend that the LA100 team consider the reflection questions for verify the Internal Consistency of their scenarios and, moreover, do not
the following metrics: narrative complexity, scenario outlook, scenario have the complexity and richness found in other scenarios. Second, there
progression, variable type, and variable control to ensure the Internal is a significant gap in Scenario Diversity with regards to the number of
Consistency and Diversity of their scenarios and scenario set. modeling frameworks which include normative and disruptive sce­
narios. Third, there is a significant lack in Variable Diversity in terms of
3.6. Case study 5: AIM implementation of Shared Socioeconomic the level of control stakeholders have over these variables. Finally, there
Pathways is a complete lack of explanation regarding the stakeholder selection
process. The following section will discuss these four findings and pro­
The Shared Socioeconomic Pathways (SSP) Framework was devel­ vide guidance for improvement.
oped by the Intergovernmental Panel on Climate Change with five sce­
narios, each at different levels of mitigation and adaptation challenges 4.1. Documentation gaps
[57]. Each scenario has a different marker scenario modeling framework
to generate the scenario results. Each modeling framework is an inte­ Using the criteria defined in the IDSD, we have identified several
grated assessment model which incorporates a variety of different sys­ areas that would significantly improve the Internal Consistency of the
tems (e.g., energy, economic, land use, air quality) across the entire 2021 WEO and further strengthen that of the 2022 AEO and the Scout
world with various regional divisions and time steps through the year Core Measure Scenario Analysis report. All reports could use additional
2100 [57]. We chose to analyze the Asia-Pacific Integrated Model (AIM) description of the Development Process in their documentation. For these
which is the marker model for SSP3 – Regional Rivalry [58]. However, reports, it is unclear whether the Development Process as defined by IDSD
unlike other marker models, AIM has the capabilities to analyze all other is truly limited in the narrative complexity, data utilization, and stake­
SSPs. AIM is a general equilibrium model with a recursive dynamic so­ holder engagement metrics, or whether this is just not well-described in
lution algorithm [57]. In the AIM implementation of Shared Socioeconomic the documentation. Regardless, either of these issues could limit the
Pathways article, the authors generated 22 scenarios across two di­ utility of these scenarios for other work. Given the widespread use of the
mensions. The first dimension is comprised of the five SSPs which are data created by these modeling frameworks, improving the Development
distinguished by several socio-economic variables. The second dimen­ Process of these scenarios could result in a myriad of positive impacts on
sion is comprised of six different climate conditions defined by their planning and policy making decisions that surround the energy system.
radiative forcing level in terms of W/m2. Each SSP features a baseline Given that three of five of the reports selected lacked either a satis­
scenario and three to four non-baseline scenarios. factory Development Process or level of documentation to verify Internal
What follows is a summary of IDSD evaluation of the AIM imple­ Consistency, it would not be surprising if other reports suffered from
mentation of Shared Socioeconomic Pathways. For Internal Consistency, similar issues. Internally inconsistent scenarios can lead to a misunder­
all documentation for the AIM and the SSP Framework were fully standing of the future and sub-optimal decisions. Therefore, our first
available online [57,58]. Detailed narratives were available and easily recommendation is for the entire energy modeling community to use the
accessible. Not only did an introductory journal article outline the SSP Development Process evaluation to ensure that 1) there is thoughtful
Framework and the various scenarios [57], each scenario and the AIM consideration of the perspectives which should be included in the

9
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

scenario development, 2) each scenario and each variable within these 4.3. Internal control variables
scenarios serves a specific purpose in the scenario analysis, and 3) the
data to describe the major trends in each scenario are examined and The third major finding is that there is a lack of Variable Diversity in
justified to match the goals of the scenario analysis. Beyond this, terms of the scenarios which use internal control variables. Only the
documentation of this development process is key to ensure a complete scenarios from the 2021 WEO and the Scout Core Measure Scenario
understanding of the scenario analysis, its utility, and limitations. To do Analysis have internal control variables. Thus, most energy modeling
this, we recommend that the energy modeling community considers the frameworks analyzed did not simulate a scenario where energy system
reflection questions in the Development Process evaluation and use the stakeholders would have full agency. The 2021 WEO scenarios show that
Access evaluation to ensure that the documentation of this process is widespread carbon taxes are critical to reducing, if not eliminating,
both available and accessible. In this way, energy modelers can ensure energy-related carbon emissions. The Scout Core Measure Scenario
that the scenarios they develop will feature plausible, coherent, and Analysis showed that even if the U.S. was to implement fuel switching
focused descriptions of possible futures. incentives, this measure, even combined with other efforts would not be
enough to fully decarbonize the U.S. residential building sector.
4.2. Scenario diversity These findings show the magnitude of change that can be imple­
mented relatively quickly through both policymaking and public and
The second major finding is that there is a lack of Scenario Diversity private energy programs. It is also possible to show the impact of these
across most reports evaluated especially in terms of normative and efforts in relation to other variables where we may only have limited
disruptive scenarios. The 2021 WEO and the AIM implementation of control. We recognize that there are many factors in the energy system
Shared Socioeconomic Pathways were the only reports to include outside of direct stakeholder control and that it is critical to understand
normative scenarios - the Sustainable Development, Net Zero Emissions the uncertainty of these factors. However, we believe that it is just as
by 2050, and SSP1 scenarios. The importance of these types of scenarios important to understand the scale of impact that energy system stake­
can be seen in the Net Zero Emissions by 2050 scenario which shows that holder decisions can have on the energy system. This type of analysis
to achieve this goal electricity would have to increase to be the majority allows for an understanding of what can and cannot be accomplished by
energy source by 2050 and that clean energy and infrastructure in­ different programs, policies, and initiatives.
vestments would need to triple by 2030 [37]. These specific findings Unfortunately, we found that the largest group of scenarios analyzed
give energy system stakeholders benchmarks to hit and it demonstrates were scenarios with a single variable of either limited or external con­
that these goals are attainable. These scenarios are important because trol. If this is representative of the larger energy modeling field, then the
they allow us to determine if pathways exist to achieve certain goals and greatest proportion of scenarios we are generating are versions of the
the scale of action needed to do so. Whereas descriptive projections have future where we have little to no control. The removal of agency in this
no explicit direction and could miss key opportunities. scenario development process creates a spectrum of futures which we
Similarly, there is an almost complete absence of disruptive sce­ can both easily conceive of, but which also, paradoxically, have un­
narios. The Scout Core Measure Scenario Analysis was the only report to avoidable consequences. Thus, our third recommendation is to urge the
include any disruptive scenarios – the HR 3T FS0 and HR 3T FS20 sce­ energy modeling community to subject each of their scenarios to the
narios. These were the only scenarios to explore the technical potential Variable Diversity evaluation of the IDSD framework to ensure that the
of technologies and simulate a marked disruption in their exogenous futures they are constructing offer actionable pathways towards a more
input data. By doing this, the authors of this study found that even sustainable, reliable, and just energy system.
through these unrealistic, boundary condition scenario definitions, it
was impossible to hit 2050 carbon reduction goals for the U.S. resi­ 4.4. Stakeholder selection process
dential building sector [46]. This is a powerful conclusion and it
completely changes this discussion. Even with the most ambitious The final major finding was the complete of lack description or
building technologies, high renewable electricity generation, and methodology of the stakeholder selection process across all the energy
incentivized fuel switching this target cannot be met. Therefore, to modeling frameworks we analyzed. While three of the five energy model
achieve this goal other, possibly more aggressive, measures must be frameworks analyzed feature purposeful stakeholder involvement, not
considered. even one outlined the methods for their selection process. Without this
It is understandable that energy modelers would stray away from information, it is impossible to judge if any stakeholders were not
disruptive scenarios, using historical data to extrapolate trends is a overlooked, nor if there was any selection bias by the scenario devel­
conservative approach to building a scenario. However, the assumption opment teams. Thus, all scenario development teams have an opportu­
that all variables will follow historical data is a highly unreliable nity to improve their stakeholder involvement by simply outlining the
assumption [67]. Furthermore, trend-based scenarios do not challenge process by which they went about selecting those who collaborated on
conventional thinking [29], the implications of not doing this include their work. Likewise, if no formal process was followed, these teams can
not understanding the sensitivities of different disruptions. Rather, the immediately improve the quality of their work by developing and
energy modeling field will be greatly enriched by an influx of disruptive deploying a selection protocol. Our final recommendation is for all
scenarios. Evaluating the impacts of different disruptions, of varying scenario development teams to document or begin to incorporate
magnitudes, occurring at different points in time would spark discussion stakeholder selection to immediately improve the stakeholder involve­
regarding the various inflection points in our energy, economic, and ment and internal consistency of their scenarios.
environmental systems.
Thus, our second recommendation is that more reports incorporate 5. Conclusion
normative and disruptive scenarios to increase the Diversity of the fu­
tures which are modeled. Normative scenarios will provide alternative Due to a level of subjectivity in some of the different evaluation
pathways and challenge the status quo found in descriptive scenarios, metrics along with different familiarity levels of those using this eval­
while disruptive scenarios will open dialogue regarding the implications uation with the reports they are evaluating, we recommend that all
and impacts of the different types of system shocks and how we can future scenario analyses by scenario development teams utilizing the
position ourselves to best, if not anticipate then, respond to an inevitable IDSD comparative framework implement inter-rater reliability approach
shift. to ensure rigorous analysis [68,69]. While we understand that this is a
time-consuming process, research has shown that the inclusion of a
single round of reliability checks by an independent reviewer will

10
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

increase the rigor of the analysis [70–72]. [12] Moss RH, Edmonds JA, Hibbard KA, Manning MR, Rose SK, van Vuuren DP, et al.
The next generation of scenarios for climate change research and assessment.
Our goal is that the IDSD comparative framework is used by the
Nature 2010;463(7282):747–56.
energy modeling community to improve the internal consistency and [13] Mohn K. The gravity of status quo: a review of IEA’s world energy outlook. Econ
diversity of scenarios reported to the greater energy system stakeholder Energy Environ Pol 2020;9:63+.
group. The focus on Internal Consistency and Diversity came out of a [14] Khatib H. IEA world energy outlook 2010—a comment. Energy Pol 2011;39(5):
2507–11.
recognition of the deficiencies found across various major energy [15] Gilbert AQ, Sovacool BK. Looking the wrong way: bias, renewable electricity, and
modeling framework reports. However, we understand that this is not a energy modelling in the United States. Energy 2016;94:533–41.
perfect tool and moreover we fully anticipate that as the scenario [16] Rosen RA. Why the shared socioeconomic pathway framework has not been useful
for improving climate change mitigation policy analysis. Technol Forecast Soc
development process continues to improve, so will the scope of this Change 2021;166:120611.
comparative framework expand as other needs arise. [17] Foster G. Concrete utopianism in integrated assessment models: discovering the
philosophy of the shared socioeconomic pathways. Energy Res Social Sci 2020;68:
101533.
Author contributions [18] Morrison R. Energy system modeling: public transparency, scientific
reproducibility, and open development. Energy Strategy Rev 2018;20:49–63.
Noah Sandoval: Conceptualization, Methodology, Validation, [19] Enserink B, Kwakkel JH, Veenman S. Coping with uncertainty in climate policy
making: (Mis)understanding scenario studies. Futures 2013;53:1–12.
Investigation, Writing – original draft, Writing – review & editing.Janet [20] Grunwald A. Energy futures: diversity and the need for assessment. Futures 2011;
L. Reyna: Methodology, Validation, Writing – review & editing, Funding 43(8):820–30.
acquisition.Amy E. Landis: Methodology, Writing – review & editing. [21] Nursimulu A. Assessment of future energy demand: a methodological review
providing guidance to developers and Users of energy models and scenarios. 2015.
[22] Newell R, Raimi D, Villanueva S, Prest B. Global energy outlook 2020: energy
transition or energy addition. Resour Future 2020.
Declaration of competing interest
[23] Cao K-K, Cebulla F, Gómez Vilchez JJ, Mousavi B, Prehofer S. Raising awareness in
model-based energy scenario studies—a transparency checklist. Energy, Sustaina
The authors declare that they have no known competing financial Soc 2016;6(1):1–20.
interests or personal relationships that could have appeared to influence [24] Funtowicz SO, Ravetz JR. Uncertainty and quality in science for policy. Springer
Science & Business Media; 1990.
the work reported in this paper. [25] Junne T, Xiao M, Xu L, Wang Z, Jochem P, Pregger T. How to assess the quality and
transparency of energy scenarios: results of a case study. Energy Strategy Rev 2019;
Data availability 26:100380.
[26] Hülk L, Müller B, Glauer M, Förster E, Schachler B. Transparency, reproducibility,
and quality of energy system analyses – a process to improve scientific work.
Data will be made available on request. Energy Strategy Rev 2018;22:264–9.
[27] Parry M, Carter T. Climate impact and adaptation assessment: a guide to the IPCC
approach. Int J Climatol 1999;19(4).
Acknowledgements [28] Rotmans J, van Asselt M, Anastasi C, Greeuw S, Mellors J, Peters S, et al. Visions for
a sustainable Europe. Futures 2000;32(9):809–31.
This work was authored in part by the National Renewable Energy [29] Rotmans J. Methods for IA: the challenges and opportunities ahead. Environ Model
Assess 1998;3(3):155–79.
Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U. [30] Parson EA. Global-change scenarios their development and use. Washington, D.C:
S. Department of Energy (DOE) under Contract No. DE-AC36- U.S. Climate Change Science Program; 2007.
08GO28308. Funding provided by U.S. Department of Energy Office of [31] Riahi K, van Vuuren DP, Kriegler E, Edmonds J, O’Neill BC, Fujimori S, et al. The
Shared Socioeconomic Pathways and their energy, land use, and greenhouse gas
Energy Efficiency and Renewable Energy Strategic Analysis Office. The emissions implications: an overview. Global Environ Change 2017;42:153–68.
views expressed in the article do not necessarily represent the views of [32] Janssen S, Ewert F, Li H, Athanasiadis IN, Jjf Wien, Thérond O, et al. Defining
the DOE of the U.S. Government. The U.S. Government retains and the assessment projects and scenarios for policy support: use of ontology in Integrated
Assessment and Modelling. Environ Model Software 2009;24(12):1491–500.
publisher, by accepting the article for publication, acknowledge that the [33] Appelrath H-J, Dieckhoff C, Fischedick M, Grunwald A, Höffler F, Mayer C, et al.
U.S. Government retains a nonexclusive, paid-up, irrevocable, world­ Consulting with energy scenarios. Requirements for scientific policy advice; 2016.
wide license to publish or reproduce the published form of this work, or [34] Wolfowitz P. Good governance and development: a time for action, vol. 11.
Jakarta, Indonesia: World Bank; 2006. p. 6.
allows other to do so, for U.S. Government purposes.
[35] Dahl C. International energy markets: understanding pricing, policies, and profits.
PennWell Books, LLC; 2015.
References [36] Kok K, Rothman DS, Patel M. Multi-scale narratives from an IA perspective: Part I.
European and Mediterranean scenario development. Futures J Pol, Plan Futures
Stud 2006;38(3):261–84.
[1] Mann ME, Bradley RS, Hughes MK. Northern hemisphere temperatures during the
[37] World energy outlook. International Energy Agency; 2021 [Available from:
past millennium: inferences, uncertainties, and limitations. Geophys Res Lett 1999;
https://www.iea.org/topics/world-energy-outlook.
26(6):759–62.
[38] World energy model documentation: October 2021. Paris: International Energy
[2] Foster V, Bedrosyan D. Understanding CO2 emissions from the global energy
Agency; 2021.
sector. The World Bank; 2014 May 2014.
[39] World energy outlook 2021. Paris: International Energy Agency; 2021.
[3] Levelized costs of new generation resources in the annual energy outlook 2021.
[40] Annual energy outlook 2022. Washington, D.C.: U.S. Energy Information
Washington, D.C.: U.S. Energy Information Administration; 2021 Februrary 2021.
Administration; 2022 March 2022.
[4] Bird L, Milligan M, Lew D. Integrating variable renewable energy: challenges and
[41] The national energy modeling system: an overview 2018. Washington, D.C.: U.S.
solutions. Golden, CO (United States): National Renewable Energy Lab. (NREL);
Energy Information Administration; 2019.
2013. Report No.: NREL/TP-6A20-60451 United States 10.2172/1097911 NREL
[42] AEO working groups Washington, D.C.: U.S. Energy Information Administration;
English.
[Available from: https://www.eia.gov/outlooks/aeo/workinggroup/.
[5] Krozer Y. Financing of the global shift to renewable energy and energy efficiency.
[43] Annual energy outlook 2022: case descriptions. Washington, D.C.: U.S. Energy
Green Finance 2019;1(3):264–78.
Information Administration; 2022 March 2022.
[6] Peterson GD, Cumming GS, Carpenter SR. Scenario planning: a tool for
[44] Harris C, Langevin J, Roth A, Phelan P, Parker A, Ball B, et al. Scout: an impact
conservation in an uncertain world. Conserv Biol 2003;17(2):358–66.
analysis tool for building energy-efficiency technologies: Washington. Golden, CO
[7] Schoemaker P. Multiple scenario development: its conceptual and behavioral
(United States): D.C.: American Council for an Energy-Efficient Economy (ACEEE);
foundation. Strat Manag J 1993;14:193–213.
National Renewable Energy Lab. (NREL); 2016. p. X. Medium.
[8] van Notten PWF, Rotmans J, van Asselt MBA, Rothman DS. An updated scenario
[45] Jared Langevin CBH, Satre-Meloy Aven, Janet L. Reyna. Scout core measures
typology. Futures 2003;35(5):423–43.
scenario analysis. Zenodo; 2021. Version 3 ed.
[9] Board TR, National Academies of Sciences E, Medicine. Strategic issues facing
[46] Langevin J, Harris CB, Reyna JL. Assessing the potential to reduce US building CO2
transportation. In: Hamilton BA, editor. Sustainability as an organizing Principle
emissions 80% by 2050. Joule 2019;3(10):2403–24.
for transportation Agencies, vol. 4. Washington, DC: The National Academies Press;
[47] Harris C. Scout: a Portfolio impact analysis tool for building energy efficiency
2014. p. 268.
technologies. U.S. Department of Energy: Energy Efficiency & Renewable Energy
[10] Alcamo J. Scenarios as tools for international environmental assessments.
Office; 2016 2 September 2016.
Copenhagen: European Environmental Agency; 2001.
[48] Cochran J, Paul Denholm, editors. The Los Angeles 100% renewable energy study.
[11] Therond O, Belhouchette H, Janssen S, Louhichi K, Ewert F, Bergez J-E, et al.
Golden, CO: National Renewable Energy Laboratory; 2021. Report No.: NREL/TP-
Methodology to translate policy assessment problems into scenarios: the example
6A20-79444.
of the SEAMLESS integrated framework. Environ Sci Pol 2009;12(5):619–30.

11
N. Sandoval et al. Renewable and Sustainable Energy Reviews 186 (2023) 113632

[49] ResStock analysis tool: National Renewable Energy Laboratory; [Available from: [58] Fujimori S, Hasegawa T, Masui T, Takahashi K, Herran DS, Dai H, et al. SSP3: AIM
https://www.nrel.gov/buildings/resstock.html. implementation of shared socioeconomic pathways. Global Environ Change 2017;
[50] ComStock analysis tool: National Renewable Energy Laboratory; [Available from: 42:268–83.
https://www.nrel.gov/buildings/comstock.html. [59] Kc S, Lutz W. The human core of the shared socioeconomic pathways: population
[51] EVI-Pro: Electric Vehicle Infrastructure - Projection Tool: National Renewable scenarios by age, sex and level of education for all countries to 2100. Global
Energy Laboratory; [Available from: https://www.nrel.gov/transportation/evi-pro Environ Change 2017;42:181–92.
.html. [60] Jiang L, O’Neill BC. Global urbanization projections for the shared socioeconomic
[52] 2015 Urban water Management plan. Los Angeles, California: Los Angeles pathways. Global Environ Change 2017;42:193–9.
Department of Water and Power; 2016. [61] Cuaresma JC. Income projections for climate change research: a framework based
[53] City of Los Angeles department of water and power 2017 retail electric Sales and on human capital dynamics. Global Environ Change 2017;42:226–36.
demand Forecast. Los Angeles, California: Los Angeles Department of Water and [62] Leimbach M, Kriegler E, Roming N, Schwanitz J. Future growth patterns of world
Power; 2017. regions–A GDP scenario approach. Global Environ Change 2017;42:215–25.
[54] Hale E, Anthony Fontanini, Wilson Eric, Henry Horsey, Parker Andrew, [63] Dellink R, Chateau J, Lanzi E, Magné B. Long-term economic growth projections in
Muratori Matteo, Colin McMillan KS, Mooney Meghan, Roberts David, the shared socioeconomic pathways. Global Environ Change 2017;42:200–14.
Reyna Janet, Rajendra, Adhikari CH, Scott Horowitz, Jones Dalton, Merket Noel, [64] Bauer N, Calvin K, Emmerling J, Fricko O, Fujimori S, Hilaire J, et al. Shared socio-
Pathak Maharshi, Joseph, Robertson AS, Bianchi Carlo, Bonnema Eric, economic pathways of the energy sector – quantifying the narratives. Global
Dahlhausen Matthew, Marlena, Praprost LZ, Wood Eric, Lee Dong-Yeon, Environ Change 2017;42:316–30.
Neuman Christopher, Oliveira Ricardo, Angineh Zohrabian JL. Chapter 3: [65] Popp A, Calvin K, Fujimori S, Havlik P, Humpenöder F, Stehfest E, et al. Land-use
electricity demand projections. In: The Los Angeles 100% renewable energy study. futures in the shared socio-economic pathways. Global Environ Change 2017;42:
Golden, CO: National Renewable Energy Laboratory; 2021. Report No.: NREL/TP- 331–45.
6A20-79444-3. [66] Rao S, Klimont Z, Smith SJ, Van Dingenen R, Dentener F, Bouwman L, et al. Future
[55] Cochran J, Paul Denholm, Mooney Meghan, Steinberg Daniel, McCamey Devonie, air pollution in the shared socio-economic pathways. Global Environ Change 2017;
Palmintier B. Chapter 2. Study approach. In: The Los Angeles 100% renewable 42:346–58.
energy study. Golden, CO: National Renewable Energy Laboratory; 2021. Report [67] Nations U. Global change and sustainable development: critical trends. United
No.: NREL/TP-6A20-79444-2. Nations Chronicle 1997;34(2):5.
[56] 100 % Renewable energy study: Los Angeles department of water and power; [68] Belotto MJ. Data analysis methods for qualitative research: managing the
[Available from: https://www.ladwp.com/ladwp/faces/ladwp/aboutus/a-po challenges of coding, interrater reliability, and thematic analysis. Qual Rep 2018;
wer/a-p-cleanenergyfuture/a-p-renewableenergystudy;jsessionid=GclLh 23(11):2622.
SGQqCGZ28pCgDPGs71M271TRl9Z0tChGT7vn37hshT9QfkT!1029959098? [69] Thomas E, Magilvy JK. Qualitative rigor or research validity in qualitative
_afrLoop=225085697395698&_afrWindowMode=0&_afrWindowId=null#%40% research. J Spec Pediatr Nurs (JSPN) 2011;16(2):151–5.
3F_afrWindowId%3Dnull%26_afrLoop%3D225085697395698%26_afrWindowMo [70] Barbour RS. Checklists for improving rigour in qualitative research: a case of the
de%3D0%26_adf.ctrl-state%3Drwkj9sloa_4. tail wagging the dog? BMJ 2001;322(7294):1115–7.
[57] O’Neill BC, Kriegler E, Ebi KL, Kemp-Benedict E, Riahi K, Rothman DS, et al. The [71] Campbell JL, Quincy C, Osserman J, Pedersen OK. Coding in-depth semistructured
roads ahead: narratives for shared socioeconomic pathways describing world interviews:problems of unitization and intercoder reliability and agreement. Socio
futures in the 21st century. Global Environ Change 2017;42:169–80. Methods Res 2013;42(3):294–320.
[72] Creswell JW. Research design : qualitative, quantitative, and mixed methods
approaches. third ed. Los Angeles: Sage; 2009.

12