Sensitivity analysis

for risk-related decision-making

Eric Marsden
<[email protected]>

What are the key drivers of my modelling results?

Sensitivity analysis: intuition

X is a sensitive
parameter

Degree of sensitivity of X = ΔY/ΔX

X is not a sensitive
parameter
Sensitivity analysis: intuition

5
𝑓 (𝑥, 𝑦) is sensitive in 𝑥 and in
𝑦

0
2

−2 0
−1
0 𝑦
1
𝑥 2 −2
Sensitivity analysis: intuition

2
𝑓 (𝑥, 𝑦) is sensitive in 𝑥 but
not in 𝑦

2
0
−2 0
−1
0 𝑦
1
𝑥 2 −2
Sensitivity analysis: intuition

▷ Consider a case where 𝑌 = 𝑓 (𝑥) and the

function 𝑓 is a “black box”

▷ We can sample 𝑌 for different values of 𝑋 to

reassemble the relationship

▷ Here: 𝑋 is a sensitive parameter

Sensitivity analysis: intuition

▷ Here, 𝑋 is not sensitive

▷ Can be “seen” visually

X
Sensitivity analysis: intuition

▷ What can we say about the sensitivity of 𝑋?

▷ No graphical interpretation

▷ Consider also functions (or computer models,

or spreadsheets) which have tens of inputs
• you can’t draw graphs in dimension 23!

▷ We need a more sophisticated method than

scatterplots…
What is sensitivity analysis?

▷ The study of how the variation (uncertainty) in the output of a

mathematical model can be apportioned, qualitatively or quantitatively, to
different sources of variation in the model inputs

▷ Answers the question “What makes a difference in this decision problem?”

▷ Can be used to determine whether further research is needed to reduce

input uncertainty before making a decision
• information is not free
Sensitivity and uncertainty analysis

Start with a simulation model that

you want better to understand (often
a computer model). It may be a “black
box” (you don’t know how it works
simulation internally).
model

Adapted from figure by A. Saltelli

Sensitivity and uncertainty analysis

Start with a simulation model that

you want better to understand (often
parameter 1 a computer model). It may be a “black
box” (you don’t know how it works
simulation internally).
model
Define which uncertain input parameters
parameter 2
you want to analyze.

Characterize their probability

distributions.
parameter 3

Adapted from figure by A. Saltelli

Sensitivity and uncertainty analysis

parameter 1 Propagate the input variability to

the output variability by running the
model a large number of times with
simulation
inputs taken from the input probability
model outputs
distributions.
parameter 2

(This is a “Monte Carlo” or “stochastic

simulation” method.)

parameter 3

Adapted from figure by A. Saltelli

Sensitivity and uncertainty analysis

Uncertainty analysis: how does

variability in the inputs propagate
parameter 1 through the model to variability in the
outputs?
uncertainty analysis

simulation
model outputs
parameter 2

parameter 3

Adapted from figure by A. Saltelli

Sensitivity and uncertainty analysis

Uncertainty analysis: how does

variability in the inputs propagate
parameter 1 through the model to variability in the
outputs?
uncertainty analysis

simulation
model outputs
parameter 2 sensitivity analysis Sensitivity analysis: what is the
relative contribution of the variability
in each of the inputs to the total output
variability?
parameter 3

Adapted from figure by A. Saltelli

Sensitivity and uncertainty analysis

parameter 1

uncertainty analysis

simulation
model outputs
parameter 2 sensitivity analysis

Insights gained from the sensitivity

analysis may help to
▷ prioritize effort on reducing input
parameter 3
uncertainties
▷ improve the simulation model

feedback on model
inputs & model structure

Adapted from figure by A. Saltelli

Applications of sensitivity analysis
▷ Risk communication
• how much of my output uncertainty is irreducible (caused by aleatory
uncertainty in input parameters)?
• how much is epistemic (related to lack of knowledge, could be reduced with
more research)?

▷ Optimize research investment to improve risk analysis

• which uncertain input parameters contribute the most to model output
uncertainty?
• on which uncertain input parameters should I spend my research money to
gain the biggest reduction in uncertainty?

▷ Model reduction
• identify ineffective parameters
• generate models with fewer parameters, but (almost) identical results
(metamodels or response surfaces)
Application areas

The European Commission recommends sensitivity

analysis in the context of its impact assessment
guidelines (2009):

‘‘
When the assumptions underlying the baseline scenario
might vary as a result of external factors, you need to do
a sensitivity analysis to assess whether the impacts of the
policy options differ significantly for different values of
the key variables.

Source: EC guidance document at ec.europa.eu/smart-regulation/impact/

Application areas

Principles for Risk Analysis published by the US Office of

Management and Budget:

‘‘
Influential risk assessments should characterize uncertainty with a
sensitivity analysis and, where feasible, through use of a numeric
distribution.
[…] Sensitivity analysis is particularly useful in pinpointing which
assumptions are appropriate candidates for additional data
collection to narrow the degree of uncertainty in the results.
Sensitivity analysis is generally considered a minimum, necessary
component of a quality risk assessment report.

Source: Updated principles for risk analysis, US OMB, whitehouse.gov/sites/default/files/omb/assets/regulatory_matters_pdf/m07-

24.pdf
Sensitivity analysis: the process

1 Specify the objective of your analysis

• Example: “Which variables have the most impact on the level of risk?”

2 Build a model which is suitable for automated numerical analysis

• Example: spreadsheet with inputs in certain cells and the output of interest in
another cell
• Example: computer code that can be run in batch mode

3 Select a sensitivity analysis method

• most appropriate method will depend on your objectives, the time available for
the analysis, the execution cost of the model

4 Run the analysis

5 Present results to decision-makers

Step 2: build a model which can be run by a computer

A useful starting point is to build

an influence diagram of the
relevant variables.
Use this diagram to build your
model (e.g. profit = revenue -
expenses), or check that all
relevant variables are integrated
in your existing numerical model.

It should be possible to run the

model in an automated way
(called “batch mode”).

Figure from Clemen, R. T., Making Hard Decisions: An Introduction to Decision Analysis, 1996
Aside: extracting a model from a spreadsheet

In some cases, the model you want to analyze

will already be implemented in a spreadsheet
such as Microsoft Excel.
The free Trace plugin for Excel can extract a
dependency graphs that depict relationships
between cells in your spreadsheet. This can
help to build a model that is easy to run in
batch mode.

Trace tool can be downloaded for free from christopherteh.com/trace/

Step 3: sensitivity analysis methods

1 Basic approach: tornado diagram

2 Screening methods

increasing
3 oat “one at a time” methods sophistication

4 Local sensitivity analysis

more information
available
5 Global sensitivity analysis
Basic SA: tornado diagram

▷ Tornado diagram: way of presenting basic sensitivity information

• mostly used for project risk management or net present value estimates
• sometimes called “what-if” analysis

the uncertain parameter with large

input parameters impact on the output

Market Size

Our Share

Selling Price

Fixed Costs parameter with small

Variable Cost
impact on the output

0 100 200 300 400 500 600

output value when all range of possible

inputs at their mean value output values
 Market Size

Our Share

How to produce a tornado diagram Selling Price

Fixed Costs

Variable Cost

0 100 200 300 400 500 600

▷ Determine lower bound, upper bound and best estimate of each uncertain
input parameter
• = 10%, 90% and 50% quantiles of parameter’s probability distribution

▷ For each uncertain parameter, calculate model output for lower and upper
bounds, while taking best estimate for all other uncertain parameters

▷ Draw a horizontal bar for each uncertain parameter between value for
lower bound and value for upper bound

▷ Vertical order of uncertain parameters given by width of the bar

• parameters which lead to large output “spread” (have more impact) at top

▷ Draw a vertical line at position of the expected value

• calculated using best estimate for each uncertain parameter
Tornado diagram example: profit calculation

Profit = (SellingPrice - VariableCost) × MarketSize × MarketShare - FixedCosts

Parameter Lower Expected Upper

Selling price 140 175 200 Market Size

Our Share
Market size 8 12 20 Selling Price

Fixed Costs
Our share 0.18 0.25 0.35
Variable Cost
Variable cost 30 40 60 0 100 200 300 400 500 600

Fixed costs 150 180 300

Interpretation: given these assumptions, the market size parameter has

most influence on profitability.

Plot generated with free Excel plugin by home.uchicago.edu/ rmyerson/addins.htm

Relevant commercial tools

Example tools with Excel integration:

▷ Palisade TopRank®

▷ Oracle Crystal Ball®

Typically quite expensive…

Screening methods
▷ “Screening” is a preliminary phase that is useful when the model has a
large number of parameters
• allows the identification, with a limited number of calculations, of those
parameters that generate significant variability in the model’s output

▷ Simple “oat” (one at a time) screening method: change one factor at a

time and look at effect on output
• while keeping other factors at their nominal value

, Intuitive approach, can be undertaken by hand

, If model fails, you know which factor is responsible for the failure

/ Approach does not fully explore input space, since simultaneous

variations of input variables are not studied

/ Cannot detect the presence of interactions between input variables

OAT screening method: example Real modelling sit
uations
ma ny more tha n two
have
ou tpu t cannot
Rosenbrock function: 𝑓 (𝑥1 , 𝑥2 ) = 100(𝑥2 − 𝑥12 )2 + (1 − 𝑥1 )2 variables, so
be plo tte d
over [-2, 2]²

101 > def rosenbrock(x1, x2):

1 return 100*(x2-x1**2)**2 + (1-x1)**2
> rosenbrock(0, 0)
1 100 1
0 > rosenbrock(1, 0)
𝑥2

100
> rosenbrock(0, 1)
−1 101
> rosenbrock(1, 1)
0
> rosenbrock(-1, -1)
−2
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2 404
𝑥1

Both 𝑥1 and 𝑥2 seem to be sensitive variables in this

example
The Elementary Effects screening method

▷ The elementary effect for the 𝑖-th input variable at x ∈ [0, 1]𝑘 is the first
difference approximation to the derivative of 𝑓 (·) at x:

𝑓 (x + ∆e𝑖 ) − 𝑓 (x)
𝐸𝐸𝑖 (x) =
∆
where e𝑖 is the unit vector in the direction of the 𝑖-th axis

▷ Intuition: it’s the slope of the secant line parallel to the input axis

▷ Average 𝐸𝐸𝑖 (x) for various points x in the input domain to obtain a
measure of the relative influence of each factor

1 𝑟
𝜇𝑖 = ∑ ∣𝐸𝐸𝑖 (𝑥𝑗 )∣
𝑟 𝑗=1
Elementary effects method: example
▷ Consider 𝑦(x) = 1.0 + 1.5𝑥2 + 1.5𝑥3 + 0.6𝑥4 + 1.7𝑥42 + 0.7𝑥5 + 0.8𝑥6 + 0.5(𝑥5 𝑥6 )
where
• x = (𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 , 𝑥5 , 𝑥6 )
Elementary effects
• 0 ≤ 𝑥1 , 𝑥2 , 𝑥4 , 𝑥5 , 𝑥6 ≤ 1
μ_6

• 0 ≤ 𝑥3 ≤ 5
μ_5

μ_4
▷ Note:
μ_3
• 𝑦(·) is functionally independent of 𝑥1
μ_2

• 𝑦(·) is linear in 𝑥2 and 𝑥3 and non-linear in 𝑥4 μ_1

• 𝑦(·) contains an interaction in 𝑥5 and 𝑥6

0.0 0.5 1.0 1.5 2.0 2.5 3.0
Relative sensitivity

▷ Sensitivity results:
• 𝜇1 = 0 as expected tails as a
Download full de
teb oo k at
Python no
• influence of 𝑥4 is highest ring.org
risk-enginee
• influence of 𝑥2 and 𝑥3 is equal, as expected
Local sensitivity analysis methods

▷ Local sensitivity analysis: investigation of response stability over a small

region of inputs

▷ Local sensitivity with respect to a factor is just the partial derivative wrt
that factor, evaluated at that location

▷ Simple example: Rosenbrock function

• 𝑓 (𝑥1 , 𝑥2 ) = 100(𝑥2 − 𝑥12 )2 + (1 − 𝑥1 )2 , 𝑥1 , 𝑥2 ∈ [−2, 2]
𝜕𝑓
• 𝜕𝑥1 = −400𝑥1 (−𝑥12 + 𝑥2 ) + 2𝑥1 − 2
𝜕𝑓
• 𝜕𝑥2 = −200𝑥12 + 200𝑥2
Rosenbrock example

𝑓 (𝑥1 , 𝑥2 ) = 100(𝑥2 − 𝑥12 )2 + (1 − 𝑥1 )2 , 𝑥1 , 𝑥2 ∈ [−2, 2]

𝜕𝑓
𝜕𝑥1 = -155
(-1.5,2)
𝜕𝑓
𝜕𝑥2 = -50

2,000 2
Local sensitivity is low
1
0
−2 0
−1 𝑥2
0 −1
1
𝑥1 2 −2
Rosenbrock example

𝑓 (𝑥1 , 𝑥2 ) = 100(𝑥2 − 𝑥12 )2 + (1 − 𝑥1 )2 , 𝑥1 , 𝑥2 ∈ [−2, 2]

𝜕𝑓
𝜕𝑥1 = -4806

(-2,-2) 𝜕𝑓
𝜕𝑥2 = -1200

2,000 2
Local sensitivity is high
1
0
−2 0
−1 𝑥2
0 −1
1
𝑥1 2 −2
Rosenbrock example

𝑓 (𝑥1 , 𝑥2 ) = 100(𝑥2 − 𝑥12 )2 + (1 − 𝑥1 )2 , 𝑥1 , 𝑥2 ∈ [−2, 2]

𝜕𝑓
𝜕𝑥1 =0

𝜕𝑓
𝜕𝑥2 =0
(1,1)
2,000 2
Local sensitivity is zero
1
0
−2 0
−1 𝑥2
0 −1
1
𝑥1 2 −2
Local sensitivity analysis methods

▷ The calculation of partial derivatives can be automated for software

packages using automatic differentiation methods
• the source code must be available
• → autodiff.org

▷ This method does not allow you to detect interaction effects between
the input variables

▷ Method cannot handle correlated inputs

▷ Widely used for optimization of scientific software

Global sensitivity analysis methods

▷ Examine effect of changes to all input variables simultaneously

• over the entire input space you are interested in (for example the uncertainty
distribution of each variable)

▷ Methods based on analysis of variance (often using Monte Carlo methods)

▷ Typical methods that are implemented in software packages:

• Fourier Analysis Sensitivity Test (fast), based on a multi-dimensional Fourier
transform
• the method of Sobol’

▷ In general, this is the most relevant method for risk analysis purposes
• allows the analysis of interactions between input variables
Sensitivity indices

▷ The sensitivity index of a parameter quantifies its impact on output

uncertainty
• measures the part of output variance which can be attributed to variability in
the parameter

▷ Properties:
• 𝑆𝑖 ∈ [0,1]
• ∑ 𝑖 𝑆𝑖 = 1

Var(𝔼[𝑧|𝑥𝑗 ])
▷ First-order index: 𝑆𝑗 = Var(𝑧)
• measures “main effect”
𝔼[Var(𝑧|𝑥𝑗 )]
▷ Total effect index: 𝑇𝑗 = Var(𝑧)
• measures residual variability due to interactions between 𝑥𝑖 and other
parameters
Estimating sensitivity indices using SciPy and SALib

import numpy
from SALib.sample import saltelli
from SALib.analyze import sobol

def rosenbrock(x1, x2):

return 100 * (x2 - x1**2)**2 + (1 - x1)**2

problem = {
"num_vars": 2,
"names": ["x1", "x2"],
"bounds": [[-2, 2], [-2, 2]]
}
sample = saltelli.sample(problem, N, calc_second_order=True) Download full de
tails as a
Y = numpy.empty([sample.shape[0]]) Python noteb oo k at
for i in range(len(Y)): ring.org
risk-enginee
x = sample[i]
Y[i] = rosenbrock(x[0], x[1])
Si = sobol.analyze(problem, Y, calc_second_order=True)

The SALib python library is free software available from github.com/SALib/SALib

 ▷ Appendix on sensitivity analysis of the epa guidance on risk assessment,
epa.gov/oswer/riskassessment/rags3adt/pdf/appendixa.pdf
▷ OpenTURNS software platform for uncertainty analysis (free software),
Further openturns.org
reading ▷ Dakota software platform for uncertainty analysis (free software),
dakota.sandia.gov
▷ Case study: Using @RISK for a Drug Development Decision: a Classroom
Example, from palisade.com/cases/ISU_Pharma.asp

For more free content on risk engineering,

visit risk-engineering.org
Feedback welcome!
This presentation is distributed under the terms of the
Creative Commons Attribution – Share Alike licence

@LearnRiskEng

fb.me/RiskEngineering
Was some of the content unclear? Which parts were most useful to
you? Your comments to [email protected]
(email) or @LearnRiskEng (Twitter) will help us to improve these
materials. Thanks!

For more free content on risk engineering,

visit risk-engineering.org