See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/318175970

OpenTURNS: An Industrial Software for Uncertainty Quantification in

Simulation

Chapter · June 2017

DOI: 10.1007/978-3-319-12385-1_64

CITATIONS READS

143 3,412

4 authors, including:

Michaël Baudin Régis Lebrun

Électricité de France (EDF) Airbus Central Research & Technology
38 PUBLICATIONS 519 CITATIONS 28 PUBLICATIONS 1,189 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Régis Lebrun on 21 March 2018.

The user has requested enhancement of the downloaded file.

OpenTURNS: An Industrial Software for
Uncertainty Quantification in Simulation

Michaël Baudin, Anne Dutfoy, Bertrand Iooss, and

Anne-Laure Popelin

Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1 Presentation of OpenTURNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 The Uncertainty Management Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Main Originality of OpenTURNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 The Flooding Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Uncertainty Quantification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Modeling of a Random Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Statistics Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Conditioned Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Bayesian Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3 Uncertainty Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1 Min-Max Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2 Central Tendency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 Failure Probability Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.1 Graphical Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2 Sampling-Based Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5 Metamodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

M. Baudin ()
Industrial Risk Management Department, EDF R&D France, Chatou, France
e-mail: [email protected]
A. Dutfoy
Industrial Risk Management Department, EDF R&D France, Saclay, France
e-mail: [email protected]; [email protected]
A.-L. Popelin
Industrial Risk Management Department, EDF R&D France, Chatou, France
e-mail: [email protected]
B. Iooss
Industrial Risk Management Department, EDF R&D France, Chatou, France

© Springer International Publishing Switzerland 2016 1

R. Ghanem et al. (eds.), Handbook of Uncertainty Quantification,
DOI 10.1007/978-3-319-11259-6_64-1
2 M. Baudin et al.

5.1 Polynomial Chaos Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.2 The Kriging Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6 The External Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.1 Fast Evaluations of G . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.2 Evaluation of the Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
6.3 High-Performance Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Abstract
The needs to assess robust performances for complex systems and to answer
tighter regulatory processes (security, safety, environmental control, health
impacts, etc.) have led to the emergence of a new industrial simulation challenge:
to take uncertainties into account when dealing with complex numerical
simulation frameworks. Therefore, a generic methodology has emerged from
the joint effort of several industrial companies and academic institutions. EDF
R&D, Airbus Group, and Phimeca Engineering started a collaboration at the
beginning of 2005, joined by IMACS in 2014, for the development of an open-
source software platform dedicated to uncertainty propagation by probabilistic
methods, named OpenTURNS for open-source treatment of uncertainty, Risk
’N Statistics. OpenTURNS addresses the specific industrial challenges attached
to uncertainties, which are transparency, genericity, modularity, and multi-
accessibility. This paper focuses on OpenTURNS and presents its main features:
OpenTURNS is an open- source software under the LGPL license that presents
itself as a C++ library and a Python TUI and which works under Linux and
Windows environment. All the methodological tools are described in the different
sections of this paper: uncertainty quantification, uncertainty propagation,
sensitivity analysis, and metamodeling. A section also explains the generic
wrappers’ way to link OpenTURNS to any external code. The paper illustrates
as much as possible the methodological tools on an educational example that
simulates the height of a river and compares it to the height of a dike that protects
industrial facilities. At last, it gives an overview of the main developments
planned for the next few years.

Keywords
OpenTURNS • Uncertainty • Quantification • Propagation • Estimation •
Sensitivity • Simulation • Probability • Statistics • Random vectors • Mul-
tivariate distribution • Open source • Python module • C++ library • Trans-
parency • Genericity

1 Introduction

The needs to assess robust performances for complex systems and to answer tighter
regulatory processes (security, safety, environmental control, health impacts, etc.)
have led to the emergence of a new industrial simulation challenge: to take uncer-
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 3

tainties into account when dealing with complex numerical simulation frameworks.
Many attempts at treating uncertainty in large industrial applications have involved
domain-specific approaches or standards: metrology, reliability, differential-based
approaches, variance decomposition, etc. However, facing the questioning of their
certification authorities in an increasing number of different domains, these domain-
specific approaches are no more appropriate. Therefore, a generic methodology
has emerged from the joint effort of several industrial companies and academic
institutions: [28] reviews these past developments. The specific industrial challenges
attached to the recent uncertainty concerns are:

• Transparency: open consensus that can be understood by outside authorities and

experts
• Genericity: multi-domain issue that involves various actors along the study
• Modularity: easy integration of innovations from the open-source community
• Multi-accessibility: different levels of use (simple computation, detailed quan-
titative results, and deep graphical analyses) and different types of end users
(graphical interface, Python interpreter, and C++ sources)
• Industrial computing capabilities: to secure the challenging number of simula-
tions required by uncertainty treatment

As no software was fully answering the challenges mentioned above, EDF R&D,
Airbus Group, and Phimeca Engineering started a collaboration at the beginning
of 2005, joined by IMACS in 2014, for the development of an open-source
software platform dedicated to uncertainty propagation by probabilistic methods,
named OpenTURNS for open-source treatment of uncertainty, Risk ’N Statistics
[10, 29]. OpenTURNS is actively supported by its core team of four industrial
partners (IMACS joined the consortium in 2014) and its industrial and academic
user community that meet through the website www.openturns.org and annually
during the OpenTURNS User’s Day. At EDF, OpenTURNS is the repository of
all scientific developments on this subject, to ensure their dissemination within the
several business units of the company. The software has also been distributed for
several years via the integrating platform Salome [27].

1.1 Presentation of OpenTURNS

OpenTURNS is an open-source software under the LGPL license that presents itself
as a C++ library and a Python TUI and which works under Linux and Windows
environment, with the following key features:

• Open-source initiative to secure the transparency of the approach and its open-
ness to ongoing research and development (R&D) and expert challenging
• Generic to the physical or industrial domains for treating of multi-physical
problems
• Structured in a practitioner guidance methodological approach
4 M. Baudin et al.

• With advanced industrial computing capabilities, enabling the use of massive dis-
tribution and high-performance computing, various engineering environments,
large data models, etc.
• Includes the largest variety of qualified algorithms in order to manage uncertain-
ties in several situations
• Contains complete documentation (reference guide, use cases guide, user man-
ual, examples guide, and developers’ guide)

All the methodological tools are described after this introduction in the different
sections of this paper: uncertainty quantification, uncertainty propagation, sensitiv-
ity analysis, and metamodeling. Before the conclusion, a section also explains the
generic wrappers’ way to link OpenTURNS to any external code.
OpenTURNS can be downloaded from its dedicated website www.openturns.
org which offers different pre-compiled packages specific to several Windows and
Linux environments. It is also possible to download the source files from the
SourceForge server (www.sourceforge.net) and to compile them within another
environment: the OpenTURNS developer’s guide provides advice to help compiling
the source files. At last, OpenTURNS has been integrated for more than 5 years in
the major Linux distributions (e.g., debian, ubuntu, redhat, and suze).

1.2 The Uncertainty Management Methodology

The uncertainty management generic methodology [29] is schematized in Fig. 1. It

consists of the following steps:

• Step A: specify the random inputs X , the deterministic inputs d, the model G
(analytical, complex computer code or experimental process), the variable of
interest (model output) Y , and the quantity of interest on the output (central dis-
persion, its distribution, probability to exceed a threshold, etc.). The fundamental
relation writes:

Y D G.X ; d / D G.X /; (1)

with X D .X1 ; : : : ; Xd /.
• Step B: quantify the sources of uncertainty. This step consists in modeling the
joint probability density function (pdf) of the random input vector by direct
methods (e.g., statistical fitting, expert judgment) [15].
• Step B’: quantify the sources of uncertainty by indirect methods using some real
observations of the model outputs [39]. The calibration process aims to estimate
the values or the pdf of the inputs, while the validation process aims to model the
bias between the model and the real system.
• Step C: propagate uncertainties to estimate the quantity of interest. With respect
to this quantity, the computational resources, and the CPU time cost of a single
model run, various methods will be applied: analytical formula, geometrical
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 5

Fig. 1 The uncertainty management methodology

approximations, Monte Carlo sampling strategies, metamodel-based techniques,

etc. [11, 21].
• Step C’: analyze the sensitivity of the quantity of interest to the inputs in order to
rank uncertainty sources [14, 36].

For each of these steps, OpenTURNS offers a large number of different methods
whose applicability depend on the specificity of the problem (dimension of inputs,
model complexity, CPU time cost for a model run, quantity of interest, etc.).

1.3 Main Originality of OpenTURNS

OpenTURNS is innovative in several aspects. Its input data model is based on

the multivariate cumulative distribution function (CDF). This enables the usual
sampling approach, as would be appropriate for statistical manipulation of large
datasets, but also facilitates analytical approaches. Distributions are classified
(continuous, discrete, elliptic, etc.) in order to take the best benefit of their properties
in algorithms. If possible, the exact final cumulative density function is determined
(thanks to characteristic functions implemented for each distribution, the Poisson
summation formula, the Cauchy integral formula, etc.).
OpenTURNS explicitly models the dependence with copulas, using the Sklar the-
orem. Furthermore, different sophisticated analytical treatments may be explored:
6 M. Baudin et al.

aggregation of copulas, composition of functions from Rn into Rd , extraction of

copula, and marginals from any distribution.
OpenTURNS defines a domain-specific oriented object language for probabil-
ity modeling and uncertainty management. This way, the objects correspond to
mathematical concepts and their interrelations map the relations between these
mathematical concepts. Each object proposes sophisticated treatments in a very
simple interface.
OpenTURNS implements up-to-date and efficient sampling algorithms
(Mersenne-Twister algorithm, Ziggurat method, the Sequential Rejection Method,
etc.). Exact Kolmogorov statistics are evaluated with the Marsaglia method
and the noncentral Student and noncentral 2 distribution with the Benton and
Krishnamoorthy method.
OpenTURNS is the repository of recent results of PhD research carried out at
EDF R&D: for instance, the sparse polynomial chaos expansion method based on
the LARS method [7], the adaptive directional stratification method [25] which is
an accelerated Monte Carlo sampling technique, and the maximum entropy order
statistics copulas [20].

1.4 The Flooding Model

Throughout this paper, the discussion is illustrated with a simple application model
that simulates the height of a river and compares it to the height of a dike
that protects industrial facilities as illustrated in Fig. 2. When the river height
exceeds that of the dike, flooding occurs. This academic model is used as a
pedagogical example in [14]. The model is based on a crude simplification of the
1D hydrodynamical equations of Saint-Venant under the assumptions of uniform

Fig. 2 The flood example:

simplified model of a river
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 7

Table 1 Input variables of the flood model and their probability distributions
Input Description Unit Probability distribution
Q Maximal annual flow rate m3 /s Gumbel G .1:8e 3 ; 1014/
Ks Strickler coefficient – Normal N .30; 7:5/
Zv River downstream level m Triangular T .47:6; 50:5; 52:4/
Zm River upstream level m Triangular T .52:5; 54:9; 57:7/

and constant flow rate and large rectangular sections. It consists of an equation that
involves the characteristics of the river stretch:
0 10:6
B Q C
H D@ q A ; (2)
Zm Zv
BKs L

where the output variable H is the maximal annual height of the river, B is the river
width, and L is the length of the river stretch. The four random input variables
Q, Ks , Zv , and Zm are defined in Table 1 with their probability distribution.
The randomness of these variables is due to their spatiotemporal variability, our
ignorance of their true value, or some inaccuracies of their estimation.

2 Uncertainty Quantification

2.1 Modeling of a Random Vector

OpenTURNS implements more than 40 parametric distributions which are continu-

ous (more than 30 families) and discrete (more than 10 families), with several sets
of parameters for each one. Some are multivariate, such as the Student distribution
or the normal one.
Moreover, OpenTURNS enables the building of a wide variety of multivariate
distributions, thanks to the combination of the univariate margins and a depen-
dence structure, the copula, according to the Sklar theorem: F .x1 ; : : : ; xd / D
C .F1 .x1 /; : : : ; Fd .xd // where Fi is the CDF of the margin Xi and C W Œ0; 1d !
Œ0; 1 the copula.
OpenTURNS proposes more than 10 parametric families of copula: Clayton,
Frank, Gumbel, Farlie-Morgenstein, etc. These copula can be aggregated to build
the copula of a random vector whose components are dependent by blocks. Using
the inverse relation of the Sklar theorem, OpenTURNS can extract the copula of
any multivariate distribution, whatever the way it has been set up: for example,
from a multivariate distribution estimated from a sample with the kernel smoothing
technique.
All the distributions can be truncated in their lower and/or upper area. In addition
to these models, OpenTURNS proposes other specific constructions. Among them,
8 M. Baudin et al.

note the random vector which writes as a linear combination of a finite set of
independent variables: X D a0 Ca1 X 1 C: : : aN X N , thanks to the Python command,
written for N D 2 with explicit notations:
>>>myX= RandomMixture ( [ d i s t X 1 , d i s t X 2 ] , [ a1 , a2 ] , a0 )

In that case, the distribution of X is exactly determined, using the characteristic

functions of the Xi distributions and the Poisson summation formula.
OpenTURNS also easily models the random vector whose probability density
function (pdf) is a linear combination of a finite set of independent pdf: fX D
a1 fX 1 C : : : aN fX N , thanks to the Python command, with the same notations as
previously (the weights are automatically normalized):
>>>mypdfX= M i x t u r e ( [ d i s t X 1 , d i s t X 2 ] , [ a1 , a2 ] )

Moreover, OpenTURNS implements a random vector that writes as the random

sum of univariate independent and identically distributed variables,
P this randomness
being distributed according to a Poisson distribution: X D N i D1 X i ; N  P./,
thanks to the Python command:
>>>d= C o m p o u n d D i s t r i b u t i o n ( lambda , d i s t X )

where all the variables Xi are identically distributed according to distX. In that case,
the distribution of X is exactly determined, using the characteristic functions of the
Xi distributions and the Poisson summation formula.
In the univariate case, OpenTURNS exactly determines the pushforward distri-
bution D of any distribution D0 through the function f W R ! R, thanks to the
Python command (with straight notations):
>>>d= C o m p o s i t e D i s t r i b u t i o n ( f , d0 )

Finally, OpenTURNS enables the modeling of a random vector .X1 ; : : : ; Xd /

which almost surely verifies the constraint X D X1      Xd , proposing a cop-
ula adapted to the ordering constraint [8]. OpenTURNS verifies the compatibility of
the margins Fi with respect to the ordering constraint and then builds the associated
distribution, thanks to the Python command, written in dimension 2:
>>>d= M a x i m u m E n t r o p y O r d e r S t a t i s t i c s D i s t r i b u t i o n ( [ d i s t X 1 , d i s t X 2 ] )

Figure 3 illustrates the copula of such a distribution, built as the ordinal sum of
some maximum entropy order statistics copulas.
The OpenTURNS Python script to model the input random vector of the tutorial
presented previously is as follows:
# Margin d i s t r i b u t i o n s :
>>> d i s t _ Q = Gumbel ( 1 . 8 e 3, 1 0 1 4 )
>>> d i s t _ Q = T r u n c a t e d D i s t r i b u t i o n ( d i s t _ Q , 0 . 0 , T r u n c a t e d D i s t r i b u t i o n .LOWER)
>>> d i s t _ K = Normal ( 3 0 . 0 , 7 . 5 )
>>> d i s t _ K = T r u n c a t e d D i s t r i b u t i o n ( d i s t _ K , 0 . , T r u n c a t e d D i s t r i b u t i o n .LOWER)
>>> d i s t _ Z v = T r i a n g u l a r ( 4 7 . 6 , 5 0 . 5 , 5 2 . 4 )
>>> d i s t _ Z m = T r i a n g u l a r ( 5 2 . 5 , 5 4 . 9 , 5 7 . 7 )
# C opula i n d i m e n s i o n 4 f o r ( Q, K , Zv , Zm )
>>>R= C o r r e l a t i o n M a t r i x ( 2 )
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 9

Fig. 3 An example of maximum entropy copula which almost surely satisfies the ordering
constraint: X1  X2

>>>R [ 0 , 1 ] = 0 . 7
>>> c o p u l a = ComposedCopula ( [ I n d e p e n d e n t C o p u l a ( 2 ) , Nor ma lC opula ( R ) ] )
# F i n a l d i s t r i b u t i o n f o r ( Q, K , Zv , Zm )
>>> d i s t I n p u t = C o m p o s e d D i s t r i b u t i o n ( [ loi_Q , loi_K , l o i _ Z v , loi_Zm ] , c o p u l a )
# F i n a l random v e c t o r ( Q, K , Zv , Zm )
>>> i n p u t V e c t o r =RandomVector ( d i s t I n p u t )

Note that OpenTURNS can truncate any distribution to a lower, an upper bound,
or a given interval. Furthermore, a normal copula models the dependence between
the variables Zv and Zm , with a correlation of 0.7. The variables .Q; K/ are
independent. Both blocks .Q; K/ and .Zv ; Zm / are independent.

2.2 Stochastic Processes

OpenTURNS implements some multivariate random fields X W ˝ D ! Rd where

D 2 Rs is discretized on a mesh. The user can easily build and simulate a random
walk, a white noise as illustrated in Figs. 4 and 5. The Python commands write:
>>>myWN = W h i t e N o i s e ( myDist , myMesh )
>>>myRW = RandomWalk ( myOrigin , myDist , myTimeGrid )

Any field can be exported into the VTK format which allows it to be visualized
using, e.g., ParaView (www.paraview.org).
10 M. Baudin et al.

Fig. 4 A normal bivariate white noise

Fig. 5 A normal bivariate random walk

Multivariate ARMA stochastic processes X W ˝  Œ0; T  ! Rd are implemented

in OpenTURNS which enables some manipulations on times series such as the Box
Cox transformation or the addition/removal of a trend. Note that the parameters of
the Box Cox transformation can be estimated from given fields of the process.
OpenTURNS models normal processes, whose covariance function is a paramet-
ric model (e.g., the multivariate exponential model) as well as defined by the user
as illustrated in Fig. 6. Stationary processes can be defined by its spectral density
function (e.g., the Cauchy model).
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 11

Fig. 6 A user-defined nonstationary covariance function and its estimation from several given
fields

With explicit notations, the following Python commands create a stationary

normal process defined by its covariance function, discretized on a mesh, with an
additional trend:
>>> m yNorm a l Proc e ss = T e m p o r a l N o r m a l P r o c e s s ( myTrend , m yCova ri a nc e Mode l , myMesh )

Note that OpenTURNS enables the mapping of any stochastic processes X into
a process Y through a function f : Y D f .X / where the function f can consist, for
example, of adding or removing a trend, applying a Box Cox transformation in order
to stabilize the variance of X . The Python command is, with explicit notations:
>>> myYprocess = C o m p o s i t e P r o c e s s ( f , myXprocess )

Finally, OpenTURNS implements multivariate processes defined as a linear

combination of K deterministic functions .i /i D1;:::;K W Rd1 7! Rd2 :

X
K
X .!; x/ D Ai .!/i .x/
i D1

where .A1 ; : : : ; AK / is a random vector of dimension K. The Python command

writes:
>>>myX = F u n c t i o n a l B a s i s P r o c e s s ( myRandomCoeff , myBasis , myMesh )
12 M. Baudin et al.

2.3 Statistics Estimation

OpenTURNS enables the user to estimate a model from data, in the univariate as
well as in the multivariate framework, using the maximum likelihood principle or
the moment-based estimation.
Some tests, such as the Kolmogorov-Smirnov test, the chi-square test, and the
Anderson-Darling test (for normal distributions), are implemented and can help to
select a model among others, from a sample of data. The Python command to build
a model and test it writes:
>>> e s t i m a t e d B e t a = B e t a F a c t o r y ( s a m p l e )
>>> t e s t R e s u l t = F i t t i n g T e s t . Kolmogorov ( sam pl e , e s t i m a t e d B e t a )

OpenTURNS also implements the kernel smoothing technique which is a

nonparametric technique to fit a model to data: any distribution can be used as
kernel. In the multivariate case, OpenTURNS uses the product kernel. It also
implements an optimized strategy to select the bandwidth, depending on the number
of data in the sample, which is a mix between the Silverman rule and the plugin one.
Note that OpenTURNS proposes a special treatment when the data are bounded,
thanks to the mirroring technique. The Python command to build the nonparametric
model and to draw its pdf is as simple as the following one:
>>> e s t i m a t e d D i s t = K e r n e l S m o o t h i n g ( ) . b u i l d ( s a m p l e )
>>> p d f G r a p h = e s t i m a t e d D i s t . drawPDF ( )

Figure 7 illustrates the resulting estimated distributions from a sample of size 500
issued from a beta distribution: the kernel smoothing method takes into account the

Fig. 7 Beta distribution estimation from a sample of size 500: parametric estimation versus kernel
smoothing technique
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 13

fact that data are bounded by 0 and 1. The histogram of the data is drawn to enable
comparison.
Several visual tests are also implemented to help select models: among them, the
QQ plot test and the Henry line test which writes (in the case of a beta distribution
for the QQ plot test):
>>> g r a p h Q Q p l o t = V i s u a l T e s t . DrawQQplot ( sam pl e , B e t a ( ) )
>>> g r a p h H e n r y L i n e = V i s u a l T e s t . DrawHenryLine ( s a m p l e )

Figure 8 illustrates the QQ plot test on a sample of size 500 issued from a beta
distribution: the adequation seems satisfying!
Stochastic processes also have estimation procedures from sample of fields or, if
the ergodic hypothesis is verified, from just one field. Multivariate ARMA processes
are estimated according to the BIC and AIC criteria and the Whittle estimator, which
is based on the maximization of the likelihood function in the frequency domain.
The Python command to estimate an ARMA.p; q/ process of dimension d , based
on a sample of time series, writes:
>>>estimatedARMA = ARMALikelihood ( p , q , d ) . b u i l d ( s a m p l e T i m e S e r i e s )

Moreover, OpenTURNS can estimate the covariance function and the spectral
density function of normal processes from given fields. For example, the Python
command to estimate a stationary covariance model from a sample of realizations
of the process writes:
>>>myCovFunc = S t a t i o n a r y C o v a r i a n c e M o d e l F a c t o r y ( ) . b u i l d ( s a m p l e P r o c e s s )

This estimation is illustrated in Fig. 6.

Sample versus Beta: QQPlot test

Test line
Data
1.0
Beta(r = 2, t = 4, a = –1, b = 1)
0.5
0.0
–0.5
–1.0

–1.0 –0.5 0.0 0.5 1.0

sample

Fig. 8 QQ plot test: theoretical model beta versus the sample of size 500
14 M. Baudin et al.

2.4 Conditioned Distributions

OpenTURNS enables the modeling of multivariate distributions by conditioning.

Several types of conditioning are implemented.
At first, OpenTURNS enables the creation of a random vector X whose
distribution DX j whose parameters  form a random vector distributed according
to the distribution D . The Python command writes:
>>>myXrandVect = C o n d i t i o n a l R a n d o m V e c t o r ( d i s t X g i v e n T h e t a , d i s t T h e t a )

Figure 9 illustrates a random variable X distributed according to a normal

distribution, DX j D.M;˙/ D Normal.M; ˙/, whose parameters are defined by
M  U nif orm.Œ0; 1/ and ˙  Expone nti al. D 4/. The probability density
function of X has been fitted with the kernel smoothing technique from n D 106
realizations of X with the normal kernel. It also draws, for comparison needs, the
probability density function of X in the case where the parameters are fixed to their
mean value.
Furthermore, when the random vector  is defined as  D g.Y / where
the random vector follows a known distribution DY and g is a given function,
OpenTURNS creates the distribution of X with the Python command:
>>> f i n a l D i s t = C o n d i t i o n a l D i s t r i b u t i o n ( d i s t X G g i v e n T h e t a , d i s t Y , g )

Figure 10 illustrates the distribution of X that follows a U nif orm.A; B/

distribution, with .A; B/ D g.Y /, g W R ! R2 ; g.Y / D .Y; 1 C Y 2 / and Y
follows a U nif orm.1; 1/ distribution.

Distribution Y
fixed parameters
bayesian random vector
1.5
1.0
PDF
0.5
0.0

−1.0 −0.5 0.0 0.5 1.0 1.5 2.0

Y

Fig. 9 Normal distribution with random or fixed parameters

OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 15

Conditional distribution

1.0
0.8
0.6
PDF
0.4
0.2
0.0

−1 0 1 2
X

Fig. 10 U nif orm.Y; 1 C Y 2 /, with Y  U nif orm.1; 1/

2.5 Bayesian Calibration

Finally, OpenTURNS enables the calibration of a model (which can be a computer

code), thanks to the Bayesian estimation, which is the evaluation of the model’s
parameters. More formally, let’s consider a model G that writes: y D G.x; /
where x 2 Rd1 , y 2 Rd3 and  2 Rd2 is the vector of unknown parameters to
calibrate. The Bayesian calibration consists in estimating , based on a certain set
of n inputs .x 1 ; : : : ; x n / (an experimental design) and some associated observa-
tions .z1 ; : : : ; zn / which are regarded as the realizations of some random vectors
.Z 1 ; : : : ; Z n /, such that, for all i , the distribution of Z i depends on y i D g.x i ; /.
Typically, Z i D Y i C "i where "i is a random measurement error. Once the user
has defined the prior distribution of , OpenTURNS maximizes the likelihood of the
observations and determines the posterior distribution of , given the observations,
using the Metropolis-Hastings algorithm [5, 23].

3 Uncertainty Propagation

Once the input multivariate distribution has been satisfactorily chosen, these
uncertainties can be propagated through the G model to the output vector Y .
Depending on the final goal of the study (min-max approach, central tendency,
or reliability), several methods can be used to estimate the corresponding quantity
16 M. Baudin et al.

of interest, tending to respect the best compromise between the accuracy of the
estimator and the number of calls to the numerical, and potentially costly, model.

3.1 Min-Max Approach

The aim here is to determine the extreme (minimum and maximum) values of the
components of Y for the set of all possible values of X . Several techniques enable
it to be done:

• Techniques based on design of experiments: the extreme values of Y are sought

for only a finite set of combinations .x1 ; : : : ; xn /.
• Techniques using optimization algorithms.

3.1.1 Techniques Based on Design of Experiments

In this case, the min-max approach consists of three steps:

• Choice of experiment design used to determine the combinations .x1 ; : : : ; xn / of

the input random variables
• Evaluation of yi D G.xi / for i D 1; : : : ; N
• Evaluation of min1i N yik and of max1i N yik , together with the combinations
related to these extreme values: xk;min D argmin1i N yik and xk;max D
argmax1i N yik

The type of design of experiments impacts the quality of the metamodel and
then on the evaluation of its extreme values. OpenTURNS gives access to two usual
families of design of experiments for a min-max study:

• Some stratified patterns (axial, composite, factorial, or box patterns). Here are
the two command lines that generate a sample from a two-level factorial pattern:
>>> m y C e n t e r e d R e d u c t e d G r i d = F a c t o r i a l ( 2 , l e v e l s )
>>>mySample = m y C e n t e r e d R e d u c e d G r i d . g e n e r a t e ( )

• Some weighted patterns that include, on the one hand, random patterns (Monte
Carlo, LHS) and, on the other hand, low-discrepancy sequences (Sobol, Faure,
Halton, Reverse Halton, and Haselgrove, in dimension n > 1). The following
lines illustrate the creation of a Faure sequence in dimension 2 or a Monte Carlo
design experiment from a bidimensional normal (0,1) distribution:
# Sobol Sequence Sampling
>>> mySobolSample = F a u r e S e q u e n c e ( 2 ) . g e n e r a t e ( 1 0 0 0 )
# Monte C a r l o S a m p l i n g
>>>myMCSample = M o n t e C a r l o E x p e r i m e n t ( Normal ( 2 ) , 1 0 0 )

Figures 11 and 12, respectively, illustrate a design of experiments issued from a

Faure sequence and a normal distribution in dimension 2.
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 17

Fig. 11 The first 1000 points Sequence of 1000 points

according to a Faure 1.0
sequence of dimension 2

0.8

0.6

x2
0.4

0.2

0.0
0.0 0.2 0.4 0.6 0.8 1.0
x1

Fig. 12 Sample of 1000 points according to a normal (0,1) distribution of dimension 2

3.1.2 Techniques Based on Optimization Algorithm

In this kind of approach, the min or max value of the output variable is sought,
thanks to an optimization algorithm. OpenTURNS offers several optimization
algorithms for the several steps of the global methodology. Here, the Truncated
Newton Constrained (TNC) is often used, which minimizes a function with variables
subject to bounds, using gradient information. More details may be found in [26].
18 M. Baudin et al.

# For t h e r e s e a r c h o f t h e min v a l u e
>>>myAlgoTNC = TNC( T N C S p e c i f i c P a r a m e t e r s ( ) , l i m i t S t a t e F u n c t i o n ,
i n t e r v a l O p t i m , s t a r t i n g P o i n t , TNC . MINIMIZATION)
# For t h e r e s e a r c h o f t h e max v a l u e
>>>myAlgoTNC = TNC( T N C S p e c i f i c P a r a m e t e r s ( ) , l i m i t S t a t e F u n c t i o n ,
i n t e r v a l O p t i m , s t a r t i n g P o i n t , TNC . MAXIMIZATION)
# Run t h e r e s e a r c h and e x t r a c t t h e r e s u l t s
>>>myAlgoTNC . r u n ( )
>>>myAlgoTNCResult = B o u n d C o n s t r a i n e d A l g o r i t h m ( myAlgoTNC ) . g e t R e s u l t ( )
>>> o p t i m a l V a l u e = myAlgoTNCResult . g e t O p t i m a l V a l u e ( )

3.2 Central Tendency

A central tendency evaluation aims at evaluating a reference value for the variable
of interest, here the water level, H , and an indicator of the dispersion of the variable
p To address this problem, mean Y D e.Y / and the standard
around the reference.
deviation Y D V.Y / of Y are here evaluated using two different methods.
First, following the usual method within the measurement science community
[12], Y and Y have been computed under a Taylor first-order approximation of
the function Y D G.X/ (notice that the explicit dependence on the deterministic
variable d is here omitted for simplifying notations):

Y ' G .E.X // (3)

X @G ˇˇ @G ˇˇ
d X
d
Y  ˇ ˇ ij i j ; (4)
i D1 j D1
@X i e.X / @Xj e.X /

i and j being the standard deviation of the ith and jth components Xi and Xj of
the vector X and ij their correlation coefficient. Thanks to the formulas above,
the mean and the standard deviation of H are evaluated as 52.75m and 1.15m,
respectively:
>>>myQuadCum = Q u a d r a t i c C u m u l ( o u t p u t V a r i a b l e )
# F i r s t o r d e r Mean
>>> m e a n F i r s t O r d e r = myQuadCum . g e t M e a n F i r s t O r d e r ( ) [ 0 ]
# Second o r d e r Mean
>>> m eanSecondOrder = myQuadCum . g e t M e a n S e c o n d O r d e r ( ) [ 0 ]
# F i r s t order Variance
>>> v a r F i r s t O r d e r = myQuadCum . g e t C o v a r i a n c e ( ) [ 0 , 0 ]

Then, the same quantities have been evaluated by a Monte Carlo evaluation: a set
of 10000 samples of the vector X is generated and the function G.X/ is evaluated,
thus giving a sample of H . The empirical mean and standard deviation of this
sample are 52.75 and 1.42 m, respectively. Figure 13 shows the empirical histogram
of the generated sample of H .
# C r e a t e a random s a m p l e o f t h e o u t p u t v a r i a b l e o f i n t e r e s t o f s i z e 10000
>>> o u t p u t S a m p l e = o u t p u t V a r i a b l e . g e t N u m e r i c a l S a m p l e ( 1 0 0 0 0 )
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 19

sample histogram

0.30
Unnamed histogram
0.25
0.20
frequency
0.15
0.10
0.05
0.00

48 50 52 54 56 58 60 62
realizations

Fig. 13 Empirical histogram of 10000 samples of H

# Ge t t h e e m p i r i c a l mean
>>> e m p i r i c a l M e a n = o u t p u t S a m p l e . computeMean ( )
# Ge t t h e e m p i r i c a l c o v a r i a n c e m a t r i x
>>> e m p i r i c a l C o v a r i a n c e M a t r i x = o u t p u t S a m p l e . c o m p u t e C o v a r i a n c e ( )

3.3 Failure Probability Estimation

This section focuses on the estimation of the probability for the output Y to exceed a
certain threshold s, noted Pf in the following. If s is the altitude of a flood protection
dike, then the above excess probability, Pf , can be interpreted as the probability of
an overflow of the dike, i.e., a failure probability.
Note that an equivalent way of formulating this reliability problem would be to
estimate the .1  p/-th quantile of the output’s distribution. This quantile can be
interpreted as the flood height qp which is attained with probability p each year.
T D 1=p is then seen to be a return period, i.e., a flood as high as q1=T occurs on
average every T years.
Hence, the probability of overflowing a dike with height s is less than p (where p,
for instance, could be set according to safety regulations) if and only if s  qp , i.e.,
if the dike’s altitude is higher than the flood with return period equal to T D 1=p:
20 M. Baudin et al.

Fig. 14 FORM importance FORM Importance Factors − Event Zc > 58.0

factors
Q : 32.4%

Zm : 1.2%

Zv : 9.5%
Ks : 56.8%

3.3.1 FORM
A way to evaluate such failure probabilities is through the so-called first-order
reliability method (FORM) [9]. This approach allows, by using an equiprobabilistic
transformation and an approximation of the limit state function, the evaluation with
a much reduced number of model evaluations, of some low probability as required
in the reliability field. Note that OpenTURNS implements the Nataf transformation
where the input vector X has a normal copula, the generalized Nataf transformation
when X has an elliptical copula, and the Rosenblatt transformation for any other
cases [16–19].
The probability that the yearly maximal water height H exceeds s = 58 m is
evaluated using FORM. The Hasofer-Lind reliability index was found to be equal to
ˇHL D 3:04; yielding a final estimate of:

PO f;FORM D 1:19  103 :

The method gives also some importance factors that measure the weight of each
input variable in the probability of exceedance, as shown on Fig. 14.
>>>myFORM = FORM( Cobyl a ( ) , myEvent , m e a n I n p u t V e c t o r )
>>>myFORM. r u n ( )
>>> F o r m R e s u l t = myFORM . g e t R e s u l t ( )
>>>pFORM = F o r m R e s u l t . g e t E v e n t P r o b a b i l i t y ( )
>>> H a s o f e r I n d e x = F o r m R e s u l t . g e t H a s o f e r R e l i a b i l i t y I n d e x ( )
# Importance f a c t o r s
>>> i m p o r t a n c e F a c t o r s G r a p h = F o r m R e s u l t . d r a w I m p o r t a n c e F a c t o r s ( )

3.3.2 Monte Carlo

Whereas the FORM approximation relies on strong assumptions, the Monte Carlo
method is always valid, independently from the regularity of the model. It is
nevertheless much more computationally intensive, covering all the input domain
to evaluate the probability of exceeding a threshold. It consists in sampling many
input values .X .i / /1i N from the input vector joint distribution, then computing
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 21

MonteCarlo convergence graph at level 0.95

probability estimate
0.003
0.002 bounds
estimate
0.001
0.000
−0.001

0 10000 20000 30000 40000 50000 60000

outer iteration

Fig. 15 Monte Carlo convergence graph

the corresponding output values Y .i / D g.X .i / /: The excess probability Pf is then

estimated by the proportion of sampled values Y .i / that exceed t W

1 X
N
PO f;M C D 1 .i / : (5)
N i D1 fY >sg
p
The sample average of the estimation error PO f;M C  Pf decreases as 1= N and
can be precisely quantified by a confidence interval derived from the central limit
theorem. In the present case, the results are:

PO f;M C D 1:50  103 ;

with the following 95 % confidence interval:

 
IPf ;M C D 1:20  103 ; 1:79  103 :

These results are coherent with those of the FORM approximation, confirming that
the assumptions underlying the latter are correct. Figure 15 shows the convergence
of the estimate depending on the size of the sample, obtained with OpenTURNS.
>>>myEvent = E v e n t ( o u t p u t V a r i a b l e , G r e a t e r ( ) , t h r e s h o l d )
>>> myMonteCarlo = M ont eCarl o ( myEvent )
# S p e c i f y t h e maximum number o f s i m u l a t i o n s
22 M. Baudin et al.

>>> num berM ax Si m u l a t i o n = 100000

>>> myMonteCarlo . set M axi m um Out erSam pl i ng ( num berM axS i m ul a t i o n )
# Perform t he al gori t hm
>>> myMonteCarlo . r u n ( )
# Get t h e c o n v e r g e n c e graph
>>> c o n v e r g e n c e G r a p h = myMonteCarlo . d r a w P r o b a b i l i t y C o n v e r g e n c e ( )
>>> c o n v e r g e n c e G r a p h . draw ( " M o n t e C a r l o C o v e r g e n c e G r a p h " )

3.3.3 Importance Sampling

An even more precise estimate can be obtained through importance sampling [31],
using the Gaussian distribution with identity covariance matrix and mean equal
to the design point u as the proposal distribution. Many values .U .i / /1i N are
sampled from this proposal. Because n .u  u / is the proposal density from which
the U .i / have been sampled, the failure probability can be estimated without bias
by:

1 X
N
O n .U .i / /
P f;IS D 1fGıT 1 U .i / >sg (6)
N i D1 n .U .i /  u /

The rationale of this approach is that, by sampling in the vicinity of the failure
domain boundary, a larger proportion of values fall within the failure domain than by
sampling around the origin, leading to a better evaluation of the failure probability
and a reduction in the estimation variance. Using this approach, the results are:

PO f;IS D 1:40  103

As in the simple Monte Carlo approach, a 95 %-level confidence interval can be

derived from the output of the importance sampling algorithm. In the present case,
this is equal to:
 
IPf ;IS D 1:26  103 ; 1:53  103 ;

and indeed provides tighter confidence bounds for Pf .

# S p e c i f y t h e s t a r t i n g p o i n t fro m FORM a l g o r i t h m
>>> s t a n d a r d P o i n t = F o r m R e s u l t . g e t S t a n d a r d S p a c e D e s i g n P o i n t ( )
# Define the importance d i s t r i b u t i o n
>>> sig m a = [ 1 . 0 , 1 . 0 , 1 . 0 , 1 . 0 ]
>>> i m p o r t a n c e D i s t r i b = Normal ( s t a n d a r d P o i n t , sigma , C o r r e l a t i o n M a t r i x ( 4 ) )
# Define the IS algorithm : event , d i s t r i b u t i o n , c r i t e r i a of convergence
>>> m y A l g o I m p o r t a n c e S a m p l i n g = I m p o r t a n c e S a m p l i n g
( myStandardEvent , i m p o r t a n c e D i s t r i b )
>>> m y A l g o I m p o r t a n c e S a m p l i n g . s etM ax im u m Ou terSam p lin g ( m ax im u m Ou terSam p lin g _ IS )
>>> m y A l g o I m p o r t a n c e S a m p l i n g . s e t M a x i m u m C o e f f i c i e n t O f V a r i a t i o n ( 0 . 0 5 )
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 23

3.3.4 Directional Sampling

The directional simulation method is an accelerated sampling method that involves
as a first step a preliminary iso-probabilistic transformation as in the FORM method.
The basic idea is to explore the space by sampling in several directions in the
standard space. The final estimate of the probability Pf after N simulations is the
following:

X N
b f;DS D 1
P qi
N i D1

where qi is the probability obtained in each explored direction. A central limit

theorem allows to access to some confidence interval on this estimate. More details
on this specific method can be found in [32].
In practice in OpenTURNS , the directional sampling simulation requires the
choice of several parameters in the methodology: a sampling strategy to choose the
explored directions, a “root strategy” corresponding to the way to seek the limit state
function (i.e., a sign change) along the explored direction, and a nonlinear solver to
estimate the root. A default setting of these parameters allows the user to test the
method in one command line:
>>>myAlgo = D i r e c t i o n a l S a m p l i n g ( myEvent )

3.3.5 Subset Sampling

The subset sampling is a method for estimating rare event probability, based on the
idea of replacing rare failure event by a sequence of more frequent events Fi :

F1 F2  Fm D F

The original probability is obtained conditionally to the more frequent events:

!
\
m Y
m
Pf D P .Fm / D P Fi D P .F1 / P .Fi jFi 1 /
i D1 i D2

In practice, the subset simulation shows a substantial improvement (NT  log Pf )

compared to crude Monte Carlo (NT  P1f ) sampling when estimating rare events.
More details on this specific method can be found in [2].
OpenTURNS provides this method through a dedicated module. Here also, some
parameters of the methods have to be chosen by the user: a few command lines
allows the algorithm to be set up before its launch.
>>>mySSAlgo= S u b s e t S a m p l i n g ( myEvent )
# Change t h e t a r g e t c o n d i t i o n a l p r o b a b i l i t y o f each s u b s e t domain
>>>mySSAlgo . s e t T a r g e t P r o b a b i l i t y ( 0 . 9 )
# S e t t h e w i d t h o f t h e MCMC random wal k u n i f o r m d i s t r i b u t i o n
>>>mySSAlgo . s e t P r o p o s a l R a n g e ( 1 . 8 )
24 M. Baudin et al.

# T h i s a l l o w s t o c o n t r o l t h e number o f s a m p l e s p e r s t e p
>>>mySSAlgo . set M axi m um Out erSam pl i ng ( 1 0 0 0 0 )
# Run t h e a l g o r i t h m
>>>mySSAlgo . r u n ( )

4 Sensitivity Analysis

The sensitivity analysis aims to investigate how a given computational model

answers to variations in its inputs. Such knowledge is useful for determining the
degree of resemblance of a model and a real system, distinguishing the factors that
mostly influence the output variability and those that are insignificant, revealing
interactions among input parameters and correlations among output variables, etc.
A detailed description of sensitivity analysis methods can be found in [36] and in
the Sensitivity analysis chapter of the Springer Handbook. In the global sensitivity
analysis strategy, the emphasis is put on apportioning the output uncertainty to the
uncertainty in the input factors, given by their uncertainty ranges and probability
distributions.

4.1 Graphical Tools

In sensitivity analysis, graphical techniques have to be used first. With all the
scatterplots between each input variable and the model output, one can immediately
detect some trends in their functional relation. The following instructions allow
scatterplots of Fig. 16 to be obtained from a Monte Carlo sample of size N D 1000
of the flooding model.
>>> i n p u t S a m p l e = i n p u t R a n d o m V e c t o r . g e t N u m e r i c a l S a m p l e ( 1 0 0 0 )
>>> i n p u t S a m p l e . s e t D e s c r i p t i o n ( [ ’Q ’ , ’K ’ , ’ Zv ’ , ’Zm ’ ] )
>>> o u t p u t S a m p l e = f i n a l M o d e l C r u e ( i n p u t S a m p l e )
>>> o u t p u t S a m p l e . s e t D e s c r i p t i o n ( [ ’H ’ ] )
# Here , s t a c k b o t h s a m p l e s i n one
>>> i n p u t S a m p l e . s t a c k ( o u t p u t S a m p l e )
>>> m y P a i r s = P a i r s ( i n p u t S a m p l e )
>>>myGraph = Graph ( )
>>>myGraph . add ( m y P a i r s )

In the right column of Fig. 16, it is clear that the strong and rather linear effects
of Q and Zv on the output variable H . In the plot of the third line and fourth
column, it is also clear that the dependence between Zv and Zm comes from the
large correlation coefficient introduced in the probabilistic model.
However scatterplots do not capture some interaction effects between the inputs.
Cobweb plots are then used to visualize the simulations as a set of trajectories. The
following instructions allow the cobweb plots of Fig. 17 to be obtained where the
simulations leading to the largest values of the model output H have been colored
in red.
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 25

10 20 30 40 50 53 54 55 56 57

2000 4000
Q

0
50
30

K
10

52
50
Zv

48
57
55

Zm
53

50 52 54 56
H

0 1000 3000 48 49 50 51 52 50 52 54 56

Fig. 16 Scatterplots between the inputs and the output of the flooding model: each combination
(input i, input j) and (input i, output) is drawn, which enables to exhibit some correlation patterns

>>> i n p u t S a m p l e = i n p u t R a n d o m V e c t o r . g e t N u m e r i c a l S a m p l e ( 1 0 0 0 )
>>> o u t p u t S a m p l e = f i n a l M o d e l C r u e ( i n p u t S a m p l e )
# Graph 1 : v a l u e b a s e d s c a l e t o d e s c r i b e t h e Y r a n g e
>>> m i nVal ue = o u t p u t S a m p l e . c o m p u t e Q u a n t i l e P e r C o m p o n e n t ( 0 . 0 5 ) [ 0 ]
>>>maxValue = o u t p u t S a m p l e . c o m p u t e Q u a n t i l e P e r C o m p o n e n t ( 0 . 9 5 ) [ 0 ]
>>>myCobweb = V i s u a l T e s t . DrawCobWeb ( i n p u t S a m p l e , o u t p u t S a m p l e ,
minValue , maxValue , ’ r e d ’ , F a l s e )

The cobweb plot allows us to immediately understand that these simulations

correspond to large values of the flow rate Q and small values of the Strickler
coefficient Ks .

4.2 Sampling-Based Methods

In order to obtain quantitative sensitivity indices rather than qualitative information,

one may use some sampling-based methods which often suppose that the input
variables are independent. The section illustrates some of these methods on the
flooding model with independence between its input variables.
If the behavior of the output Y compared to the input vector X is overall
linear, it is possible to obtain quantitative measurements of the inputs influences
from the regression coefficients ˛i of the linear regression connecting Y to the
26 M. Baudin et al.

Q
K
Zv
Zm
H

Fig. 17 Cobweb plot for the flooding model: each simulation is drawn. The input marginal values
are linked to the output value (last column). All the simulations that led to a high quantile of the
output are drawn in red: the cobweb plot enables to detect all the combinations of the inputs they
come from

X D .X1 ; : : : ; Xp /. The standard regression coefficient (SRC), defined by:

i
SRCi D ˛i (for i D 1 : : : p), (7)
Y

with i (resp. Y ), the standard deviation of Xi (resp. Y ), measures the variation of

the response for a given variation of the parameter Xi . In practice, the coefficient R2
(the variance percentage of the output variable Y explained by the regression model)
also helps to check the linearity: if R2 is close to one, the relation connecting Y to
all the parameters Xi is almost linear and the SRC sensitivity indices make sense.
The following instructions allow the results of Table 2 to be obtained from a
Monte Carlo sample of size N D 1000.
>>> i n p u t S a m p l e = i n p u t R a n d o m V e c t o r . g e t N u m e r i c a l S a m p l e ( 1 0 0 0 )
>>> o u t p u t S a m p l e = f i n a l M o d e l C r u e ( i n p u t S a m p l e )
>>> S R C C o e f f i c i e n t = C o r r e l a t i o n A n a l y s i s . SRC
( inputSample , outputSample )
>>> l i n R e g M o d e l = L i n e a r M o d e l F a c t o r y ( ) . b u i l d
( inputSample , outputSample , 0 . 9 0 )
>>> R s q u a r e d = L i n e a r M o d e l T e s t . L i n e a r M o d e l R S q u a r e d ( i n p u t S a m p l e ,
o u t p u t S a m p l e , l i nRegM odel , 0 . 9 0 )
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 27

Table 2 Regression coefficients and SRC of the flood model inputs (˛0 D 0:1675 and R2 D
0:97)
Q Ks Zv Zm
˛i 3:2640 0:0012 0:0556 1:1720
SRCi 0:3462 0:0851 0:6814 0:0149

The SRC values confirm our first conclusions drawn from the scatterplots visual
analysis. As R2 D 0:97 is very close to one, the model is quasi-linear. The SRC
coefficients are sufficient to perform a global sensitivity analysis.
Several other estimation methods are available in OpenTURNS for a sensitivity
analysis purpose:

• Derivatives and Pearson correlation coefficients (linearity hypothesis between

output and inputs).
• Spearman correlation coefficients and standard rank regression coefficients
(monotonicity hypothesis between output and inputs).
• Reliability importance factors with the FORM/SORM importance measures
presented previously (Sect. 3).
• Variance-based sensitivity indices (no hypothesis on the model). These last
indices, often known as Sobol indices and defined by:

Si D VarVar
ŒE.Y jXi /
.Y /
(first order index) and (8)
Pp P
STi D i D1 Si C i <j Sij C : : : (total index),

are estimated in OpenTURNS with the classic pick-freeze method based on two
independent Monte Carlo samples [34]. In OpenTURNS, other ways to compute
the Sobol indices are the extended FAST method [35] and the coefficients of the
polynomial chaos expansion [38].

5 Metamodels

When each model evaluation is time consuming, it is usual to build a surrogate

model which is a good approximation of the initial model and which can be
evaluated at negligible cost. OpenTURNS proposes some usual polynomial approx-
imations: the linear or quadratic Taylor approximations of the model at a particular
point, or a linear approximation based on the least squares method. Two recent
techniques implemented in OpenTURNS are detailed here: the polynomial chaos
expansion and the Kriging approximation.
28 M. Baudin et al.

5.1 Polynomial Chaos Expansion

The polynomial chaos expansion enables the approximation of the output random
variable of interest Y D G.X / with g W Rd ! Rp by the surrogate model:
X
YQ D ˛k k ı T .X /
k2K

where ˛k 2 Rp , T is an iso-probabilistic transformation (e.g., the Rosenblatt

Qdmaps the multivariate distribution of X into the multivariate
transformation) which
distribution  D i D1 i , and . k /k2N is a multivariate polynomial basis of
L2 .Rd ; Rp / which is orthonormal according to the distribution . K is a finite
subset of N. Y is supposed to be of finite second moment.
OpenTURNS proposes the building of the multivariate orthonormal basis
. k .x//k2N as the Cartesian product of orthonormal univariate polynomial family
. li .zi //l2N :

k .z/ D 1
k1 .z1 /
2
k2 .z2 /  d
kd .zd /

The possible univariate polynomial families associated to continuous measures

are:

• Hermite, which is orthonormal with respect to the normal.0; 1/ distribution

• Jacobi(˛, ˇ, param), which is orthonormal with respect to the Beta.ˇ C 1; ˛ C
ˇ C2; 1; 1/ distribution if param D 0 (default value) or to the Beta.˛; ˇ; 1; 1/
distribution if param D 1
• Laguerre(k), which is orthonormal with respect to the Gamma.k C 1; 1; 0/
distribution
• Legendre, which is orthonormal with respect to the Uniform.1; 1/ distribution

OpenTURNS proposes three strategies to truncate the multivariate orthonormal

basis to the finite set K: these strategies select different terms from the multivariate
basis, based on a convergence criterion of the surrogate model and the cleaning of
the less significant coefficients.
The coefficients of the polynomial decomposition writes:
2 !2 3
X
˛ D argmin˛2RK E 4 g ı T 1 .Z /  ˛k k .Z / 5 (9)
k2K

as well as:
  
˛ D E g ı T 1 .Z / k .Z / k
(10)

where Z D T .X / is distributed according to .

OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 29

It corresponds to two points of view implemented by OpenTURNS: the relation

(9) means that the coefficients .˛k /k2K minimize the mean quadratic error between
the model and the polynomial approximation; the relation (10) means that ˛k is
the scalar product of the model with the k  th element of the orthonormal basis
. k /k2K . In both cases, the expectation E is approximated by a linear quadrature
formula that writes, in the general case:
X
E Œf .Z / ' !i f . i/ (11)
i 2I

where f is a function L1 ./. The set I , the points . i /i 2I , and the weights
.!i /i 2I are evaluated from weighted designs of experiments which can be random
(Monte Carlo experiments and importance sampling experiments) or determinis-
tic (low-discrepancy experiments, user-given experiments, and Gaussian product
experiments).
At last, OpenTURNS gives access to:

• The composed model h W Z 7! X Y D G ı T 1 .Z /, which is the model of the

reduced variables Z . Then, h D ˛k k .
k2N
• The coefficients of the polynomial approximation: .˛k /k2K .
• The composed metamodel: h, O which is the model of the reduced variables
X
reduced to the truncated multivariate basis . k /k2K . Then, hO D ˛k k .
k2K
• The metamodel: gO W X 7! Y D hO ı T .X / which isX the polynomial chaos
approximation as a NumericalMathFunction. Then, gO D ˛k k ı T .
k2K

When the model is very expensive to evaluate, it is necessary to optimize the

number of coefficients of the polynomial chaos expansion to be calculated. Some
specific strategies have been proposed by [6] for enumerating the infinite polynomial
chaos series: OpenTURNS implements the hyperbolic enumeration strategy which
is inspired by the so-called sparsity-of-effects principle. This strategy states that
most models are principally governed by main effects and low-order interactions.
This enumeration strategy selects the multi-indices related to main effects.
The following lines illustrate the case where the model G W x 7! x sin x and
where the input random vector follows a uniform distribution on Œ1; 1:
# D e f i n e t h e model
>>> model = N u m e r i c a l M a t h F u n c t i o n ( [ ’ x ’ ] , [ ’ x * s i n ( y ) ’ ] )
# Create the i nput d i s t r i b u t i o n
>>> d i s t r i b u t i o n = Uni form ( )
# Const ruct i on of the orthonormal basi s
>>> p o l y C o l l = [ 0 . ]
>>> p o l y C o l l [ 0 ] = S t a n d a r d D i s t r i b u t i o n P o l y n o m i a l F a c t o r y
( distribution )
>>> e n u m e r a t e F u n c t i o n = L i n e a r E n u m e r a t e F u n c t i o n ( 1 )
30 M. Baudin et al.

>>> p r o d u c t B a s i s = O r t h o g o n a l P r o d u c t P o l y n o m i a l F a c t o r y
( polyColl , enumerateFunction )
# Truncature s t r a t e g y of the m u l t i v a r i a t e orthonormal basi s
# Choose a l l t h e p o l y n o m i a l s o f d e g r e e <= 4
>>> d e g r e e = 4
>>> indexMax = e n u m e r a t e F u n c t i o n . g e t S t r a t a C u m u l a t e d C a r d i n a l ( d e g r e e )
# Keep a l l t h e p o l y n o m i a l s o f d e g r e e <= 4
# whi ch c o r r e s p o n d s t o t h e 5 f i r s t o n e s
>>> a d a p t i v e S t r a t e g y = F i x e d S t r a t e g y ( p r o d u c t B a s i s , indexMax )
# Evaluation stra te gy of the approximation c o e f f i c i e n t s
>>> s a m p l i n g S i z e = 50
>>> e x p e r i m e n t = M o n t e C a r l o E x p e r i m e n t ( s a m p l i n g S i z e )
>>> p r o j e c t i o n S t r a t e g y = L e a s t S q u a r e s S t r a t e g y ( e x p e r i m e n t )
# C r e a t i o n o f t h e F u n c t i o n a l Chaos A l g o r i t h m
>>> a l g o = F u n c t i o n a l C h a o s A l g o r i t h m ( model , d i s t r i b u t i o n ,
adaptiveStrategy , . . . projectionStrategy )
>>> a l g o . r u n ( )
# Get t h e r e s u l t
>>> f u n c t i o n a l C h a o s R e s u l t = a l g o . g e t R e s u l t ( )
>>> m et am odel = f u n c t i o n a l C h a o s R e s u l t . get M et aM odel ( )

Figure 18 illustrates the result.

Fig. 18 An example of a polynomial chaos expansion: the blue line is the reference function
G W x 7! x sin x and the red one its approximation only valid on Œ1; 1
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 31

5.2 The Kriging Approximation

Kriging (also known as Gaussian process regression) [24, 30, 33, 37] is a Bayesian
technique that aims at approximating functions (most often in order to surrogate
them because they are expensive to evaluate). In the following, it is assumed that
the aim is to surrogate a scalar-valued model G W x 7! y. Note the OpenTURNS
implementation of Kriging can deal with vector-valued functions (G W x 7! y),
with simple loops over each output. It is also assumed that the model was run over
a design of experiments
 in order to produce  a set of observations gathered in the
following dataset: x i ; y i ; i D 1; : : : ; n . Ultimately Kriging aims at producing a
predictor (also known as a response surface or metamodel) denoted as G. Q
It is assumed that the model G is a realization of the normal process Y W ˝ 
Rd ! R defined by:

Y .!; x/ D m.x/ C Z.!; x/ (12)

where m.x/ is the trend and Z.x/ is a zero-mean Gaussian process with a
covariance function c W Rd  Rd ! R which depends on the vector of parameters
 2 Rn :

EŒZ.x/; Z.y/ D c .x; y/ (13)

The trend is generally taken equal to the generalized linear model:

m.x/ D .f .x//t ˇ (14)

   
where .f .x//t D f1 ; : : : ; fp and ˇ D ˇ1 ; : : : ; ˇp . Then, the Kriging method
approximates the model f by the mean of the Y given that:

Y .!; x .i / / D y .i / 8i D 1; : : : ; n (15)

The Kriging metamodel GQ of G writes:

Q
G.x/ D EŒY .!; x/jY .!; x .i / / D y .i / ; 8i D 1; : : : ; n (16)

The metamodel is then defined by:

Q
G.x/ D .f .x//t Q̌ C .c  .x//t C1 .y  F Q̌ / (17)

where Q̌ is the least squares estimator for ˇ defined by:

 
Q̌ D F t C 1 F 1 F t C 1 y (18)
 
32 M. Baudin et al.

10
l

5
l

l
l
0
model, kriging

l
−5

l
−10
−15

model
−20

kriging
data

0 2 4 6 8
x

Fig. 19 An example of Kriging approximation based on six observations: the blue line is the
reference function G W x 7! x sin x and the red one its approximation by a realization of a Gaussian
process

and C D Œc .xi ; xj /i;j D1:::n , F D Œf .xi /t i D1:::n and c t .x/ D Œc .x; xi /i D1:::n .
The line command writes:
>>> a l g o = K r i g i n g A l g o r i t h m ( i n p u t S a m p l e , o u t p u t S a m p l e , b a s i s ,
covarianceModel )
>>> a l g o . r u n ( )
>>> r e s u l t = a l g o . g e t R e s u l t ( )
>>> m et am odel = r e s u l t . get M et aM odel ( )
>>> g r a p h = m et am odel . draw ( )

Figure 19 approximates the previously defined model G W x 7! x sin x with a

realization of a Gaussian process based on six observations.

6 The External Simulator

6.1 Fast Evaluations of G

On the practical side, the OpenTURNS software provides features which make the
connection to the simulator G easy and make its evaluation generally fast. Within
the OpenTURNS framework, the method to connect to G is called “wrapping.”
In the simplest situations, the function G is analytical and the formulas can be
provided to OpenTURNS with a character string. Here, the Muparser C++ library
[4] is used to evaluate the value of the mathematical function. In this case, the
evaluation of G by OpenTURNS is quite fast.
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 33

In the following Python script, consider the function G W R3 ! R2 , where

G1 .x/ D x1 C x2 C x3 and G2 .x/ D x1  x2 x3 , for any real numbers x1 , x2 ,
and x3 . The input argument of the NumericalMathFunction class is a Python
tuple, where the first item describes the three input variables, the second item
describes the two output variables, and the last item describes the two functions G1
and G2 .
>>>G = N u m e r i c a l M a t h F u n c t i o n (
( " x0 " , " x1 " , " x2 " ) ,
( " y0 " , " y1 " ) ,
( " x0+x1+x2 " , " x0x1 * x2 " ) )

Once created, the function G can be used as a regular Python function or can be
passed as an input argument of other OpenTURNS classes.
In most cases, the function G is provided as a Python function, which can be
connected to OpenTURNS with the PythonFunction class. This task is easy
(for those who are familiar with this language) and allows the scientific packages
already available in Python to be combined. For example, if the computational code
uses XML files on input or output, it is easy to make use of the XML features of
Python (e.g., the minidom package). Moreover, if the function evaluation can be
vectorized (e.g., with the numpy package), then the func_sample option of the
PythonFunction class can improve the performance a lot.
The following Python script creates the function G associated with the flooding
model. The flood function is first defined with the def Python statement.
This function takes the variable X as input argument, which is an array with
four components, Q, K_s, Z_v, and Z_m, which correspond to the input random
variables in the model. The body of the flood function is a regular Python script,
so that all Python functions can be used at this point (e.g., the numpy or scipy
functions). The last statement of the function returns the overflow S. Then the
PythonFunction class is used in order to convert this Python function into an
object that OpenTURNS can use. This class takes as input arguments the number
of input variables (in this case, 4), the number of outputs (in this case, 1), and the
function and returns the object G.
>>>from o p e n t u r n s i m port P y t h o n F u n c t i o n
>>> d e f f l o o d (X) :
L = 5 . 0 e3 ; B = 300.0
Q, K_s , Z_v , Z_m = X
a l p h a = ( Z_m  Z_v ) / L
H = (Q / ( K_s *B* s q r t ( a l p h a ) ) ) * * ( 0 . 6 )
return H
>>>G = P y t h o n F u n c t i o n ( 4 , 1 , f l o o d )

If, as many of the computational codes commonly used, the data exchange
is based on text files, OpenTURNS provides a component (coupling_tools)
which is able to read and write structured text files based, for example, on line
indices and perhaps containing tables (using line and column indices). Moreover,
OpenTURNS provides a component which can evaluate such a Python function
using the multi-thread capabilities that most computers have.
34 M. Baudin et al.

Fig. 20 Performance of various connection methods in OpenTURNS

Finally, when the computational code G is provided as a C or Fortran library,

OpenTURNS provides a generic component to exchange data by memory, which is
much faster than with files. In this case, the evaluation of G is automatically multi-
thread. This component can be configured by Python, based on a XML file. If this
method is not flexible enough, then the connection can be done with the C++ library.
The previous techniques are documented in the OpenTURNS developer’s guide
[1].
Figure 20 compares the performance of three methods to connect to the
function G: the PythonFunction class, the PythonFunction class with
the func_sample option, and the analytical function. This test was performed
with a basic MS Windows laptop computer. Obviously, the fastest method is the
analytical function, which can provide as many as 0.2 million samples per second, a
performance which is four times the performance of the PythonFunction class.

6.2 Evaluation of the Derivatives

When the algorithm involves an optimization step (e.g., in the FORM-SORM

method) or a local approximation of G (e.g., in the Taylor development used to
approximate the expectation and variance), the derivatives of G are required.
When the computer code can compute the gradient and Hessian matrix of G, this
information can be used by OpenTURNS . This happens sometimes, for example,
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 35

when the computer code has been differentiated with automatic differentiation
methods, such as forward or adjoint techniques.
In the case where the function is analytical and is provided as a character string,
OpenTURNS is able to compute the exact derivatives of G. In order to do this, the
software uses the Ev3 C++ library [22] to perform the symbolic computation of the
derivatives and MuParser [4] to evaluate it.
In most common situations, however, the code does not compute its derivatives.
In this case, OpenTURNS provides a method to compute the derivatives based on
finite difference formulas. By default, a centered finite difference formula for the
gradient and a centered formula for the Hessian matrix are used.

6.3 High-Performance Computing

For most common engineering practices, OpenTURNS can evaluate G with the
multi-thread capabilities of most laptop and scientific workstations. However, when
the evaluation of G is more CPU consuming or when the number of evaluations
required is larger, these features are not sufficient by themselves, and it is necessary
to use a high-performance computer such as the Zumbrota, Athos, or Ivanhoe super-
computers available at EDF R&D which have from 16,000 to 65,000 cores [40].
In this case, two solutions are commonly used. The first one is to use a feature
which can execute a Python function on remote processors, connected on the
network with ssh. Here, the data flow is based on files, located in automatically
generated directories, which prevents the loss of intermediate data. This feature
(the DistributedPythonFunction) allows each remote processor to use its
multi-thread capabilities, providing two different levels of parallel computing.
The second solution is to use the OpenTURNS component integrated in the
Salome platform. This component, along with a graphical user interface, called
“Eficas,” makes use of a software, called “YACS,” which can call a Python script.
The YACS module allows calculation schemes in Salome to be built, edited, and
executed. It provides both a graphical user interface to chain the computations
by linking the inputs and outputs of computer codes and then to execute these
computations on remote machines.
Several studies have been conducted at EDF based on the OpenTURNS com-
ponent of Salome. For example, an uncertainty propagation study (the thermal
evaluation of the storage of high-level nuclear waste) was making use of a computer
code where one single run required approximately 10 min on the 8 cores of a
workstation (with shared memory). Within Salome, the OpenTURNS simulation
involving 6000 unitary evaluations of the function G required 8000 CPU hours on
32 nodes [3].
36 M. Baudin et al.

7 Conclusions

This educational example has shown a number of questions and problems that can be
addressed by UQ methods, uncertainty quantification, central tendency evaluation,
excess probability assessment, and sensitivity analysis, that can require the use of a
metamodel.
Different numerical methods have been used for solving these three classes
of problems, leading substantially to the same (or very similar) results. In the
industrial practice of UQ, the main issue (which actually motivates the choice of
one mathematical method instead of another) is the computational budget, which
is actually given by the number of allowed runs of the deterministic model G.
When the computer code implementing G is computationally expensive, one needs
specifically designed mathematical and software tools.
OpenTURNS is specially intended to meet these issues: (i) it includes a set of
efficient mathematical methods for UQ and (ii) it can be easily connected to any
external black box model G. Thanks to these two main features, OpenTURNS is a
software that can address many different physics problems and thus help to solve
industrial problems. From this perspective, the partnership around OpenTURNS
focuses efforts on the integration of the most efficient and innovative methods
required by the industrial applications that takes into account both the need of
genericity and of ease to communicate. The main projects for 2015 concern the
improvement of the Kriging implementation to integrate some very smart methods
of optimization. Around this theme, some other classical optimization methods will
also be generalized or newly implemented.
A growing need for model exploration and analysis of uncertainty problem in
industrial applications is to better visualize the information provided by such a
volume of data. In this area, specific visualization software, such as ParaView,
can provide very efficient and interactive features. Taking the advantage of the
integration of OpenTURNS in the Salome platform, EDF is working on a better
link between the ParaView module in Salome (called ParaVIS) and the uncertainty
analysis with OpenTURNS: in 2012, functional boxplot [13] has been implemented.
Some recent work around in situ visualization for uncertainty analysis should also
be developed and implemented and so benefit very computationaly expensive model
physics that generate an extremely high volume of data.
Part of this work has been backed by French National Research Agency (ANR)
trough the Chorus project (no. ANR-13-MONU-0005-08). We are grateful to the
OpenTURNS Consortium members. We also thank Regis Lebrun, Mathieu Couplet,
and Merlin Keller for their help.

References
1. Airbus, EDF, Phimeca: Developer’s guide, OpenTURNS 1.4 (2014). http://openturns.org
2. Au, S., Beck, J.L.: Estimation of small failure probabilities in high dimensions by subset
simulation. Probab. Eng. Mech. 16, 263–277 (2001)
OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation 37

3. Barate, R.: Calcul haute performance avec OpenTURNS, workshop du GdR MASCOT-NUM,
Quantification d’incertitude et calcul intensif. http://www.gdr-mascotnum.fr/media/openturns-
hpc-2013-03-28.pdf (2013)
4. Berg, I.: muparser, http://muparser.beltoforion.de, fast Math Parser Library (2014)
5. Berger, J. (ed.): Statistical Decision Theory and Bayesian Analysis. Springer, New York (1985)
6. Blatman, G.: Adaptative sparse polynomial chaos expansions for uncertainty propagation and
sensitivity anaysis. PhD thesis, Clermont University (2009)
7. Blatman, G., Sudret, B.: Adaptive sparse polynomial chaos expansion based on least angle
regression. J. Comput. Phys. 230, 2345–2367 (2011)
8. Butucea, C., Delmas, J., Dutfoy, A., Fischer, R.: Maximum entropy copula with given diagonal
section. J. Multivar. Anal. 137, 61–81 (2015)
9. Ditlevsen, O., Madsen, H.: Structural Reliability Methods. Wiley, Chichester/New York (1996)
10. Dutfoy, A., Dutka-Malen, I., Pasanisi, A., Lebrun, R., Mangeant, F., Gupta, J.S., Pendola, M.,
Yalamas, T.: OpenTURNS, an open source initiative to treat uncertainties, Risks’N statistics in
a structured industrial approach. In: Proceedings of 41èmes Journées de Statistique, Bordeaux
(2009)
11. Fang, K.T., Li, R., Sudjianto, A.: Design and Modeling for Computer Experiments. Chapman
& Hall/CRC, Boca Raton (2006)
12. gum08: JCGM 100-2008 – Evaluation of measurement data – guide to the expression of
uncertainty in measurement. JCGM (2008)
13. Hyndman, R., Shang, H.: Rainbow plots, bagplots, and boxplots for functional data. J. Comput.
Graph. Stat. 19, 29–45 (2010)
14. Iooss, B., Lemaître, P.: A review on global sensitivity analysis methods. In: Meloni, C.,
Dellino, G. (eds.) Uncertainty Management in Simulation-Optimization of Complex Systems:
Algorithms and Applications. Springer, New York (2015)
15. Kurowicka, D., Cooke, R.: Uncertainty Analysis with High Dimensional Dependence Mod-
elling. Wiley, Chichester/Hoboken (2006)
16. Lebrun, R., Dutfoy, A.: Do rosenblatt and nataf isoprobabilistic transformations really differ?
Probab. Eng. Mech. 24, 577–584 (2009)
17. Lebrun, R., Dutfoy, A.: A generalization of the nataf transformation to distributions with
elliptical copula. Probab. Eng. Mech. 24, 172–178 (2009)
18. Lebrun, R., Dutfoy, A.: An innovating analysis of the nataf transformation from the viewpoint
of copula. Probab. Eng. Mech. 24, 312–320 (2009)
19. Lebrun, R., Dutfoy, A.: A practical approach to dependence modelling using copulas. J. Risk
Reliab. 223(04), 347–361 (2009)
20. Lebrun, R., Dutfoy, A.: Copulas for order statistics with prescribed margins. J. Multivar. Anal.
128, 120–133 (2014)
21. Lemaire, M.: Structural Reliability. Wiley, Hoboken (2009)
22. Liberty, L.: Ev3: a library for symbolic computation in c++ using n-ary trees, http://www.lix.
polytechnique.fr/~liberti/Ev3.pdf (2003)
23. Marin, J.M., Robert, C. (eds.): Bayesian Core: A Practical Approach to Computational
Bayesian Statistics. Springer, New York (2007)
24. Marrel, A., Iooss, B., Van Dorpe, F., Volkova, E.: An efficient methodology for modeling
complex computer codes with Gaussian processes. Comput. Stat. Data Anal. 52, 4731–4744
(2008)
25. Munoz-Zuniga, M., Garnier, J., Remy, E.: Adaptive directional stratification for controlled
estimation of the probability of a rare event. Reliab. Eng. Syst. Saf. 96, 1691–1712 (2011)
26. Nash, S.: A survey of truncated-newton methods. J. Comput. Appl. Math. 124, 45–59 (2000)
27. OPEN CASCADE S.: Salome: the open source integration platform for numerical simulation.
http://www.salome-platform.org (2006)
28. Pasanisi, A.: Uncertainty analysis and decision-aid: methodological, technical and managerial
contributions to engineering and R&D studies. Habilitation Thesis of Université de Technolo-
gie de Compiègne, France https://tel.archives-ouvertes.fr/tel-01002915 (2014)
 38 M. Baudin et al.

29. Pasanisi, A., Dutfoy, A.: An industrial viewpoint on uncertainty quantification in simulation:
stakes, methods, tools, examples. In: Dienstfrey, A., Boisvert, R. (eds.) Uncertainty Quantifi-
cation in Scientific Computing – 10th IFIP WG 2.5 Working Conference, WoCoUQ 2011,
Boulder, 1–4 Aug 2011. IFIP Advances in Information and Communication Technology,
vol. 377, pp. 27–45. Springer, Berlin (2012)
30. Rasmussen, C., Williams, C., Dietterich, T.: Gaussian Processes for Machine Learning. MIT,
Cambridge (2006)
31. Robert, C.P., Casella, G.: Monte Carlo Statistical Methods. Springer, New York (2004)
32. Rubinstein, R.: Simulation and the Monte-Carlo Methods. Wiley, New York (1981)
33. Sacks, J., Welch, W., Mitchell, T., Wynn, H.: Design and analysis of computer experiments.
Stat. Sci. 4, 409–435 (1989)
34. Saltelli, A.: Making best use of model evaluations to compute sensitivity indices. Comput.
Phys. Commun. 145, 280–297 (2002)
35. Saltelli, A., Tarantola, S., Chan, K.: A quantitative, model-independent method for global
sensitivity analysis of model output. Technometrics 41, 39–56 (1999)
36. Saltelli, A., Chan, K., Scott, E. (eds.): Sensitivity Analysis. Wiley Series in Probability and
Statistics. Wiley, Chichester/New York (2000)
37. Santner, T., Williams, B., Notz, W.: The Design and Analysis of Computer Experiments.
Springer, New York (2003)
38. Sudret, B.: Global sensitivity analysis using polynomial chaos expansion. Reliab. Eng. Syst.
Saf. 93, 964–979 (2008)
39. Tarantola, A.: Inverse Problem Theory and Methods for Model Parameter Estimation. Society
for Industrial and Applied Mathematics, Philadelphia (2005)
40. Top 500 Supercomputer Sites: Zumbrota http://www.top500.org/system/177726, BlueGene/Q,
Power BQC 16C 1.60GHz, Custom (2014)

View publication stats