Computer Predictions With Quantified Uncertainty, Part II
Computer Predictions With Quantified Uncertainty, Part II
Computer Predictions
with Quantified Uncertainty, Part II
By Tinsley Oden, Robert Moser, and Omar Ghattas
Having developed and calibrated a model to be used for a particular prediction (see Part I of this article, SIAM News, November 2010, page 1), we
are ready to begin the validation processthat is, to assess the suitability of the calibrated model for the prediction.
case inconsistencies in the prediction of the most probable value should be evaluated. Alternatively,
it might be important to determine the probability that a presumably unlikely failure will occur. The
invalidation metric should then assess the consistency of this prediction.
Another important issue is the selection of validation scenarios and measurements that challenge models in a way that is similar to the prediction scenario. Generally, such selections are based on the modelers understanding of the potential weaknesses in the models (e.g., questionable assumptions made)
and judgment regarding aspects of the models important for the predictions. The selection can also make
use of the computational model, analyzing sensitivities of a QoI to parameter and model variation, and
maximizing the information about the QoI to be gained. In complex multiphysics systems, designing a
validation process is also complicated by the need to perform validation at many levels of complexity,
from simple scenarios involving models of a single physical phenomenon, to complex scenarios involving all the models needed in the prediction scenario.
We represent these possible scenarios symbolically as a pyramid of complexity, as shown in Figure 3,
in which the lowest level involves calibration and simple validation tests for components of the system Figure 3. The prediction pyramid.
or single physical phenomena. At this level experiments generally involve simple scenarios (Sc ), and
are inexpensive and copious. Multicomponent and multiphysics validation is done at a higher level in the pyramid, based on experiments with more
complex scenarios (Sv). These experiments are designed to effectively challenge the ability of the model to predict the QoIs (the Qps in the diagram)
and are generally more expensive and less numerous. Finally, at the top of the pyramid, the full system is used in the prediction scenarios Sp , and
predictions q of Qp are made. Commonly, no observations are available for validation at this level.
Once a calibrated model has been judged likely to be valid, it is used to compute a predictable quantity of interest Qp, resulting in a distribution
of predictions q. What do we now mean by quantified uncertainty? The answer depends on the purpose of the predictions. Generally, we would be
concerned with some statistical properties of q, such as its moments, which effectively quantify the uncertainty in the prediction.
M ( m )( d m )
,
D ( d )
(1)
can actually be computationally formidable to evaluate, and a number of numerical algorithms have been developed for this purpose. The fundamental
problem is to generate samples from, or other representations of, the posterior probability distribution in, say, the calibration or inverse problem. The
most widely used algorithm is Markov chain Monte Carlo (MCMC), of which there are many variants. MCMC calculations generally converge slowly
and require many evaluations of the physical model (evaluating the likelihood), making them computationally expensive. Alternative representations
of the posterior pdfs commonly suffer from the curse of dimensionalitythe computational complexity, in other words, increases catastrophically
with the number of uncertain parameters. Once posterior pdfs of model parameters are known, the computational challenge is to calculate the probability distribution of output QoIs (the forward uncertainty propagation problem). Here, classic algorithms like Monte Carlo sampling are commonly
used, but they too converge slowly. As for inverse problems, each sample requires ev-aluation of the model, which makes them computationally
expensive. More advanced algorithms that converge more rapidly also commonly suffer from the curse of dimensionality. The development of rapidly
See, for example, C.P. Robert and G. Casella, Monte Carlo Statistical Methods, Springer, 2004.
See, for example, C.P. Robert, The Bayesian Choice, Springer, 2007.
converging algorithms for both forward and inverse uncertainty propagation problems that avoid the curse of dimensionality is a great need and a
topic of current research.
A promising approach for addressing these computational issues is to use more information about the structure of the physical model than is available from simple point evaluations of the inputoutput map. In particular, algorithms now being developed use derivatives of the QoIs with respect
to the uncertain inputs; in some cases these algorithms can accelerate convergence by orders of magnitude. When input parameters far outnumber
QoIs, gradients and even Hessians of the QoIs can be efficiently computed from solutions of adjoint problems, which are also used to determine a
posteriori estimates of numerical errors. Regardless of the specifics, algorithms are needed that can provide predictions with quantified uncertainty,
scaling to expensive models and large numbers of parameters.
Example
To see how the process described here works in a simulation, we consider a model of a turbulent boundary layer with several different pressure
gradients. A boundary layer is a thin region that forms near a solid surface when fluid flows over it, as in the flow of air over a car or airplane.
Turbulent boundary layers mediate the transfer of momentum from the fluid to the surface, resulting in viscous drag. In our example, turbulence
is represented with the SpalartAllmaras (SA) model, which has seven parameters that have to be calibrated. Because turbulence models are
imperfect, a statistical model of the model inadequacy is also considered. Experimental measurements of the streamwise velocity and of the
wall shear stress have been made. In a Bayesian inference framework, these measurements, along with estimates of the uncertainty in the data,
are used to determine the parameters for the SA model, both alone and with the inadequacy model. The posterior probability distributions of
two SA model parameters for cases with and without the inadequacy model are shown in Figure 4. The parameter k (which is identified with
Figure 4. Posterior marginal probability distributions of the parameters k and cv1 in the SpalartAllmaras turbulence model and probability distributions of the wall shear stress (q = tw / rf , the QoI, where rf is the fluid mass density) in the prediction scenario. Shown are probability density
functions for models with and without inadequacy models and the prior pdfs, which are uniform distributions over a selected range.
the Karman constant) is well informed by the data, with rather narrow posterior distributions, while cv1 is less well informed (with a distribution 2.5 times wider).
We wish to use the calibrated model to predict the wall shear stress for a stronger pressure gradient than that for which data are available.
This is the QoI, which we want to predict to within, say, 5%. To see if the model is valid for this purpose, we again use Bayesian inference to
evaluate the posterior probabilities, given the data, of the model with and without the inadequacy model (this is the plausibility). The probability
of the model with no inadequacy model is essentially zero (order 10 10), indicating that the inadequacy model is necessary for consistency with
the data. Furthermore, as shown in Figure 4, where the predicted distributions of the QoI are plotted, the inadequacy model makes a difference
of more than 5% to the prediction. This indicates that, according to the available data, the model as calibrated (without the inadequacy model)
is not valid for predicting the QoI within the required tolerance. In particular, without the inadequacy model, the uncertainty in the prediction
is greatly underestimated. These results, however, say nothing about the validity of the model that includes the inadequacy model.
Summary
We have seen that the consideration and quantification of uncertainty in computational predictions introduce new challenges to the processes of calibration and validation of mathematical models and their use in making predictions. In calibration, uncertainties in experimental
observations and uncertainties arising from inadequacies or randomness in the mathematical models must be accounted for to obtain estimates
of input parameters and their uncertainties. The validation process is designed to challenge the models in their ability to predict the quantities
of interest, and must account for the uncertainty of the observations and of the model and its parameters. Finally, issuing predictions of the
quantities of interest involves estimating the QoIs and their uncertainties given the uncertainties in models and inputs. These uncertainty considerations require a mathematical representation of uncertainty, and we argue that probability in the context of Bayesian statistics provides a
particularly powerful representation. Many challenges remain, however, for the development of new or improved conceptual and computational
tools needed to treat uncertainty in computational simulations of complex systems. Among the challenges are representing model inadequacy,
characterizing the usefulness of particular validation tests, determining the consequences of validation comparisons on the quantities of interest,
posing validation metrics that are appropriate in various situations, and devising efficient algorithms for the required computations, especially
when there are many uncertain parameters and the models are computationally expensive.
Challenges are to be expected in such a young field, but we build on a foundation in the philosophy of science that was laid long ago by the
likes of the Reverend Thomas Bayes and Sir Karl Popper.
For details, see Cheung et al., to appear in Reliab. Eng. Syst. Safety special issue on quantification of margins and uncertainty (QMU), 2010.
Acknowledgments
We have benefited from numerous discussions with I. Babuka and R. Tempone and from their published work on verification, validation, and uncertainty
quantification. We thank the staff of the PECOS Center for their contributions to pursuing these ideas, and in particular T. Oliver, E. Prudencio, and S.H.
Cheung for the results shown in Figure 4. The financial support of the PECOS Center by the Department of Energy under the National Nuclear Security
Administrations Predictive Science Academic Alliance Program is gratefully acknowledged (DEFC5208NA28615).
Robert Moser is PI, and Tinsley Oden and Omar Ghattas are Co-PIs of the Center for Predictive Engineering and Computational Sciences
(PECOS) of the Institute for Computational Engineering and Sciences (ICES) at the University of Texas at Austin.