Comparison of Different Uses of Metamodels for Robust

Design Optimization
Johan A. Persson1 and Johan lvander2
Linkping University, 581 83, Linkping, Sweden

This paper compares different approaches for using kriging metamodels for robust
design optimization, with the aim of improving the knowledge of the performance of the
approaches. A popular approach is to first fit a metamodel to the original model and then
perform the robust design optimization on the metamodel. However, it is also possible to
create metamodels during the optimization. Additionally, the metamodel need not
necessarily reanimate the original model; it may also model the mean value, variance or the
actual objective function. The comparisons are based on two analytical functions and a
dynamic simulation model of an aircraft system as an engineering application. In the
comparisons, it is seen that creating a global metamodel during the optimization slightly
outperforms the other approaches that involve metamodels.

Nomenclature
A2
As
i
k

mp
n
ncalls
P
R(x1,x2)
s
2
tfill
xi
Z(x)

=
=
=
=
=
=
=
=
=
=
=
=
=
=
=

an area of the piston found inside the main valve in the dynamic pressure regulator
an area of the piston found inside the main valve in the dynamic pressure regulator
the ith regression coefficient
number of points used for the complex algorithm
the mean value
the mass of the piston inside the main valve
number of optimizations
required number of function evaluations needed for the optimization algorithm to converge
the probability of finding the global optimum by performing one optimization
the correlation function between x1 and x2
an estimation of the standard deviation
the variance
the time it takes to fill the environmental system
value of the ith variable
A stochastic process

I. Introduction

obust design optimization (RDO) may be used in the design process to find optimal designs which are
insensitive to uncertainties and errors.1-3 Since decisions about designs need to be taken by the developers even
though information is lacking, there will always be uncertainties and variations present.4,5 These may among other
things stem from measurements, manufacturing tolerances and environmental conditions.6
Optimization Algorithm
Objective Function
Estimate Statistics
Model

Figure 1. A general workflow of a Robust Design Optimization process.

1
2

PhD Student, Department of Management and Engineering, [email protected].

Professor, Department of Management and Engineering, [email protected].
1
American Institute of Aeronautics and Astronautics

A general workflow of an RDO is shown in Figure 1. The mean value and standard deviation are estimated each
time the objective function value is called by the optimization algorithm. Consequently RDO requires numerous
simulations of the computer model. To avoid unrealistically long simulation times and occupation of precious
resources, methods have been suggested where computationally effective global metamodels are used instead of the
original models.1 A metamodel is a numerical model of how a quantity varies when the parameters that are affecting
it are varied and consequently it is possible to replace each of the activities in Figure 1 with a metamodel. Several
possible RDO approaches therefore exist, where different entities are replaced by metamodels.
A common approach is to fit a global metamodel to the desired output of the original model.7-9 The global model
is then called to estimate the objective function in the RDO. Since the position of the optimum is to some degree
unknown beforehand, the metamodel needs to reanimate the whole design space accurately. This requires numerous
simulations of the original model and some of them may be in areas which the optimization algorithm never reaches.
Another approach is to fit a global metamodel to the original model as the RDO progresses, meaning that the only
simulations of the original model that are performed are made with parameter values which are used by the RDO.10
The aim of this paper is to compare several RDO approaches, where different entities are replaced by
metamodels, for two analytical examples and an aircraft system model. Apart from giving insights about the
modeled system, a comparison increases knowledge of when to apply the different methods. There exist numerous
types of metamodels and to narrow down the number of compared approaches in this paper, only one type of
metamodel is used. While many comparisons of metamodels have been made,11-14 none stand out as clearly
superior9. However, the kriging metamodel performs well in the comparisons and is also applied to RDO problems
by several authors.15-17 It has therefore been chosen as metamodel for this comparison.
To further decrease the number of model calls, an efficient sampling method and optimization algorithm are
chosen. Latin Hypercube Sampling (LHS) is an efficient sampling based method which has been proven to yield
accurate estimations for just tens of samples,18,19 and is therefore chosen as the sampling method in this work. The
Complex-RF algorithm is a fairly efficient optimization algorithm and its performance has been demonstrated for
system models.20,21 The same sampling methods and optimization algorithms are used in all approaches to make the
comparison fair and keep differences to a minimum.
A brief description of Robust Design Optimization can be found in section II, together with descriptions of Latin
Hypercube Sampling, Kriging and Complex-RF. The compared approaches are presented in section III and the
comparison is presented in section IV. The final sections, V and VI, respectively comprise a discussion and the
conclusions.

II. Robust Design Optimization

It is possible to find an optimal solution which is insensitive to uncertainties and errors by performing an RDO.
This is achieved by optimizing the mean value and standard deviation of the desired system properties. Mean values
and standard deviations are statistical entities which can be estimated in different ways.
For all methods, the model parameters which should be used as variables in the robust design optimization are
identified and their uncertainties estimated or measured. The parameters which influence the system properties of
interest may be identified by performing a local sensitivity analysis22,23 or a screening24,25. By assigning the values of
the model parameters to standard probability distributions, it is possible to describe the characteristics of the
uncertainties with just a few parameters.
Many methods exist for estimating how the uncertainties and errors in the model parameters affect the system
properties of interest and a few are presented in Ref 7 and Ref 19. Generally, the methods return probability
distributions for the system properties of interest. Those may then be used for taking decisions based on
probabilities. A combination of the mean values and standard deviations of those distributions are used as an
objective function for RDO, while the probability of failure is used as the objective function for Reliability-Based
Design Optimization (RBDO).1
In this paper, sampling based approaches are used to estimate the probability distributions of the system
characteristics. The most renowned sampling method is probably the Monte Carlo Simulation (MCS) which draws
random samples from the probability distributions of the input parameters. As the number of samples increases, the
estimation will converge towards the real probability distributions according to the law of large numbers, but this
might mean tens of thousands of samples.19 This is unrealistic for RDO since the probability distributions need to be
calculated each time the objective function is called. A more effective sampling method, Latin Hypercube Sampling,
is instead used for the sampling in this paper.

2
American Institute of Aeronautics and Astronautics

A. Latin Hypercube Sampling

LHS is an effective sampling method where the design space is divided into n intervals of equal probability.18
One sample is drawn from each interval which ensures that the n samples are spread around the design space. It has
been shown to give accurate estimations of the mean value and standard deviation of the model output for less than
one hundred samples.18,19 One advantage of LHS is that the required number of samples does not scale with the
number of design parameters. A drawback compared to MCS, however, is that it is difficult to add samples to a
performed LHS. For MCS it is possible to add samples until the desired error is reached, whereas it is only possible
to increase the LHS by doubling the number of samples.
B. Kriging
Originally, kriging metamodels were used for geostatistical applications but they have also been demonstrated to
perform well for high fidelity computer models.11,15 A kriging model is a group of interpolation techniques for
spatially correlated data and is based on a stochastic model. When a kriging model is used as a deterministic
metamodel, the expected value at the desired point is used as the approximate value at the point.16
A kriging model is a combination of a polynomial model and local deviations of the form seen in Eq. (1).

y = i fi ( x ) + Z ( x )
i =1

(1)

i is the ith regression coefficient, whereas fi(x) is a function of the variables x and Z(x) a stochastic process with
zero mean and a spatial correlation function given by Eq. (2).

Cov Z (xi ), Z (x j ) = 2 R(xi , x j )

(2)

2 is the variance of the process and R(xi,xj) is a correlation function between xi and xj. The first term in Eq. (1)
is an approximation of the trend of the model output, resembling a polynomial metamodel, and is modified
according to the problem.17 The correlation function includes unknown parameters and unfortunately it is timeconsuming to determine the values of the parameters. This is an optimization problem in its own right and may be
solved by using maximum likelihood estimation; see for example.17
C. Complex-RF
The Complex-RF optimization algorithm was developed from the Nelder-Mead Simplex method. The algorithm
uses k number of points for converging and in each iteration the worst point is moved along a line through the center
of the other points until it is no longer the worst.20,21,26 Another point is now the worst and consequently moved until
another point is worse. This process continues until the stop criterion is met or the maximum number of evaluations
is reached.
Since the Simplex and Complex methods use several starting points and spread them around the design space,
they are suitable for problems which are sensitive to the choice of starting point. Furthermore, they do not use any
derivatives of the objective function and are not dependent on the linearity or differentiability of the objective
functions, and hence are applicable to a wide range of problems.

III. Compared Approaches

This section presents six different approaches for performing robust design optimization which are compared in
this paper. Schematic workflows for the six approaches can be seen in Figure 2.
A. Approach 1
The first approach is the most straightforward and almost a brute force method. Each time the optimization
algorithm requests the value of the objective function, the mean value and standard deviation of the model
parameters are sent to the Latin Hypercube Sampler. The sampler draws samples from the true model and estimates
the mean value and standard deviation of the system properties of interest. A combination of these values is used to
calculate the objective function. This process can be seen in Figure 2a).

3
American Institute of Aeronautics and Astronautics

B. Approach 2
Shown in Figure 2b), this approach starts by drawing a predefined number of samples from the original model by
means of LHS, which is used to spread samples for metamodels by several authors.16,17 These samples are used to fit
a global kriging model to the output of the original model. The optimization then works as in Approach 1, with one
exception. Each time LHS draws samples, they are drawn from the kriging model instead of the original model.
Consequently, the only simulations of the original model are the ones that are performed to fit the kriging model.
This is probably the most widely used approach where metamodels are utilized.
Set up
problem
Optimization

Yes

No
Create a MM of the
system
Yes
No

Enough
samples to fit
a MM?

No

Draw samples from

the model to
estimate mean and
standard deviation

Draw samples from

the MM to estimate
mean and standard
deviation

Build objective
function from mean
and standard
deviation

Build objective
function from mean
and standard
deviation

Convergence?

Convergence?

New variable values

New variable values

Does MM
exist?

No

Yes

Yes

Optimal
solution

a) Approach 1

b) Approach 2

c) Approach 3

d) Approach 4
e) Approach 5
Figure 2. Workflows for the five approaches
C. Approach 3
The workflow can be seen in Figure 2c) and at the beginning of the optimization process this approach performs
the same operations as Approach 1. However, all calls of the original model by the LHS are stored until a predefined
number of such calls have been made. A kriging model of the output of the original model is now fitted by using
these model calls as samples. During the rest of the optimization, the LHS calls the kriging model instead of the
4
American Institute of Aeronautics and Astronautics

original model to draw the remaining samples. As in Approach 2, the samples required to fit the kriging model are
the only simulations of the original model that are performed.
D. Approach 4
In this approach, shown in Figure 2d), the optimization process progresses as in Approach 1 until a predefined
number of mean values and standard deviations have been estimated. These estimations are used as samples to fit
one kriging model of how the mean value varies in the design space, and one kriging model for the variation of the
standard deviation. When the kriging models have been created, the objective function calls the two metamodels in
order to calculate its value during the rest of the optimization. The approach with creating metamodels of the mean
value and standard deviation instead of the original model itself has been demonstrated by for example Lnn et al.27
This method has a similar drawback to Approach 1. The LHS draws samples from the original model each time
the mean value and standard deviation are estimated until they have been estimated the required number of times to
enable the fitting of kriging models. Consequently, the required number of simulations of the real model is the
number of samples drawn to estimate each mean value multiplied by the required number of mean values to fit a
kriging model.

New variable values

E. Approach 5
As can be seen from Figure 2e), this approach is rather similar to Approach 4, but instead of creating kriging
models of the mean value and standard deviation, a kriging model is fitted directly to the objective function value.
When the predefined number of objective function values have been estimated using LHS, a kriging model is fitted.
During the remainder of the optimization, the estimations of the objective function are calculated from the kriging
model.

Figure 3. Workflow for the sixth approach.

F. Approach 6
The comparison that is seen in section IV reveals that approaches 3 to 5 are inferior to approach 2, with approach
3 superior to 4 and 5. This would lead to the assumption that it is inferior to create a metamodel during the
optimization compared to creating it before the optimization is started. However, it should be possible to find an
5
American Institute of Aeronautics and Astronautics

approach that fits a metamodel during the optimization, which performs better than approach 2. Therefore, a sixth
approach with the workflow seen in Figure 3 is suggested. The comparison in section IV also suggests that it is
better to let the metamodel reanimate the response from the original model than the mean value and standard
deviation or the whole objective function. The metamodels in Approach 6 therefore reanimate the response of the
original model.
Approach 6 starts by spreading k points in the design space similar to the start of the Complex optimization
algorithm. The deterministic values at the points are calculated by calling the original function. A metamodel is then
fitted to the values. LHS is then used to estimate the mean values and standard deviations at the k points by calling
the metamodel. The number of points, k, is set to eight for the comparison in this paper. k needs to be large enough
to enable a metamodel to be fitted in the initial stage, but it is also desirable to keep it small in order to save as many
calls to the original model as possible for later iterations of the optimization process.
As long as the maximum number of calls to the original model is not, the deterministic value at the point is
calculated by calling the original model reached each time the optimization algorithm wants to estimate the
objective function value. The surrogate model is then updated by including the new value in the samples that are
used to fit the surrogate model.
Latin Hypercube Sampling calls the surrogate model to estimate the mean value and standard deviation that are
needed to determine the value of the objective function. This means that only one call to the original model is made
each time the value of the objective function is calculated, until the maximum number of calls to the original
function is reached.

IV. Comparison of the Approaches

The performance of each of the presented approaches is compared for two analytical functions and a dynamic
system model of an airplane system. For an optimization algorithm, the performance of interest is the chance of
finding the global optimum, denoted hit-rate, and the number of function calls made, ncalls. To estimate the hit-rate,
1,000 optimizations are made for each approach and problem.
For RDO, it is desirable to find a design where the expected performance is as optimal as possible, while the
dispersion is minimal. This can be formulated as an objective function with a linear combination of the mean value,
, and standard deviation, s, of the stochastic model output. Since the mean value and standard deviation of a
stochastic response often are unknown, they need to be approximated. One option is to estimate them according to
Eq. (3) by drawing samples using LHS. For the objective functions in this paper, the weight of the mean value is set
to one and the weight of the standard deviation is set to three, as shown in Eq. (3). The higher weight for the
standard deviation specifies that in this case a design with a small dispersion in performance is more important than
one with a better mean value.

min f (x ) = ( g ( x )) + 3 (g ( x ))

( g ( x )) =

( g ( x )) =

1 m
g (x j )
m j =1

1 m
(g (x j ) )2

m 1 j =1

(3)

A. Performance Index
It is desirable to enable comparison of the performance of different optimization algorithms by using a numerical
value. A numerical value may also be used as an objective function for optimizing the parameters of optimization
algorithms. The performance index suggested in this paper is derived as follows.
If P is the chance of finding the global optimum by performing one optimization, the hit-rate, then Eq. (4)
specifies the probability of not finding the optimum by performing n optimizations.

Pnot _ found = (1 P)

6
American Institute of Aeronautics and Astronautics

(4)

Consequently, Eq. (5) describes the probability of finding the optimum by performing n optimizations.

Pfound = 1 Pnot _ found = 1 (1 P)

(5)

To make a fair comparison between methods which require different numbers of function calls, n should be
replaced by the allowed number of function calls divided by the required number of function calls, ncalls, as shown in
Eq. (6). The allowed number of function calls depends on the problem, but as long as it is the same for all compared
methods, the actual number is not important. Here, the allowed number of function calls is set to 100, meaning that
the performance index is the probability of finding the global optimum if 100 calls of the original model are made.

Perf _ index = 1 (1 P)

100 / ncalls

(6)

This performance index is not adequate for trivial problems where the hit-rate is 100%, since Eq. (6) will equal
one if P = 1 regardless of the required number of model calls. But if two optimization algorithms have the same hitrate, it is just a matter of comparing the required number of model calls for the two algorithms, since the one
requiring the fewest number of model calls is the better.
B. The Peaks function
The approaches are compared for the Peaks function which is implemented in MATLAB and is called by writing
peaks in the command window. Here, it is shown in Figure 4 and Eq. (7).

1
x

2
2
2
g1 (x ) = 3(1 x1 ) exp x12 ( x 2 + 1) 10 1 x13 x 25 exp x12 x 22 exp ( x1 + 1) x 22
5
3

subject to

3 xi 3, i = 1,2

(7)

It is a multimodal function, with a global deterministic minimum in [0.231 -1.626], whereas [-1.348 0.205] is a
local deterministic minimum. For deterministic optimizations with Complex-RF, the probability of finding the
global optimum is 73%.20 To convert the deterministic problem into a robustness problem, randomness is introduced
by allowing the parameter values of a point to vary in an interval of 0.1 around the point.

Figure 4. Graph over how the value of the Peaks function varies in the design space.
The optimal point from a robust point of view depends on how much the standard deviation is weighted in
relation to the mean value. If Figure 4 had been a plot of how the performance of a product varies with varying
modeling parameters, it would probably be desirable to choose the global deterministic optimum as the design that
should be realized. Even though the slopes are steep around the global deterministic optimum, its performance is
7
American Institute of Aeronautics and Astronautics

still better than most other design points. Consequently, the difference with a deterministic optimization is that it is
computationally more expensive to perform a robust design optimization since the mean value and standard
deviation need to be estimated for each design point. The criterion for a successful optimization is chosen to be an
optimization where the optimal solution lies in the vicinity of the global deterministic optimum, i.e. fulfills 0.2 < x1
< 0.3, -1.7 < x2 < -1.6.
The comparisons of the different approaches can be seen in Table 1. The aim is to make the comparisons fair by
allowing the same number of calls of the Peaks function for all approaches where a predefined number of calls can
be set.
The row named hit-rate indicates what percentage of the 1,000 optimizations performed converged to a design
point fulfilling the criterion for a successful optimization. A high number is advantageous since it indicates that the
probability of finding the global optimum by performing one optimization is high.
The number of objective function evaluations is a measurement of how many times the optimization algorithm
calculated the value of the objective function, either by calling a kriging metamodel (A4 & A5) or performing an
LHS of a kriging model (A2, A3 and A6) or the original model (A1). Since it can be seen as a measure of how many
operations the algorithm performs, it is an advantage to have as few objective function evaluations as possible.
The number of samples for each LHS indicates how many samples of the original model or kriging model each
time an LHS is performed. A high number is advantageous since having more samples improves the accuracy of the
estimations of the mean value and standard deviations. However, a high number also means that more samples need
to be drawn each time an LHS is performed and consequently the computational time will increase.
The next row indicates the chosen number of samples used to fit the kriging models for each approach. No
kriging model is created for Approach 1, whereas Approach 2, 3 and 6 fit their kriging models from 100 samples of
the original model in Table 1. From the same table it can be seen that the kriging models in Approach 4 are fitted
when 10 mean values and standard deviations have been estimated by performing 10 LHSs of the original model.
Similarly, a kriging model is created in Approach 5 when 10 objective function values have been estimated.
The number of calls of the original model is an interesting entity since it is used to estimate the performance
index of the optimization algorithms. For expensive models, each simulation of the original model might be more
computationally demanding than all the operations performed by the optimization algorithm. Consequently, the
number of calls of the original model determines the wall clock time for the robust design optimization process for
expensive models.
In the bottommost row, the performance index for each optimization can be seen. This is the most revealing
characteristics of the performance of each approach for the Peaks function, since it is an estimation of how probable
it is that the global optimum will be found by performing 100 simulations of the original model. As mentioned in
Eq. (6), it is calculated from the hit-rate and the required number of calls of the original model, which corresponds to
rows three and seven in the table.
Table 1. Comparison of the five approaches for 100 allowed calls of Peaks.
Approach
A1
A2
A3
A4
A5
Hit-Rate [%]
59.0
42.8
30.9
0.3
0.1
Number of objective function evaluations 628
602
544
95
94
Number of samples for LHS
10
20
10
10
10
Number of samples for kriging
100
100
10
10
Number of calls of the original model
6280
100
100
100
100
Performance Index
0.014 0.428 0.309 0.003 0.001

A6
86.0
2000
10
100
100
0.86

Approach 6 is superior to the other approaches according to the performance indices in Table 1. The large
number of objective function evaluations for Approach 6 is not a great issue since most of them are made on the
computationally effective metamodel.
The large number of calls of the original model for Approach 1 deteriorates its performance index making it a
poor choice even though its hit-rate is the second highest. Approaches 2 and 3 have significantly higher hit-rates
than Approach 4 and Approach 5, where Approach 2 is the best and Approach 5 the worst. The hit-rates in
Approach 4 and Approach 5 are close to zero, indicating that the probability of finding the global optimum is
minimal, making them unsuitable for solving this problem.

8
American Institute of Aeronautics and Astronautics

C. The Aspenberg Function

An interesting function from a robustness point of view is the function presented for one variable in Ref 28.
Figure 5a) shows how its function value changes when the value of its variable is altered. For this comparison it has
been modified to be a function of two variables, shown in Eq. (8), to increase the complexity of the function.
n

g 2 ( x ) = 4 x + e
2
i

30

xi2
i

+e

10

( xi 0.25 )2
i

subject to

3 xi 3, i = 1,2

(8)

The function has a deterministic optimum in [-0.280 -0.280], which is almost always found for deterministic
optimizations using Complex-RF. For the objective function that is presented in Eq. (3) it has robust optimums in [0.25 0.10] and [0.10 -0.25], which can be seen in Figure 5b). The criterion for a successful optimization is chosen to
be when the optimization algorithm suggests a point fulfilling either -0.3 < x1 < -0.2, 0.01< x2 < 0.18 or 0.01 < x1 <
0.18, -0.3< x2 < -0.2. These two parts of the design space are similar and therefore it does not matter which of them
the suggested optimum belongs to.
The Aspenberg function for one variable

The value of Eq. 3 for the Aspenberg function

4.5

0.5

0.4

3.5

0.3
0.2

0.1
x2

g(x)

2.5
0

2
-0.1
1.5

-0.2

-0.3

0.5
0
-1

-0.4
-0.8

-0.6

-0.4

-0.2

0
x1

0.2

0.4

0.6

0.8

-0.5
-0.5

-0.4

-0.3

-0.2

-0.1

0
x1

0.1

0.2

0.3

0.4

0.5

a) 1D
b) 3D-figure
Figure 5. Two Graphs that shows a) the deterministic value of Eq. (8) for one variable and b) the value of
Eq. (3) for two variables.
In Table 2, a comparison of the different approaches can be seen for 100 samples. It is worth noting that
Approach 1 finds the robust optimum almost every time. Naturally, this leads to a reasonably high performance
index even though the number of calls of the function is huge. 1310 simulations of a computationally expensive
model are probably unrealistically many. Approach 2 and Approach 3 need over a thousand objective function
evaluations to converge, but as most of the evaluations are made on a computationally cheap kriging model it is not
an important issue. Approaches 4 and 5 display poor hit-rates for this problem and consequently get worse
performance indices than the other approaches. The sixth approach gets the highest performance index, mainly due
to its high hit-rate.

9
American Institute of Aeronautics and Astronautics

Table 2. Comparison of the five approaches for 100 allowed calls of Aspenbergs function.
Approach
A1
A2
A3
A4
A5
A6
Hit-Rate [%]
99.9
24.6
21.2
5.3
6.2
99.0
Number of objective function evaluations 131
1200 1189
137
140 2000
Number of samples for LHS
10
10
10
10
10
10
Number of samples for kriging
100
100
10
10
100
Number of calls of the original model
1310
100
100
100
100
100
Performance Index
0.410 0.227 0.212 0.053 0.062 0.99

D. An Aircraft System A Dynamic Pressure Regulator

The aircraft system used for this comparison is the design of a dynamic pressure regulator (DPR) used to control
the air pressure delivered to an environmental control system. The system is modeled in Dymola and two
screenshots from Dymola can be seen in Figure 6. A more thorough description can be found in.19

Figure 6. Screenshots from Dymola that display the pressure regulator and the
components inside it.
The most important function of the DPR is to ensure that the control system is filled as fast as possible.
Consequently, the filling time, tfill, is the most important system characteristics. It is desirable to find the parameters
that effect the characteristics most to improve the performance of the DPR. One way to identify important
parameters is to perform a local sensitivity analysis in a reference design point.22 Partial derivatives for how each
parameter affects the system characteristics are received by varying one parameter at a time. The normalized
sensitivities from a local sensitivity are shown in Table 3. It can be seen that the three most important parameters are
the areas of the piston inside the main valve, A2 and As, and the mass of the piston, mp.
Max opening area of
on/off-valve

Max opening area of

vent valve, Av

Max opening area of

main valve, Am

Friction coefficient
in main valve

Mass of piston, mp

Preloading of piston

Piston area for

output flow, A2

Piston area for

support pressure As

tf

Tank pressure, pt

Table 3. Normalized sensitivities for a specific design of the dynamic pressure regulator

1.4E-4

-5.7E-5

-8.4E-10

2.5E-4

0.0075

0.50

4.6E-4

-0.25

-0.26

10
American Institute of Aeronautics and Astronautics

When the important system characteristics and parameters are identified, it is time to set up an objective
function. It is desirable to minimize both the time it takes to fill the control system and its standard deviation. The
mathematical formulation can be found in Eq. (9).

min f ( x ) = (t fill ( x )) + 3 (t fill ( x ))

(g ( x )) =

( g (x )) =

1 m
g (x j )
m j =1

1 m
(g (x j ) )2

m 1 j =1
subject to

10 4 x1 10 6

106 x2 105
0.1 x3 1.0

(9)

The system parameters are considered to follow uniform distributions with a spread of 10% around their mean
values. This leads to Table 4, which contains statistics for the different approaches. Fortunately, all the approaches
which involve metamodels perform better than the brute force method even for this engineering application.
Approach 6 performs best, followed by Approach 2 and 3.
Table 4. Comparison of the five approaches for 100 allowed calls of the dynamic pressure
regulator.
Approach
A1
A2
A3
A4
A5
A6
Hit-Rate [%]
84.0
51.3
47.0
26.0
22.8
76.0
Number of objective function evaluations 499
4000 4000 4000 4000 2000
Number of samples for LHS
5
5
5
5
5
5
Number of samples for kriging
50
50
10
10
50
Number of calls of the original model
2495
50
50
50
50
50
Performance Index
0.070 0.762 0.719 0.452 0.404 0.942

V. Discussion
Approach 1, which does not involve any metamodels, has the highest hit-rate. However, thousands of calls of the
original model are needed for the algorithm to converge. This number of simulations is probably unrealistically large
for computationally expensive models. If it is realistic to perform so many model simulations it is doubtful whether
the model should be described as computationally expensive. The fact that the approaches that involve metamodels
need more objective function evaluations than Approach 1 is not a major issue, since most are computationally
cheap calls of a kriging metamodel.
The common approach of fitting a global metamodel to the output of the original model and performing the
optimization on the metamodel has its advantages. The global metamodel may be used instead of the original model
for other analyses, which can reduce the time for analyses significantly. In the other approaches that involve
metamodels, the metamodels are fitted with the aim of improving accuracy in the vicinity of the optimum of the
performed optimization and consequently lose precision in other parts of the design space, which might be
interesting for other analyses. Optimization algorithms with high hit-rates, but requires many function evaluations to
11
American Institute of Aeronautics and Astronautics

converge, may be used to perform the optimization since all function evaluations are made by calling the
metamodel.
It might also be that the original model is too computationally expensive to enable tens of simulations to be run,
making Approach 2 the only realistic approach. Naturally, its precision decreases with a decreasing number of
samples, but so does the precision of the others. Approaches 4 and 5 require several simulations of the original
model for each sample that can be used to fit the metamodels. If few calls of the original model are allowed, the
metamodels might need to be fitted using only a few samples which leads to low accuracy.
The poor performance of Approach 3 compared to Approach 2 can be explained by the organization of the
samples used for creating the metamodel. Since the LHS draws samples to estimate the mean value and standard
deviation each time the objective function is called by the optimization algorithm, the samples will be clustered
around each design point, like islands in a lake. Unfortunately samples may be drawn far away from the islands,
where precision is less good, when the metamodel is used during later iterations of the optimization algorithm.
For Approach 4 and Approach 5 to work, enough iterations of the optimization need to be made to enable the
fitting of an accurate metamodel in the vicinity of the optimum. Since an LHS is performed for each sample used to
fit the kriging model, the required number of model simulations equals the number of samples drawn for each LHS
multiplied by the number of samples needed to fit an accurate metamodel. Consequently, the number of simulations
will be quite high and as can be seen in the examples for Approach 2, quite accurate global metamodels can be fitted
using the same number of simulations.
In these examples, it would seem that the allowed numbers of function calls is too small to create accurate
metamodels for Approach 4 and Approach 5 since their hit-rates are not as good as in Approach 1. It is possible to
reallocate the calls of the original function between the samples that are used to estimate the mean value and
standard deviation, and the samples that are used to construct the kriging models. But if the number of function calls
should remain the same, using more samples to construct the kriging models will result in fewer samples for
estimating the mean value and standard deviation.
Approach 2 and Approach 3 seem more robust than Approach 4 and Approach 5 since all allowed calls of the
original function for Approach 2 and Approach 3 are used to fit a kriging model, whereas the allowed calls are split
between fitting a kriging model and estimating the mean value and standard deviation for Approach 4 and Approach
5.
Approach 6 performs better than the other methods for all of the tested problems, which indicates that it is
possible to fit a metamodel iteratively during the optimization process and perform robust design optimization
effectively. It is possible to increase the performance further by increasing the complexity of the approach, but it is
important that the mechanisms are not too complicated. It is desirable to use an approach which is effective, robust
and easy to use in real industrial problems.

VI. Conclusion
The comparison of the different approaches highlights their differences and can be used as a guideline for
choosing an approach in other engineering problems. Since Approach 1 does not involve metamodels, it can be seen
as a reference for the precision of the other approaches. The two analytical functions support the comparison since
the analytically optimal point is known, whereas the aircraft system model demonstrates the performance of the
approaches for engineering problems.
The approaches also suggest optimal parameters for the aircraft system model and consequently knowledge of
the system is increased.
The sixth approach, which creates metamodels iteratively during the optimization process, outperforms the other
approaches. It is also quite easy to use which means that it is a promising candidate for anyone which interested in
performing robust design optimization. The other three approaches which create metamodels iteratively during the
design process perform inadequately. When the metamodels are created after several iterations of the optimization
process, the samples that are used to fit the metamodels will be placed in the vicinity of the optimum. The
metamodels will as a result focus on reanimating the vicinity of the optimum accurately, which leads to good
estimations when the metamodels are used during the later iterations of the optimization process. The drawback
shown in the examples is that these approaches may need many samples to create accurate metamodels and that they
therefore need to be carefully designed to get a desirable performance. The approaches which reanimate the mean
value and standard deviation or the objective function perform worse than the other approaches that uses
metamodels and are therefore deemed unsuitable for robust design optimization.
The common approach of creating a global metamodel of the original model and performing an RDO on the
metamodel also performs well. The fitting of a global metamodel has two other advantages. The global metamodel
12
American Institute of Aeronautics and Astronautics

can be used after the optimization for other analyses. The metamodels fitted during the optimization are accurate in a
smaller part of the design space. Moreover, the original model might be too expensive to enable tens of simulations,
leaving the fitting of a global metamodel as the only realistic option.
Naturally, it is up to the system engineer to decide which method is best for a particular problem. However, it
seems appropriate to use a method which involves metamodels for RDO of computationally expensive models, to
avoid too tedious calculations.

Acknowledgments
The research performed in this paper has received founding by the European Communitys Seventh Framework
Program under grant agreement no. 234344 (www.crescendo-fp7.eu).

References
1

Lnn, D., Robust design - Accounting for uncertainties in engineering, Linkping Studies in Science and Technology.,
Thesis No 1389, Linkping University, Linkping, Sweden 2008.
2
Kazemi, M. Metamodel-Based Optimization for Problems With Expensive Objective and Constraint Functions, Journal of
Mechanical Design, Vol. 133, January 2011.
3
Padulo, M., Computational Engineering Design under uncertainty An aircraft conceptual design perspective, Cranfield
University, PhD Thesis, July 2009.
4
Mavris, D.N., DeLaurentis, D.A., Bandte, O. and Hale, M.A.,A Stochastic Approach to Multi-Disciplinary Aircraft
Analysis and Design, AIAA 98-0912, 1998.
5
Andersson J., Multiobjective Optimization in Engineering Design - Applications to Fluid Power Systems, Dissertation,
Linkping studies in Science and Technology, Dissertation No. 675, Linkping University, Linkping, Sweden, 2001.
6
Beckwith, T. G., Marangoni, R.D. and Lienhard V, J.H., Mechanical Measurements, sixth edition, Pearson Prentice Hall,
Upper Saddle River, NJ, 2009, Chaps. 3.
7
Beyer, H-G., and Sendhoff, B., Robust Optimization A comprehensive survey, Computer Methods in Applied
Mechanics and Engineering, Vol. 196, 2007, pp. 3190-3218.
8
Park, H-S., and Dang, X-P., Structural optimization based on CAD-CAE integration and metamodeling techniques,
Computer-Aided Design, Vol. 42, 2010, pp. 889-902.
9
Wang, G., and Shan, S., Review of Metamodeling Techniques in Support of Engineering Design Optimization, Journal of
Mechanical Design, Vol. 129, No. 2, April 2007, pp. 370-380.
10
Duvigneau, R., and Praveen, C., Meta-Modeling for Robust Design and Multi-Level Optimization, 42nd AAAF Congress
on Applied Aerodynamics, Sophia-Antipolis, France, March 2007
11
Jin, R., Du, X., and Chen, W., The use of metamodeling techniques for optimization under uncertainty, Journal of
Structural & Multidisciplinary Optimization, Vol. 25, No. 2, 2003, pp. 99-116.
12
Jin, R., Chen, W., and Simpson, T. W., Comparative Studies of Metamodeling Techniques under Multiple Modeling
Criteria, Journal of Structural & Multidisciplinary Optimization, Vol. 23, No. 1, 2001, pp. 1-13.
13
Shan, S., and Wang, G., Metamodeling for High Dimensional Simulation-Based Design Problems, Journal of Mechanical
Design, Vol. 132, May 2010.
14
Tarkian, M., Persson, J. A., lvander, J., and Feng, X., Multidisciplinary Design Optimization of Modular Industrial
Robots, Proceedings of the ASME 2011 International Design Engineering Technical Conferences & Computers and information
in Engineering Conference, 2011, pp. 1-10.
15
Martin, J. D. and Simpson, T. W., A methodology to manage uncertainty during system-level conceptual design,
Proceedings of IDETC/CIE 2005 ASME 2005 International Design Engineering Technical Conferences & Computers and
information in Engineering Conference, 2005, pp. 1-11.
16
Martin, J. D. and Simpson, T. W., A Monte Carlo Simulation of the Kriging Model, 10th AIAA/ISSMO Symposium on
Multidiscipilinary Analysis and Optimization, AIAA-004-4483, 2004.
17
Shao, X. Y., Chu, X. Z., Gao. L., and Qui, H. B., A novel method to manage uncertainty for multidisciplinary design and
optimization in conceptual design, Advanced Materials Research, Vol. 44-46, 2008, pp. 225-232.
18
McKay, M.D., Beckman, R.J. and Conover, W.J., A Comparison of Three Methods for Selecting Values of Input
Variables in the Analysis of Output from a Computer Code, Technometrics, Vol. 21. No. 2 , 1979, pp. 239-245.
19
Persson, J. A., and lvander, J., Comparison of Sampling Methods for a Dynamic Pressure Regulator, AIAA-2011-1205,
Proceedings of the 49th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition,
AIAA, Orlando, FA, 4-7 Jan. 2011.
20
lvander J., and Krus P., "Optimizing the Optimization - A Method for Comparison of Optimization Algorithms",
proceedings of the 2nd AIAA Multidisciplinary Design Optimization Specialists Conference, Newport, RI, USA, May 1-4, 2006.
21
Persson, J. A., and lvander, J., A Modified Complex Algorithm Applied to Robust Design Optimization, AIAA-20111205, Proceedings of the 13th AIAA Non-Deterministic Approaches Conference, AIAA, Denver, CO, 4-7 Apr. 2011.
22
Saltelli, A., Chan, K. and Scott, E.M., Sensitivity Analysis, John Wiley & Sons, Chichester, West Sussex, 2000, pp. 10.
23
Steinkellner, S. and Krus, P., "Balancing Uncertainties in Aircraft System Simulation Models", 2nd European Air & Space
Conference CEAS, Manchester, United Kingdom, 2009.

13
American Institute of Aeronautics and Astronautics

24
Forrester, A. I. J., Sbester, A. and Keane A, J., Engineering Design via Surrogate Modelling: A Practical Guide, John
Wiley & Sons Ltd, Chichester, West Sussex, United Kingdom, 2008, Chaps. 1.
25
Simpson, T. W., Peplinski, J., Koch, P. N., and Allen, J. K., Metamodels for Computer-Based Engineering Design: Survey
and Recommendations, Engineering with Computers, Vol. 17, No. 2, 2001, pp. 129-150.
26
Krus, P. and Andersson, J., "Optimizing Optimization for Design Optimization", Proceedings of ASME Design Automation
Conference , Chicago, USA, September 2-6, 2003.
27
Lnn, D., Fyllingen, ., and Nilsson, L., An approach to robust optimization of impact problems using random samples
and metamodelling, International Journal of Impact Engineering, Vol. 37, 2010, pp. 723-734.
28
Aspenberg, D., Jergeus, J., and Nilsson, L., Robust optimization of front members in a full frontal car impact,
Engineering Optimization, 2012, DOI: 10.1080/0305215X.2012.669380.

14
American Institute of Aeronautics and Astronautics