Benchmarking the Differential Evolution with Adaptive

Encoding on Noiseless Functions

Petr Pok

Vclav Klem

Czech Technical University in Prague

FEE, Dept. of Cybernetics
Technick 2, 16627 Prague 6, Czech Republic

Czech Technical University in Prague

FEE, Dept. of Cybernetics
Technick 2, 16627 Prague 6, Czech Republic

[email protected]

[email protected]

ABSTRACT
The dierential evolution (DE) algorithm is equipped with
the recently proposed adaptive encoding (AE) which makes
the algorithm rotationally invariant. The resulting algorithm, DEAE, should exhibit better performance on nonseparable functions. The aim of this article is to assess what
benets the AE has, and what eect it has for other function groups. DEAE is compared against pure DE, an adaptive version of DE (JADE), and an evolutionary strategy
with covariance matrix adaptation (CMA-ES). The results
suggest that AE indeed improves the performance of DE,
particularly on the group of unimodal non-separable functions, but the adaptation of parameters used in JADE is
more protable on average. The use of AE inside JADE is
envisioned.

Categories and Subject Descriptors

G.1.6 [Numerical Analysis]: Optimizationglobal optimization, unconstrained optimization; F.2.1 [Analysis of
Algorithms and Problem Complexity]: Numerical Algorithms and Problems

General Terms
Algorithms

Keywords
Benchmarking, Black-box optimization, Dierential evolution, Evolution strategy, Covariance matrix adaptation, Adaptive encoding

1. INTRODUCTION

invariant, however, the crossover is not. On separable functions, the crossover helps to properly mix the good values of
solution components in the population. On non-separable
functions, however, it mostly only destroys the potentially
good combinations of values generated by the mutation.
There are several possibilities how to overcome the crossover issue for non-separable functions. (1) Turn o the
crossover operator completely. The DE then relies on the
mutation operator only and may have worse performance
on (partially) separable functions. (2) Choose the suitable
operators adaptively. There are several algorithms [8, 1, 10]
able to choose suitable DE operators and their parameters
during the optimization run. For non-separable functions,
they may actually nd that the use of crossover is not profitable at all and may switch it o eectively. (3) Use adaptive encoding. If we were able to perform the crossover in
a suitable coordinate system, we may enjoy the benets of
crossover even for the non-separable functions.
In this article, the last listed possibility is explored. We
chose the recently proposed adaptive encoding (AE) procedure [2] which adapts the coordinate system in a step-wise
manner during the search. The goal of this paper is to assess
how AE aects the DE algorithm, what benets and what
downsides it has, and also to compare the potential of parameter adaptation as used in JADE on the one hand, and
encoding adaptation brought by AE on the other hand.
The rest of this article is organized as follows. Section 2
reviews the DE algorithm, and describes the use of AE inside
DE, i.e. the proposed DEAE algorithm. Section 3 describes
the experiment carried out, together with the COCO benchmarking framework. The results are presented in Sec. 4 and
discussed in Sec. 5. Sec. 6 concludes the paper and points
out some directions for future work.

2. ALGORITHMS

Dierential evolution (DE) [9] is a population-based optimization algorithm, popular thanks to its simplicity and
good results on many practical problems. To create an ospring individual, it uses a mutation operator followed by a
crossover. The mutation operators are usually rotationally

The following paragraphs review the DE algorithm and

the AE procedure, introduce the DEAE algorithm and shortly
describe the reference algorithms used in this paper.
Dierential evolution (DE) [9] is a simple and easy-toimplement optimization algorithm (see the unshaded lines in
Alg. 1). DE mutation operators create the donor individuals
vi as a linear combination of several individuals randomly
chosen from the current population.

Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
GECCO12 Companion, July 711, 2012, Philadelphia, PA, USA.
Copyright 2012 ACM 978-1-4503-1178-6/12/07 ...$10.00.

vi = xbest + F (xr1 xr2 ),

(1)

Eq. 1 describes the so called best/1 mutation operator, a

highly exploitative mutation variant, where F is the mutation factor (a positive number typically chosen from [0.5, 1]).

189

rithm thus does not adapt the coordinate system, does not
adaptively select the operators it uses, but thanks to the
adaptation of CR, it can eectively turn o the crossover.
CMA-ES, evolution strategy with covariance matrix adaptation [5] was chosen for the comparison because the AE
procedure is largely based on this algorithm. The algorithm
samples new candidate solutions from a multivariate Gaussian distribution and adapts its mean and covariance matrix (i.e. it actually uses the adaptation of the coordinate
system). The algorithm CMA-ES used in this paper is a
conventional multistart version.

The crossover creates the ospring ui by taking some solution components from the parent xi and other components
from the donor vi . Eq. (2) describes the binomial crossover.
It creates the ospring individual ui = (ui,1 , . . . , ui,D ) as
follows:

vi,j if rj CRi or j = ji,rand ,
ui,j =
(2)
xi,j otherwise,
where rj is a random number uniformly distributed in [0, 1],
CRi [0, 1] is the crossover probability representing the
average proportion of components the ospring gets from
its donor, and ji,rand is the randomly chosen index of the
solution component surely donated from the donor.
Due to the crossover, DE is biased towards separable functions, and is not rotationally invariant. This bias, however,
can be controlled with the parameter CR. The tuning of CR
is part of many adaptive DE variants which try to nd the
right operators and/or parameter values [8, 1, 10] to make
the resulting algorithm more robust.
DE and adaptive encoding. The adaptive encoding
(AE) framework [2] is a general method that makes an optimization algorithm rotationally invariant. It maintains a
linear transformation of the coordinate systemthe candidate solutions are evaluated in the original space, but the
ospring creation takes place in a dierent space given by
the linear transformation. Alg. 1 shows a simple combination of the basic DE algorithm with AE, i.e. the DEAE
algorithm, rst proposed in [6]. The shaded lines are the
modications needed for AE.

3.

Algorithm 1: DE with Adaptive Encoding

1
2
3

P
Initialize the population P {xi }N
i=1 .

Initialize the transformation matrix B RDD

while stopping criteria not met do
Transform P : P  {xi |xi B1 xi }.

4
5
6
7

for i 1 to N P do
vi mutate(i, P  ) (Eq. 1)
ui crossover(xi , vi ) (Eq. 2)
Transform ospring back: ui Bui .

8
9
10
11
12

if f (ui ) < f (xi ) then

xi u i
end
end
B update(B, x(1) , . . . , x() )

13
14

EXPERIMENT DESIGN

In the experiments, we compare DE, DEAE, JADE, and

CMA-ES. By comparing DE to DEAE, we can assess the
performance boost the DE algorithm can gain using AE. By
comparing DEAE with CMA-ES, we can get some insight if
the sampling process of CMA-ES (drawing points from normal distribution) is more suitable than the sampling process
of DE (using mutation and crossover). The comparison of
DEAE with JADE shall reveal which of the two dierent
types of adaptation is more suitable for which kinds of functions.
Each of the algorithms was run on 15 instances of all
the 24 functions in dimensions 2, 3, 5, 10, 20, and 40.
The evaluations budget was set to 5 104 D for each run.
All algorithms were restarted when they stagnate for more
than
the population diversity measure
D30 generations and
10
1
.
i=1 V ar(Xi ) < 10
D
The multistart CMA-ES algorithm was benchmarked anew
with its default settings using the BBOB 2012 procedure.
For most parameters of DE and JADE, default values
from the literature were used. For DE: the binomial crossover with CR = 0.5, the best mutation strategy with F
U (0.5, 1) (sampled anew each generation). For JADE: initial
CR = 0.5, initial F = 0.5, the parameter of the currentto-pbest mutation is p = 0.1, the archive size |A| = 0.1N P .
The population size was set to N P = 5D for both algorithms after a small systematic study performed on JADE
and DE using the values (3, 4, 5, 6, 8, 10, 15, 20)D. Values of
N P lower than 5D gave erratic behavior even on uni-modal
functions, values larger than 5D wasted evaluations on unimodal functions and did not bring signicant advantages on
multi-modal functions.
The DEAE algorithm inherited the parameters of DE. The
AE part of DEAE uses a learning rate parameter c = 8 chosen after testing the values 1, 4, 8, 10, 15, and 20 (increasing
the learning rate from 1 to 8 brought signicant speedups,
further increase provided questionable advantage only).

end

The forward and backward linear transformations are implemented by matrix multiplication (using the transformation matrix B). The procedure for updating B is crucial
for the algorithm success. We adopted the method derived
from the CMA-ES algorithm (we refer the reader to [2] for
more details).
Reference algorithms. JADE [10] serves as a reference
adaptive DE algorithm. It was chosen because it was reported [10] to have a better performance than other adaptive
DE variants. JADE uses a special mutation strategy called
current-to-pbest, but most importantly it adapts the crossover probability CR and mutation factor F to values which
turned out to be benecial in recent generations. This algo-

4.

RESULTS

Results from experiments according to [3] on the benchmark functions given in [4] are presented in Figures 1, 2 and
3 and in Tables 1 and 2. The expected running time
(ERT), used in the gures and table, depends on a given
target function value, ft = fopt + f , and is computed over
all relevant trials as the number of function evaluations executed during each trial while the best function value did not
reach ft , summed over all trials and divided by the number
of trials that actually reached ft [3, 7]. Statistical signicance is tested with the rank-sum test for a given target ft
(108 as in Figure 1) using, for each trial, either the number

190

also showed quite competitive results. The two forms of

adaptation are based on dierent principles and are in fact
complementary. Implementing the AE procedure inside JADE
may be very protable: JADE may adapt the probability of
applying AE in a similar way it adapts the CR and F parameters. The evaluation of such approach remains a topic
for the future work.

of needed function evaluations to reach ft (inverted and

multiplied by 1), or, if the target was not reached, the best
f -value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial
under consideration.

4.1 CPU Timing Experiments

The timing experiments were carried out with f8 on a
machine with Intel Core 2 Duo processor, 2.4 Ghz, with
4 GB RAM, on Windows 7 64bit in MATLAB R2009b 64bit.
The average time per function evaluation in 2, 3, 5, 10, 20,
40 dimensions was about 52, 35, 21, 12, 8, and 7106 s for
DE, about 70, 45, 28, 16, 9, 10106 s for JADE, and 68,
45, 27, 15, 9, 10 for DEAE, i.e. the cost of AE updates is
negligible.

5.

Acknowledgements
This work was supported by the Ministry of Education,
Youth and Sports of the Czech Republic with the grant
No. MSM6840770012 entitled Transdisciplinary Research
in Biomedical Engineering II.

7.

REFERENCES

[1] J. Brest, S. Greiner, B. Boskovic, M. Mernik, and

V. Zumer. Self-Adapting control parameters in
dierential evolution: A comparative study on
numerical benchmark problems. Evolutionary
Computation, IEEE Transactions on, 10(6):646657,
Dec. 2006.
[2] N. Hansen. Adaptive encoding: How to render search
coordinate system invariant. In G. Rudolph, editor,
Parallel Problem Solving from Nature PPSN X,
volume 5199 of LNCS, pages 205214. Springer, 2008.
[3] N. Hansen, A. Auger, S. Finck, and R. Ros.
Real-parameter black-box optimization benchmarking
2012: Experimental setup. Technical report, INRIA,
2012.
[4] N. Hansen, S. Finck, R. Ros, and A. Auger.
Real-parameter black-box optimization benchmarking
2009: Noiseless functions denitions. Technical Report
RR-6829, INRIA, 2009. Updated February 2010.
[5] N. Hansen and A. Ostermeier. Completely
derandomized self-adaptation in evolution strategies.
Evolutionary Computation, 9(2):159195, 2001.
[6] V. Klems. Dierential evolution with adaptive
encoding. Masters thesis, Czech Technical University
in Prague, 2011. Available online, http://cyber.
felk.cvut.cz/research/theses/papers/177.pdf.
[7] K. Price. Dierential evolution vs. the functions of the
second ICEO. In Proceedings of the IEEE
International Congress on Evolutionary Computation,
pages 153157, 1997.
[8] A. K. Qin and P. N. Suganthan. Self-adaptive
dierential evolution algorithm for numerical
optimization. In Evolutionary Computation, 2005. The
2005 IEEE Congress on, volume 2, pages 17851791
Vol. 2. IEEE, 2005.
[9] R. Storn and K. Price. Dierential evolution a
simple and ecient heuristic for global optimization
over continuous spaces. Journal of Global
Optimization, 11(4):341359, Dec. 1997.
[10] J. Zhang and A. C. Sanderson. JADE: Adaptive
dierential evolution with optional external archive.
Evolutionary Computation, IEEE Transactions on,
13(5):945958, Oct. 2009.

DISCUSSION

Considering the comparison of DEAE and DE, it can be

stated that the application of AE to DE generally helps the
DE algorithm to solve a higher percentage of problems, i.e.
to nd more precise optima of the functions, and to solve
them faster, especially in the group of non-separable unimodal functions (for the ill-conditioned functions, speedup
factors of 10 are observed in 5D, the percentage of solved
problems arose from about 20 % to 100 % in 20-D), which
is an expected result. In case of multi-modal functions, the
dierence is not that large, but DEAE is only seldom worse
than the pure DE. The only exception in this comparison is
the group of separable functions (namely f3 and f4 ), where
the application of AE actually destroys the initially ideal
coordinate system and prevents the DEAE algorithm from
solving these functions.
The comparison of DEAE to CMA-ES reveals that on
the group of unimodal functions, the multistart CMA-ES is
usually faster than DEAE (about 2 to 5 times faster, depending on dimensionality), probably thanks to its much
smaller population. The exception are the functions f3 and
f4 (where neither of the 2 algorithms is competitive), and f7
and f13 (where the DEAE prots from its larger population
size). On the group of multi-modal functions with adequate
structure, DEAE performs better (larger population), while
on the group of weakly structured functions, CMA-ES is
comparable or better (thanks to larger number of restarts).
Comparing DEAE to JADE, the rst observation is that
JADE has an advantage in case of separable functions. For
non-separable unimodal functions, DEAE is (up to 5 times)
faster. For multimodal functions, the results are quite mixed.
In general (and especially in higher dimensions), JADE is expected to solve a larger proportion of functions than DEAE.

6. SUMMARY AND CONCLUSIONS

The search space representation is a key issue when designing a well performing optimization algorithm. In this
work, the AE procedure was applied to the DE algorithm.
The resulting DEAE algorithm was compared with a conventional DE algorithm, JADE, an adaptive version of DE,
and with CMA-ES.
The application of AE signicantly improved the performance of the DE algorithm for moderate and ill-conditioned
unimodal functions, as expected, but also had a positive (although less pronounced) eect on multimodal functions.
JADE (with a dierent kind of adaptation than DEAE)

191

Figure 1: Expected running time (ERT in number of f -evaluations) divided by dimension for target function
value 108 as log10 values versus dimension. Dierent symbols correspond to dierent algorithms given in
the legend of f1 and f24 . Light symbols give the maximum number of function evaluations from the longest
trial divided by dimension. Horizontal lines give linear scaling, slanted dotted lines give quadratic scaling.
Black stars indicate statistically better result compared to all other algorithms with p < 0.01 and Bonferroni
correction number of dimensions (six). Legend: : CMAES, : DE, : JADE, 2: DEAE.

192

separable fcts

moderate fcts
JADE

best 2009

best 2009

DEAE

DE

CMAES

DEAE

JADE

CMAES

DE

ill-conditioned fcts

multi-modal fcts
best 2009

best 2009

DEAE

DEAE

CMAES

DE

JADE

CMAES

DE

JADE

weakly structured multi-modal fcts

all functions
best 2009

best 2009

DEAE

DEAE

DE

DE

JADE

JADE

CMAES

CMAES

Figure 2: Bootstrapped empirical cumulative distribution of the number of objective function evaluations
divided by dimension (FEvals/D) for 50 targets in 10[8..2] for all functions and subgroups in 5-D. The best
2009 line corresponds to the best ERT observed during BBOB 2009 for each single target.

193

separable fcts

moderate fcts
JADE

best 2009

best 2009

DEAE

DE

JADE

CMAES

CMAES

DEAE

DE

ill-conditioned fcts

multi-modal fcts
best 2009

best 2009

DEAE

DEAE

CMAES

JADE

JADE

CMAES

DE

DE

weakly structured multi-modal fcts

all functions
best 2009

best 2009

JADE

JADE

CMAES

DEAE

DEAE

CMAES

DE

DE

Figure 3: Bootstrapped empirical cumulative distribution of the number of objective function evaluations
divided by dimension (FEvals/D) for 50 targets in 10[8..2] for all functions and subgroups in 20-D. The best
2009 line corresponds to the best ERT observed during BBOB 2009 for each single target.

194

fopt 1e1
f1
11
CMAES2.3(2)
DE
5.0(4)
JADE 4.1(3)
DEAE 5.1(6)
fopt 1e1
f2
83
CMAES 16(4)
DE
11(1)
JADE 10(1)
DEAE 12(3)
fopt 1e1
f3
716
CMAES 1.7(2)
DE
1.1(0.4)
JADE 1.1(0.5)
DEAE 2.4(1)

1e0
1e-1
12
12
8.6(3)3 15(5)3
21(8)
39(8)
18(6)
36(7)
23(10)
42(10)
1e0
87
17(3)
13(2)
12(2)
14(2)

1e-3
12
27(5)4
82(8)
77(7)
82(6)

1e-1
88
18(3)
16(2)
15(2)
17(3)

1e-5
12
41(4)4
122(9)
115(10)
124(8)

1e-3
90
20(2)
21(2)
20(2)
22(3)

1e-5
92
21(2)3
27(2)
26(2)
27(3)

1e-7
12
53(6)4

#succ
15/15
15/15
15/15
15/15
15/15

fopt 1e1
f13
132
CMAES 4.5(3)
DE
14(4)
8.2(1)
JADE
6.0(0.8)
DEAE

1e-7
94
22(2)4
31(2)
31(2)
32(3)

#succ
15/15
15/15
15/15
15/15
15/15

fopt 1e1
f14
10
CMAES 1.5(1)
DE
1.8(3)
JADE 0.95(0.8)
DEAE 2.3(3)

164(7)
154(10)
166(8)

1e0
195
4.8(3)
26(11)
12(2)
6.4(1.0)

1e-1
250
5.1(2)
39(11)
12(2)
6.7(0.5)

1e-3
1e-5
1e-7
#succ
1310
1752
2255
15/15
1.5(0.4)3 1.5(0.4)3 1.4(0.4)3 15/15

19(7)
3.1(0.3)
2.0(0.1)

28(9)
2.9(0.2)
2.0(0.1)

45(23)
2.6(0.2)
2.0(0.1)

1e0
41
2.9(1)2
7.2(3)
5.3(2)
6.3(3)

1e-1
1e-3
58
139
3.8(0.8)4 4.2(1)4
12(2)
15(4)
8.9(1)
10(2)
11(2)
9.4(1)

1e-5
251
5.4(1)4

1e-7
#succ
476
15/15
4.4(0.5)3 15/15

1e-1
19369
15(14)
6.0(7)
39(51)
4.8(5)

1e0
1e-1
1622
1637
23(26)
293(319)
1.4(0.2)
2.5(2)
1.6(0.3)
2.2(0.4)
11(11)
70(81)

1e-3
1646
292(304)
2.8(2)
2.7(0.3)
70(78)

1e-5
1650
291(324)
3.1(2)
3.0(0.3)
70(86)

1e-7
1654
290(321)
3.4(2)
3.4(0.3)
70(80)

#succ
15/15
6/15
15/15
15/15
13/15

fopt 1e1
f15
511
CMAES2.1(2)
DE
6.1(5)
JADE 3.7(1)
DEAE 3.9(4)

1e0
9310
5.0(5)
4.3(4)
7.5(14)
4.2(4)

1e0
1e-1
fopt 1e1
f4
809
1633
1688
216(189)

CMAES 2.2(1)
DE
1.2(0.3)
1.7(0.3)2 9.4(14)
2.0(0.5)
3.9(3)
JADE 1.4(0.4)
47(26)
323(304)
DEAE 4.1(3)

1e-3
1817

9.0(13)
4.1(3)
300(275)

1e-5
1886

9.0(13)
4.4(3)
290(294)

1e-7
1903
3e5
9.2(13)
4.7(3)
287(291)

#succ
15/15
0/15
15/15
15/15
6/15

fopt 1e1
f16
120
CMAES2.0(2)
DE
5.1(6)
JADE 2.9(5)
DEAE 3.2(4)

1e0
612
3.6(4)
31(13)
10(5)
26(15)

1e-1
2662
2.7(4)2
17(16)
36(47)
12(9)

1e-3
20073
15(15)
5.9(6)
39(42)
4.6(5)

1e0
10
5.8(2)2
12(4)
20(8)
12(7)

1e-1
10
5.8(2)2
12(3)
21(7)
13(7)

1e-3
10
5.9(2)2
13(3)
21(7)
13(7)

1e-5
10
5.9(2)2
13(3)
21(7)
13(7)

1e-7
10
5.9(2)2
13(3)
21(7)
13(7)

#succ
15/15
15/15
15/15
15/15
15/15

fopt 1e1
f17
5.2
CMAES 4.1(4)
4.1(6)
DE
JADE 3.3(4)
DEAE 3.1(3)

1e0
1e-1
1e-3
215
899
3669
2
2
0.96(0.4) 0.74(0.2) 1.2(2)
3.2(1.0)
2.2(0.5)
2.1(0.5)
2.5(0.7)
1.9(0.3)
1.2(0.2)
2.7(1)
1.5(0.2)
0.85(0.1)

fopt 1e1
f6
114
CMAES2.4(0.7)
5.4(1)
DE
JADE 4.1(2)
DEAE 3.7(2)

1e0
214
2.0(0.5)4
6.6(2)
4.5(1.0)
4.8(1)

1e-1
281
2.2(0.5)4
8.4(2)
5.4(0.9)
5.7(0.9)

1e-3
580
1.7(0.4)4
8.0(2)
4.4(0.5)
4.6(0.5)

1e-5
1038
1.3(0.3)4
6.3(1)
3.6(0.4)
3.8(0.4)

1e-7
1332
1.3(0.2)4
6.7(2)
3.7(0.5)
3.9(0.3)

#succ
15/15
15/15
15/15
15/15
15/15

fopt 1e1
f18
103
CMAES1.2(0.7)
2.4(2)
DE
JADE 1.9(1)

1e0
378
2.2(4)
4.4(2)
3.3(0.7)

fopt 1e1
f7
24
CMAES 5.1(3)
13(11)
DE
8.4(5)
JADE
7.9(5)
DEAE

1e0
324
1.1(0.9)
3.4(2)
2.0(0.7)
1.4(0.4)

1e-1
1171
2.0(2)
1.9(0.6)
1.3(0.4)
0.68(0.1)

1e-3
1572
2.6(3)
2.5(0.7)
1.4(0.3)
0.87(0.2)

1e-5
1572
2.6(3)
2.5(0.7)
1.4(0.3)
0.87(0.2)

1e-7
1597
2.6(3)
2.7(0.8)
1.6(0.3)
0.94(0.2)

#succ
15/15
15/15
15/15
15/15
15/15

fopt 1e1
f8
73
CMAES4.3(4)
9.3(4)
DE
JADE 7.5(3)
DEAE 8.6(3)

1e0
273
3.6(1)2
14(5)
7.3(3)
10(12)

1e-5
1e-7
410
422
4
5.0(0.9)
5.2(1)4
40(7)
49(8)
13(1)
14(1)
11(8)
12(8)

#succ
15/15
15/15
15/15
15/15
15/15

1e-1
336
4.4(1)3
21(8)
10(3)
11(10)

1e-3
391
4.7(1)3
30(8)
12(2)
11(8)

fopt 1e1
f9
35
CMAES 5.5(1)4
22(8)
DE
JADE 14(2)
DEAE 17(5)

1e0
127
7.1(3)2
37(7)
24(7)
13(4)

1e-1
214
6.5(2)2
36(10)
22(5)
11(4)

1e-3
300
5.8(1)3
40(11)
19(5)
10(3)

fopt 1e1
f10
349
CMAES 3.5(0.7)
27(8)
DE
6.1(1)
JADE
3.0(0.4)
DEAE

1e0
500
2.9(0.5)
28(6)
5.1(0.8)
2.5(0.3)

1e-1
574
2.7(0.4)
32(6)
5.0(0.8)
2.6(0.4)

1e-3
1e-5
1e-7
#succ
626
829
880
15/15
2
4
4
2.8(0.3)
2.3(0.2)
2.3(0.2)
15/15
43(8)
42(6)
49(7)
15/15
5.4(0.7)
4.8(0.5)
5.1(0.5)
15/15
3.2(0.2)
3.0(0.3)
3.4(0.2)
15/15

fopt 1e1
f11
143
CMAES 9.1(1)
32(23)
DE
8.1(3)
JADE
DEAE
4.4(1)

1e0
202
7.7(1)
42(19)
9.0(1)
4.9(1)3

1e-1
1e-3
763
1177
2.3(0.3)
1.6(0.2)
16(5)
17(4)
2.8(0.4)
2.3(0.3)
1.7(0.3)3 1.5(0.2)

fopt 1e1
f12
108
CMAES 8.4(9)
103(94)
DE
24(3)
JADE
15(5)
DEAE

1e0
268
5.9(7)
107(85)
13(4)
11(4)

1e-1
371
7.1(7)
117(116)
12(7)
11(6)

1e-3
461
7.9(5)
135(129)
14(7)
13(6)

1e-5
1e-7
#succ
335
369
15/15
4
4
5.8(1.0)
5.7(0.9)
15/15
50(14)
58(17)
15/15
19(4)
18(4)
15/15
10(3)
11(3)
15/15

1e-5
1467
1.4(0.2)
19(4)
2.2(0.2)
1.5(0.2)
1e-5
1303
3.5(2)
63(51)
6.1(3)
5.6(3)

1e-7
#succ
1673
15/15
3
1.3(0.1)
15/15
23(4)
15/15
2.3(0.2)
15/15
1.6(0.2)
15/15
1e-7
1494
3.8(2)
66(52)
6.2(4)
5.6(3)

#succ
15/15
15/15
15/15
15/15
15/15

2.4(1)

fopt 1e1
f19
1
CMAES21(13)
32(28)
DE
JADE 35(26)
DEAE 32(38)
fopt 1e1
f20
16
CMAES 3.7(3)
8.8(6)
DE
4.9(3)
JADE
DEAE 10(7)
fopt 1e1
f21
41
CMAES 1.8(1)
2.7(3)
DE
JADE 1.7(2)
DEAE 3.1(3)

1e-5
11644
5.8(7)
10(9)

3.3(3)

47(12)
8.2(0.5)
5.6(0.6)

#succ
14/15
9/15
13/15
1/15
14/15

1e-7
12095
5.7(7)
10(8)
2e5
3.2(2)

#succ
15/15
15/15
14/15
0/15
15/15

1e-5
1e-7
#succ
6351
7934
15/15
8.5(11)
30(32)
9/15
2.8(2)
3.3(3)
15/15
1.2(0.3)
1.2(0.2)
15/15
0.80(0.1) 0.91(0.1)215/15

1e-3
1e-5
1e-7
9280
10905
12469
11(11)
76(81)
92(103)
2.8(3)
7.7(9)
40(46)
0.60(0.1)2 1.1(0.2)
1.1(0.2)
2

2.8(0.8)
0.50(0.1)
0.43(0.1)3
4 0.53(0.1) 0.69(0.5)
1e0
1e-1
1e-3
1e-5
1e-7
1
242
1.2e5
1.2e5
1.2e5
2230(1651)
327(335)

3e5
4210(2571) 1106(988)
15(16)
15(16)
15(15)
2139(2314)
276(160)

2e5
3555(2186)
388(229)
6.7(7)
6.7(7)
6.6(7)
1e0
851
27(28)
1.9(0.7)
2.3(2)
3.0(2)
1e0
1157
3.1(3)
3.9(6)
1.1(1)
1.2(0.7)

1e-1
1e-3
38111
54470

0.57(0.8) 0.41(0.6)
0.25(0.2)30.25(0.2)
1.6(1)
1.1(1)
1e-1
1674
3.9(5)
3.7(4)
1.2(1)
2.5(3)

1e0
386
21(28)
13(16)
3.4(3)
6.2(7)

1e-1
938
18(31)
17(25)
8.0(11)
10(14)

fopt 1e1
f23
3.0
CMAES 2.9(5)
2.9(4)
DE
JADE 2.4(3)
DEAE 2.8(4)

1e0
518
13(15)
48(59)
37(32)
72(58)

1e-1
14249
6.9(9)
57(61)
8.3(6)
124(132)

1e0
2.2e5
3.1(3)
5.4(6)

3.7(4)

1e-1
6.4e6

15/15
15/15
15/15

1e-7
21359
14(13)
5.6(6)
175(193)
4.4(5)

1e-1
3968
1.5(2)
1.4(0.7)
0.72(0.1)

fopt 1e1
f22
71
CMAES 9.3(10)
5.0(4)
DE
JADE 2.1(1)
DEAE 3.2(2)

fopt 1e1
f24
1622
CMAES 1.6(2)
DE
10(8)
6.5(5)
JADE
6.1(5)
DEAE

1e-5
20769
14(15)
5.7(6)
59(61)
4.5(5)

1e-3
10449
4.5(8)
11(10)

3.2(2)

fopt 1e1
f5
10
CMAES 4.3(2)
DE
8.3(3)
JADE 11(6)
8.0(4)
DEAE

DEAE

37(8)
12(1)
7.9(0.9)

6/15
15/15
15/15

1e-5
54861

0.42(0.6)
0.29(0.2)
1.1(1)

#succ
15/15
3/15
5/15
15/15
15/15
#succ
15/15
0/15
2/15
0/15
4/15

1e-7
55313
3e5
0.42(0.5)
0.33(0.2)
1.1(1)

#succ
14/15
0/15
15/15
15/15
14/15

1e-7
1757
3.9(5)
4.5(4)
8.2(13)
2.9(3)

#succ
14/15
15/15
15/15
15/15
15/15

1e-3
1705
3.9(5)
4.0(4)
2.8(4)
2.7(3)

1e-5
1729
3.9(5)
4.3(4)
5.0(9)
2.9(3)

1e-3
1008
17(28)
17(22)
12(18)
10(13)

1e-5
1040
16(28)
18(22)
16(27)
10(12)

1e-7
1068
16(27)
19(21)
17(30)
10(12)

#succ
14/15
15/15
15/15
15/15
15/15

1e-3
31654
4.9(5)
35(39)
7.6(8)
56(59)

1e-5
33030
4.7(5)
53(56)
7.3(7)
54(57)

1e-7
34256
4.5(5)
105(120)
7.1(6)
52(58)

#succ
15/15
12/15
1/15
10/15
2/15

1e-3
9.6e6

1e-5
1.3e7

1e-7
1.3e7
3e5
2e5
2e5
2e5

#succ
3/15
0/15
0/15
0/15
0/15

Table 1: Expected running time (ERT in number of function evaluations) divided by the respective best ERT
measured during BBOB-2009 (given in the respective rst row) for dierent f values in dimension 5. The
central 80% range divided by two is given in braces. The median number of conducted function evaluations
is additionally given in italics, if ERT(107 ) = . #succ is the number of trials that reached the nal target
fopt + 108 . Best results are printed in bold.

195

fopt 1e1
f1
43
CMAES 7.5(2)4
89(17)
DE
JADE 47(7)
DEAE 75(22)
fopt 1e1
f2
385
CMAES 37(4)
41(3)
DE
JADE 28(1)3
DEAE 53(6)

1e0
43
13(2)4
162(31)
94(8)
144(28)

1e-1
43
20(2)4
241(28)
143(8)
214(24)

1e-3
43
33(2)4
400(27)
240(8)
352(32)

1e0
386
43(3)
50(3)
34(1)4
61(6)

1e-1
387
45(2)
59(3)
39(2)3
69(7)

1e-3
390
47(1)2
76(5)
50(2)
84(7)

1e-5
43
45(3)4
558(30)
340(10)
492(42)
1e-5
391
48(1)4
93(5)
61(3)
101(6)

1e-7
43
58(4)4
717(39)
437(13)
635(42)

#succ
15/15
15/15
15/15
15/15
15/15

fopt 1e1
f13
652
CMAES 6.3(5)4
41(7)
DE
JADE 17(2)
DEAE 22(2)

1e0
2021
5.1(3)2
214(306)
14(5)
10(0.6)

1e-7
393
50(1)4
110(5)
71(4)
116(7)

#succ
15/15
15/15
15/15
15/15
15/15

1e0
1e-1
1e-3
1e-5
1e-7
#succ
fopt 1e1
f14
75
239
304
932
1648
15661
15/15
4
4
4
4
4
4
CMAES 4.2(1)
3.0(0.5)
3.7(0.4)
4.2(0.3)
6.2(0.5)
1.2(0.1) 15/15
55(19)
46(6)
53(6)
502(176)

1e6
0/15
DE
18(1)
23(2)
20(1)
38(24)
62(72)
5/15
JADE 18(6)
DEAE 30(4)
27(3)
37(4)
24(2)
19(1)
2.7(0.2)
15/15

1e-7
#succ fopt 1e1
f15
30378
7651
15/15
1e6
0/15 CMAES37(36)

169(169)
9/15 DE
11(0.2)4 15/15 JADE 39(35)
1e6
0/15 DEAE
1e-5
1e-7
#succ fopt 1e1
f16
1384
7758
1.4e5
9/15
4

1e6
0/15 CMAES 1.9(0.8)

1e6
0/15 DE
4
4
24(8)
JADE
11(0.3)
0.71(0.0) 15/15

1e6
0/15 DEAE

fopt 1e1
1e0
1e-1
1e-3
1e-5
f3
5066
7626
7635
7643
7646

CMAES638(691)
39(10)
67(49)
167(170)
168(170)
168(169)
DE
JADE
6.4(0.3)4 6.0(0.2)4 6.8(0.2)4 8.3(0.2)410(0.2)4

DEAE
fopt 1e1
1e0
1e-1
1e-3
f4
4722
7628
7666
7700

CMAES
30(9)

DE
JADE
8.0(0.4)4 7.0(0.3)4 8.0(0.2)4 10(0.2)4

DEAE

1e0
1.5e5

1e-1
2751
4.5(3)2
702(849)
15(4)
10(0.5)

1e-3
18749
1.9(2)

3.6(0.6)
2.1(0.1)

1e-5
24455
4.6(5)

4.8(0.8)
2.1(0.1)

1e-3
3.2e5

1e-5
4.5e5

1e-7
4.6e5
1e6
1e6
1e6
1e6

#succ
15/15
0/15
0/15
0/15
0/15

1e-1
77015

1e-3
1.9e5

1e-5
2.0e5

1e-7
2.2e5
1e6
1e6
1e6
1e6

#succ
15/15
0/15
0/15
0/15
0/15

1e-7
80472
1e6
1e6
23(25)
90(106)

#succ
15/15
0/15
0/15
2/15
2/15

1e-7
1.5e5
1e6
1e6
1e6
14(14)4

#succ
15/15
0/15
0/15
0/15
4/15

1e-7
6.7e6
1e6
1e6
1e6
1e6

#succ
15/15
0/15
0/15
0/15
0/15

1e-7
5.6e6
1e6
1e6
1e6
1e6

#succ
14/15
0/15
0/15
0/15
0/15

1e0
27265
2.7(2)4

fopt 1e1
f17
63
CMAES 2.8(2)2
26(19)
DE
JADE
7.8(5)
DEAE 19(11)

16(4)
7.4(1)
13(2)

13(4)
4.4(0.7)
8.4(0.9)

1e-3
1e-5
30677
56288
8.1(8)

8.8(9)

2
1.7(0.5)
7.2(9)
2.3(0.3)
37(37)

1e0
1e-1
1e-3
1e-5
1e-7
#succ
fopt 1e1
f6
1296
2343
3413
5220
6728
8409
15/15
4
4
4
4
4
4
CMAES 1.7(0.5)
1.3(0.3)
1.2(0.3)
1.2(0.1)
1.2(0.2)
1.2(0.1)
15/15
53(9)
46(7)
46(8)
50(9)
57(11)
61(12)
15/15
DE
9.4(0.7)
7.8(0.8)
7.3(0.7)
7.2(0.9)
7.4(0.9)
7.4(0.9)
15/15
JADE
18(3)
16(2)
16(2)
17(1)
17(1)
15/15
DEAE 21(5)

1e0
fopt 1e1
f18
621
3972
4
CMAES 1.1(0.4)
2.0(2)
DE
17(4)
16(5)
7.2(2)
4.4(1)
JADE
DEAE 12(4)
6.3(1)

1e-1
19561
4.2(5)
15(7)
1.5(0.4)
2.3(0.3)

1e-3
67569

19(22)
2.5(2)

fopt 1e1
f7
1351
CMAES 1.5(1)3
24(6)
DE
4.8(0.9)
JADE
DEAE
4.7(0.4)

fopt 1e1
f19
1
CMAES 162(70)4
2739(802)
DE
856(206)
JADE
DEAE 1434(466)

1e0
1e-1
1e-3
1e-5
1e-7
#succ
fopt 1e1
f8
2039
3871
4040
4219
4371
4484
15/15
CMAES 3.9(0.8)4 4.3(0.7)4 4.7(0.7)4 4.8(0.7)4 4.9(0.7)4 4.9(0.7)415/15
53(4)
95(6)
106(9)
120(11)
130(11)
138(12)
14/15
DE
16(0.6)
16(0.6)
17(0.6)
17(0.7)
18(0.7)
15/15
JADE 18(1)
21(15)
22(14)
23(14)
23(14)
24(13)
15/15
DEAE 19(3)

fopt 1e1
f20
82
CMAES 5.2(1)4
51(9)
DE
JADE 24(3)
DEAE 39(9)

fopt 1e1
f9
1716
CMAES 4.5(1)4

DE
JADE 36(3)
DEAE 21(3)

1e0
1e-1
1e-3
1e-5
1e-7
#succ fopt 1e1
3102
3277
3455
3594
3727
15/15
f21
561
5.0(0.7)4 5.4(0.6)4 5.5(0.6)4 5.5(0.6)4 5.5(0.6)4 15/15 CMAES 5.4(4)
33(64)
DE

1e6
0/15
7.2(2)
30(3)
32(3)
33(2)
33(2)
33(2)
15/15 JADE
25(17)
26(16)
27(15)
28(15)
29(14)
15/15 DEAE 28(53)
1e1
f
1e0
1e-1
1e-3
1e-5
1e-7
#succ
fopt 1e1
opt
f22
467
f10
7413
8661
10735
14920
17073
17476
15/15
2
CMAES 1.7(0.1)4 1.7(0.2)4 1.6(0.1)4 1.2(0.0)4 1.1(0.0)4 1.1(0.0)4 15/15 CMAES 3.8(5)
23(10)
DE

1e6
0/15
DE
15(4)
15(4)
15(4)
15(3)
18(4)
15/15 JADE 25(45)
JADE 12(5)
DEAE
2.6(0.3)
2.6(0.2)
2.3(0.2)
2.1(0.1)
2.2(0.2)
2.5(0.2)
15/15 DEAE 50(121)
1e0
1e-1
1e-3
1e-5
1e-7
#succ fopt 1e1
fopt 1e1
f23
3.2
f11
1002
2228
6278
9762
12285
14831
15/15
CMAES 11(0.8)
5.3(0.2)
2.0(0.1)3 1.4(0.0)4 1.2(0.0)4 1.0(0.0)415/15 CMAES 2.9(3)
2.4(2)
DE
7377(7977)

1e6
0/15
DE
92(19)
46(8)
18(4)
15(3)
15(3)
15(3)
14/15 JADE 2.0(2)
JADE
DEAE
13(4)
7.4(2)
3.2(0.7)
2.7(0.5)
2.6(0.4)
2.6(0.3) 15/15 DEAE 1.8(2)
1e0
1e-1
fopt 1e1
f12
1042
1938
2740
CMAES 2.2(0.2)4 3.2(3)4
4.0(3)4
99(6)
478(538)
1497(1825)
DE
20(15)
28(15)
JADE 19(4)
DEAE 22(1)
14(3)
14(4)

1e-3
1e-5
1e-7
#succ
4140
12407
13827
15/15
4.0(2)4 1.8(0.8)4 1.9(0.7)415/15

1e6
0/15
28(10)
13(3)
14(3)
15/15
14(3)
6.1(1)
6.4(1)
15/15

fopt
f24
CMAES
DE
JADE
DEAE

1e1
1.3e6

1e0
1e-1
1030
4005
0.96(0.4)41.5(2)

1e0
1e-1
1
3.4e5
4
4.8e4(4e4)

7.0e5(6e5)

1e0
46150
156(172)
2.4(1)
1.2(0.2)3
56(60)

1e0
6541
9.4(10)
45(68)
33(62)
26(33)
1e0
5580
42(90)
92(107)
261(277)
123(179)
1e0
1614
571(707)

131(68)

1e0
7.5e6

#succ
15/15
12/15
0/15
15/15
15/15

1e-1
3.1e5

1e0
1e-1
1e-3
1e-5
1e-7
#succ
fopt 1e1
f5
41
41
41
41
41
41
15/15
4
4
4
4
4
4
CMAES 5.1(1.0)
6.2(0.9)
6.2(0.9)
6.2(0.9)
6.2(0.9)
6.2(0.9)
15/15
DE
27(6)
35(4)
36(6)
36(6)
36(6)
36(6)
15/15
52(7)
54(8)
54(8)
54(8)
54(8)
15/15
JADE 43(9)
33(6)
34(7)
34(7)
34(7)
34(7)
15/15
DEAE 28(7)

1e0
1e-1
1e-3
1e-5
1e-7
#succ
4274
9503
16524
16524
16969
15/15
191(210)
754(842)

1e6
0/15
94(122)
235(265)
256(302)
256(302)
249(296)
3/15
272(351)
686(842)
402(480)
402(482)
391(442)
2/15
3.5(0.5)
2.2(0.3)2 1.9(0.3)4 1.9(0.3)4 1.9(0.2)415/15

1e-7
30201
8.4(9)
1e6
9.0(2)
2.1(0.1)

1e-3
6.2e6

1e-5
1.3e5

6.7(7)4
1e-5
6.7e6

1e-1
1e-3
1e-5
3.1e6
5.5e6
5.6e6

0.79(0.8)

0.46(0.4) 0.85(0.9) 2.7(3)

1e-1
14103
5.5(6)
30(35)
21(38)
19(35)

1e-3
14643
5.3(6)
29(34)
20(34)
18(34)

1e-5
15567
5.0(5)
28(34)
19(34)
17(32)

1e-7
17589
4.5(5)
25(31)
18(31)
16(28)

#succ
15/15
15/15
13/15
13/15
13/15

1e-1
23491
142(156)

638(681)
616(766)

1e-3
24948
133(153)

601(621)
580(671)

1e-5
26847
124(123)

559(577)
539(596)

1e-7
1.3e5
25(26)
1e6
1e6
107(122)

#succ
12/15
4/15
0/15
0/15
1/15

1e-7
8.4e5
1e6
1e6
1e6
1e6

#succ
15/15
0/15
0/15
0/15
0/15

1e-1
67457
63(67)

1e-1
5.2e7

1e-3
4.9e5

1e-3
5.2e7

1e-5
8.1e5

1e-5
5.2e7

1e-7
5.2e7
1e6
1e6
1e6
1e6

#succ
3/15
0/15
0/15
0/15
0/15

Table 2: Expected running time (ERT in number of function evaluations) divided by the respective best ERT
measured during BBOB-2009 (given in the respective rst row) for dierent f values in dimension 20. The
central 80% range divided by two is given in braces. The median number of conducted function evaluations
is additionally given in italics, if ERT(107 ) = . #succ is the number of trials that reached the nal target
fopt + 108 . Best results are printed in bold.

196