European Journal of Operational Research 292 (2021) 397–422

Contents lists available at ScienceDirect

European Journal of Operational Research

journal homepage: www.elsevier.com/locate/ejor

Invited Review

Performance indicators in multiobjective optimization

Charles Audet a,∗, Jean Bigeon b, Dominique Cartier a, Sébastien Le Digabel a,
Ludovic Salomon a,b
a
GERAD and Département de mathématiques et génie industriel, Polytechnique Montréal, C.P. 6079, Succ. Centre-ville, Montréal, Québec H3C 3A7, Canada
b
Univ. Grenoble Alpes, CNRS, Grenoble INP, G-SCOP, Grenoble 38000, France

a r t i c l e i n f o a b s t r a c t

Article history: In recent years, the development of new algorithms for multiobjective optimization has considerably
Received 3 December 2018 grown. A large number of performance indicators has been introduced to measure the quality of Pareto
Accepted 11 November 2020
front approximations produced by these algorithms. In this work, we propose a review of a total
Available online 18 November 2020
of 63 performance indicators partitioned into four groups according to their properties: cardinality,
Keywords: convergence, distribution and spread. Applications of these indicators are presented as well.
Multiobjective optimization © 2020 Elsevier B.V. All rights reserved.
Quality indicators
Performance indicators

1. Introduction ity of an approximation of a Pareto set is non trivial. The ques-

tion is important for the comparison of algorithms, the definition
Since the eighties, a large number of methods has been devel- of stopping criteria, or even the design of multiobjective optimiza-
oped to treat multiobjective optimization problems (e.g Branke, tion methods. According to Zitzler, Deb, and Thiele (20 0 0), a Pareto
Deb, Miettinen, & Slowiński, 2008; Collette & Siarry, 2011; Custó- set approximation should satisfy the following:
dio, Emmerich, & Madeira, 2012; Deb, 2001; Slowinski & Teghem, • The distance between the Pareto front and its representation in
1990). Given that conflicting objectives are provided, the set of the objective space should be minimized.
solutions, the Pareto set, is described as the set of best decision • A good (according to some metric) distribution of the points of
vectors corresponding to the best trade-off points in the objective the corresponding approximated front in the objective space is
space. Knowledge of the Pareto set enables the decision maker to desirable.
visualize the consequences of his/her choices in terms of perfor- • The extent of the corresponding approximated front should
mance for a criterion at the expense of one or other criteria, and be maximized, i.e., for each objective, a wide range of values
to make appropriate decisions. should be covered by the non-dominated points.
Formally, a feasible vector x1 is said to (Pareto)-dominate an-
other feasible vector x2 if x1 is at least as good as x2 for all the To answer this question, many metrics called performance indi-
objectives, and strictly better than x2 for at least one objective. cators Okabe, Jin, and Sendhoff (2003); Zitzler, Knowles, and Thiele
The decision vectors in the feasible set that are not dominated (2008) have been introduced. Performance indicators can be con-
by any other feasible vector are called Pareto optimal. The set of sidered as mappings that assign scores to Pareto front approxima-
non-dominated points in the feasible set is the set of Pareto so- tions.
lutions, whose images (by the objective functions) constitute the Surveys of performance indicators already exist but
Pareto front. they focus only on some specific properties. In (Collette &
In single-objective minimization problems, the quality of a Siarry, 2011, chapter 7), the authors list some performance
given solution is trivial to quantify: the smaller the corresponding indicators to measure the quality of a Pareto front approximation.
objective function value, the better. However, evaluating the qual- In Okabe et al. (2003), an exhaustive survey is conducted on a vast
number of performance indicators which are grouped according
to their properties. Mathematical frameworks to evaluate perfor-
mance indicators are proposed in Knowles and Corne (2002) and
∗
Corresponding author.
Zitzler, Thiele, Laumanns, Fonseca, and da Fonseca (2003) and
E-mail addresses: [email protected], [email protected] (C. Audet),
[email protected] (J. Bigeon), [email protected] (D. Cartier),
additional measures and algorithms are listed in Faulkenberg
[email protected] (S. Le Digabel), [email protected] (L. Sa- and Wiecek (2009). In Cheng, Shi, and Qin (2012), the authors
lomon). review some performance indicators and analyze their drawbacks.

https://doi.org/10.1016/j.ejor.2020.11.016
0377-2217/© 2020 Elsevier B.V. All rights reserved.
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

In Jiang, Ong, Zhang, and Feng (2014), an empirical study fo- • y1 < y2 (y1 strictly dominates y2 ) if and only if y1i < y2i for all
cuses on the correlations between different indicators with their i = 1, 2, . . . , m.
computational complexity on concave and convex Pareto fronts.
Finally, the usage of indicators proposed by the multiobjective In the case when neither y1  y2 nor y2  y1 , y1 and y2 are said
evolutionary optimization community prior to 2013 is analyzed to be incomparable.
in Riquelme, Lücken, and Baran (2015). We can now present the concept of dominance relations for the
Another survey Li and Yao (2019) on the quality evaluation of decision vectors.
solution sets was recently published. It complements the present
study by categorizing more performance indicators, but presents Definition 2 (Dominance relations for decision vectors Audet,
them at a higher level of description. Most of the indicators in Savard, & Zghal, 2008). Given two decision vectors x1 and x2 in
the present work are listed in Li and Yao (2019), but not all of the feasible set X ⊆ Rn , we write:
them (e.g., the ones proposed in Custódio, Madeira, Vaz, & Vicente, • x1 x2 (x1 weakly dominates x2 ) if and only if F (x1 )  F (x2 ).
2011). While Li and Yao (2019) is mainly oriented toward the evo- • x1 ≺ x2 (x1 dominates x2 ) if and only if F (x1 ) ≤ F (x2 ).
lutionary algorithms community, the present work addresses the • x1 ≺≺ x2 (x1 strictly dominates x2 ) if and only if F (x1 ) < F (x2 ).
whole operational research community, and also addresses some • x1 x2 (x1 and x2 are incomparable) if neither x1 weakly dom-
issues in more detail, such as complexity computational costs, as
inates x2 nor x2 weakly dominates x1 .
well as data and performance profiles. The reader is invited to con-
sult their survey as a useful complement. With these relations, we now precise the concept of solution in
The present work is an attempt to propose a survey offering a the multiobjective optimization framework.
panorama on all important aspects of performance indicators con-
trary to the previous surveys, addressed to the whole multiobjec- Definition 3 (Pareto optimality and Pareto solutions Ehrgott,
tive optimization community. This work systematically analyzes 63 2005). The vector x ∈ X is a Pareto-optimal solution if there is no
performance indicators by partitioning them into these four cat- other vector in X that dominates it. The set of Pareto-optimal so-
egories: Cardinality, Convergence, Distribution and spread, Conver- lutions is called the Pareto set, denoted XP , and the image of the
gence and distribution. The use of performance metrics targets four Pareto set is called the Pareto front, denoted YP .
cases: comparison of algorithms, embedding of performance indi- In single-objective optimization, the set of optimal solutions
cators into multiobjective methods, suggestion of stopping crite- is often composed of a singleton. In the multiobjective case, the
ria for multiobjective optimization and identification of promising Pareto front usually contains many elements (an infinity in con-
distribution-based performance indicators. Table 1 lists these indi- tinuous optimization and an exponential number in discrete op-
cators and their category, classifies them based on their properties, timization Ehrgott, 2005). For a problem with m objectives, the
and indicates the section in which they are discussed. This work is Pareto front YP is at most of dimension m − 1. For three objectives,
organized as follows. Section 2 introduces the notations and defi- the Pareto front is a surface, a curve, a point, or combinations of
nitions related to multiobjective optimization and performance in- surfaces, curves and/or points, or the empty set. For two objec-
dicators. Section 3 is the core of this work, and is devoted to clas- tives, the Pareto front can be a curve or a point or a combination
sification of the indicators according to their specific properties. Fi- of curves and/or points, or the empty set. Also, it is interesting to
nally, Section 4 presents some applications. define some bounds on this set.

2. Notations and definitions Definition 4 (Ideal and nadir objective vectors). The ideal objec-
tive vector yI Ehrgott (2005) is defined as the objective vector
To apprehend performance indicators, the first part of this sec- whose components are the solutions of each single-objective prob-
tion describes the main concepts related to multiobjective opti- lem min fi (x ), i = 1, 2, . . . , m. The nadir objective vector yN is de-
x∈X
mization. The second part focuses on the theory of Pareto set ap- fined as the objective vector whose components are the solutions
proximations and performance indicators. of the single-objective problems max fi (x ), i = 1, 2, . . . , m.
x∈XP

2.1. Multiobjective optimization and Pareto dominance For computational reasons, the nadir objective vector is often
approximated by  yN for which the coordinates are defined the fol-
We consider the following continuous multiobjective optimiza- lowing way: let xk, be the solution of the single-objective problem
tion problem: min fk (x ) for k = 1, 2, . . . , m. The ith coordinate of 
yN is given by:
x∈X

min F (x ) = [ f1 (x ) f2 (x ) . . . fm (x )] 
yNi = max fi (xk, ).
x∈X
k=1,2,...,m
where X ⊆ Rn = ∅ is called the feasible set, and fi : Rn → R are m
For a biobjective optimization problem, yN equals  yN . It is not al-
objective functions for i = 1, 2, . . . , m, with m ≥ 2. The image of the
ways the case when m ≥ 3.
feasible set Y = {F (x ) ∈ Rm : x ∈ X } is called the (feasible) objective
An illustration is given in Fig. 1 where the Pareto front is piece-
set. The sets Rn and Rm are respectively denoted as the decision
wise continuous. To simplify the notation, continuous Pareto and
space and the objective space.
piecewise continuous Pareto fronts will be respectively designed
To compare functions objective values, the following cone order
as continuous and discontinuous Pareto fronts.
relation is adopted (Ehrgott, 2005).
Remark. In a multiobjective optimization problem, objectives are
Definition 1 (Dominance relations between objective vec-
not necessarily contradictory, and the set of Pareto solutions may
tors Dächert, Klamroth, Lacour, & Vanderpooten, 2017). Given
be a singleton. In this study, we assume that this is not the case.
two objective vectors y1 and y2 in the objective space Rm , we
write:
2.2. Approximation sets and performance indicators
• y1  y2 (y1 weakly dominates y2 ) if and only if y1i ≤ y2i for all
i = 1, 2, . . . , m. Generally, whether in the context of continuous or discrete op-
• y1 ≤ y2 (y1 dominates y2 ) if and only if y1  y2 and y1 = y2 . timization, it is not possible to find or enumerate all elements

398
 C. Audet, J. Bigeon, D. Cartier et al.
Table 1
A summary of performance indicators. The 9 rightmost columns indicate references where the indicators are presented.

Category Performance indicators Sect. Cheng et al. Collette Knowles Zitzler et al. Okabe et al. Faulkenberg Jiang et al. Riquelme Li and Yao
(2012) and Siarry and Corne (2003) (2003) and Wiecek (2014) et al. (2015) (2019)
(2011) (2002) (2009)
√ √ √ √ √ √ √
Cardinality 3.1 C-metric/Two sets Coverage Zitzler and Thiele (1998) 3.1.5
√ √ √ √ √ √
Error ratio Van Veldhuizen (1999) 3.1.4
√ √ √ √
Generational non dominated vector 3.1.3
generation Van Veldhuizen and Lamont (2000)
√ √ √
Generational non dominated vector generation 3.1.3
ratio Van Veldhuizen and Lamont (2000)
√
Mutual domination rate Martí et al. (2016) 3.1.6
√ √ √
Nondominated vector additional Van Veldhuizen and 3.1.3
Lamont (2000)
√ √ √ √ √ √ √ √
Overall nondominated vector generation Van Veldhuizen 3.1.1
(1999)
√ √ √ √ √ √ √
Overall nondominated vector generation 3.1.2
ratio Van Veldhuizen (1999)
√ √ √
Ratio of non-dominated points by the reference 3.1.5
set Hansen and Jaszkiewicz (1998)
√ √ √
399

Ratio of the reference points Hansen and Jaszkiewicz 3.1.4

(1998)
√
Convergence 3.2 Degree of Approximation Dilettoso et al. (2017) 3.2.7
√ √ √ √
 -family Zitzler et al. (2003) 3.2.6
√ √ √ √ √ √ √ √
Generational distance Van Veldhuizen (1999) 3.2.1
√ √ √ √ √
γ -metric Deb et al. (2000) 3.2.1
√ √ √ √ √ √ √
Maximum Pareto front error Van Veldhuizen (1999) 3.2.4
√ √ √ √ √ √
M1 -metric Zitzler et al. (2000) 3.2.1
√
Progression metric Van Veldhuizen (1999) 3.2.5
√ √ √
Seven points average distance Schott (1995) 3.2.3
√

European Journal of Operational Research 292 (2021) 397–422

Standard deviation from the Generational 3.2.2
distance Van Veldhuizen (1999)
√ √ √ √ √ √
Distribution Cluster Wu and Azarm (2000) 3.3.18
√ √ √ √ √
and spread 3.3 -index Deb et al. (2000) 3.3.2
√ √ √ √
 -index Deb et al. (2000) 3.3.2
√ √ √ √
 spread metric Zhou et al. (2006) 3.3.2
Distribution metric Zheng et al. (2017) 3.3.12
√
Diversity comparison indicator Li et al. (2014) 3.3.18
√
Diversity indicator Cai et al. (2018) 3.3.15
√ √ √ √ √
Entropy metric Farhang-Mehr and Azarm (2004) 3.3.18
(continued on next page)
 C. Audet, J. Bigeon, D. Cartier et al.
Table 1 (continued)

Category Performance indicators Sect. Cheng et al. Collette Knowles Zitzler et al. Okabe et al. Faulkenberg Jiang et al. Riquelme Li and Yao
(2012) and Siarry and Corne (2003) (2003) and Wiecek (2014) et al. (2015) (2019)
(2011) (2002) (2009)
√ √
Evenness Messac and Mattson (2004) 3.3.7
√ √
Extension Meng et al. (2005) 3.3.14
√
 -metric Custódio et al. (2011) 3.3.3
√ √ √ √
Hole Relative Size Collette and Siarry (2005) 3.3.4
√
Laumanns metric Laumanns et al. (2000) 3.3.17
√
Modified Diversity indicator Asafuddoula et al. (2015) 3.3.18
√ √ √ √ √ √
M2 -metric Zitzler et al. (2000) 3.3.5
√ √ √ √ √ √ √ √
M3 -metric Zitzler et al. (2000) 3.3.5
√ √ √ √ √
Number of distinct choices Wu and Azarm (2000) 3.3.18
√ √
Outer diameter Zitzler et al. (2008) 3.3.11
√ √ √ √ √ √
Overall Pareto Spread Wu and Azarm (2000) 3.3.10
Riesz S-Energy Hardin and Saff (2004) 3.3.16
√ √
Sigma diversity metric Mostaghim and Teich (2005) 3.3.18
√ √ √ √ √ √ √ √
Spacing Schott (1995) 3.3.1
√ √
U-measure Leung and Wang (2003) 3.3.9
√
Uniform assessment metric Li et al. (2008) 3.3.13
√ √ √
Uniform distribution Tan et al. (2002) 3.3.5
400

√ √
Uniformity Sayın (2000) 3.3.6
√ √
Uniformity Meng et al. (2005) 3.3.8
√ √
Convergence and Averaged Hausdorff distance Schutze et al. (2012) 3.4.2
distribution 3.4 Cone-based hypervolume Emmerich et al. (2013b) 3.4.7
√
Dominance move Li and Yao (2017) 3.4.6
√ √ √ √ √ √
D-metric/Difference coverage of two sets Zitzler (1999) 3.4.7
√ √ √ √ √
DR -metric Czyzzak and Jaszkiewicz (1998) 3.4.3
√ √ √ √ √
Hyperarea difference Wu and Azarm (2000) 3.4.7
√ √ √ √ √ √ √ √ √
Hypervolume indicator (or S-metric) Zitzler et al. (2000) 3.4.7

European Journal of Operational Research 292 (2021) 397–422

Hypervolume Sharpe-ratio indicator Yevseyeva et al. 3.4.8
(2014)
√ √ √ √
Inverted generational distance Coello and Cortés (2005) 3.4.1
√
Inverted generation distance with non contributed 3.4.1
solutions detection Tian et al. (2016)
√
G-metric Lizarraga-Lizarraga et al. (2008) 3.4.5
Logarithmic hypervolume indicator Friedrich et al. (2011) 3.4.7
√
Modified inverted generational distance Ishibuchi et al. 3.4.3
(2015)
√
Performance comparison indicator Li et al. (2015) 3.4.6
√
p, q-averaged distance Vargas and Bogoya (2018) 3.4.2
√ √ √ √ √ √
R-metric Hansen and Jaszkiewicz (1998) 3.4.4
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Table 2
Comparison relations between Pareto front approximations Zitzler et al. (2003). Note that YN1 ≺≺ YN2 ⇒ YN1 ≺ YN2 ⇒ YN1  YN2 ⇒ YN1 YN2 .
1 2
Relation Objective vectors y and y Pareto front approximations YN1 and YN2

Strictly dominates y1 < y2 y1 is better than y2 in all objectives YN1 ≺≺ YN2 Every y2 ∈ YN2 is strictly dominated
by at least one y1 ∈ YN1
Dominates y1 ≤ y2 y1 is not worse than y2 in all YN1 ≺ YN2 Every y2 ∈ YN2 is dominated by at
objectives and better in at least one least one y1 ∈ YN1
objective
Weakly dominates y1  y2 y1 is not worse than y2 in all YN1 YN2 Every y2 ∈ YN2 is weakly dominated
objectives by at least one y1 ∈ YN1
Is better YN1  YN2 Every y2 ∈ YN2 is weakly dominated
by at least one y1 ∈ YN1 and YN1 = YN2
Is incomparable Neither y1 weakly dominates y2 nor YN1 YN2 Neither YN1 weakly dominates YN2
y2 weakly dominates y1 nor YN1 weakly dominates YN2

These relations orders can be naturally extended to Pareto set

approximations.
Measures are defined on Pareto front approximations. They are
designed as quality indicators or performance indicators Zitzler
et al. (2003).

Definition 6 (Performance indicator Zitzler et al., 2003). A k-ary

performance indicator is a function I : k → R which assigns to
each collection YN1 , YN2 , . . . , YNk of k Pareto front approximations a
real value I (YN1 , YN2 , . . . , YNk ).

A performance indicator may consider several Pareto front ap-

proximations. The most common ones are mappings that take only
one or two Pareto front approximations as arguments. They are
known respectively as unary and binary performance indicators.
Fig. 1. Objective space, Pareto front YP , ideal objective vector yI and nadir objective With such a quality indicator, one can define a relation order be-
vector yN (inspired by Ehrgott, 2005). tween different Pareto front approximations. The interesting indi-
cators are those that capture the Pareto dominance in the objective
space.
of the Pareto front. Hence to solve a multiobjective problem, one
must look for the best discrete representation of the Pareto front. Definition 7 (Monotonicity Zitzler et al., 2008). Assuming a greater
Evaluating the quality of a Pareto front approximation is not triv- indicator value is preferable, a performance indicator I :  → R is
ial. It itself involves several factors such as the closeness to the monotonic if and only if
Pareto front and the coverage in the objective space. Indicators
should capture these factors. To compare multiobjective optimiza- for all YN1 , YN2 ∈ , YN1 YN2 ⇒ I (YN1 ) ≥ I (YN2 ).
tion algorithms, the choice of a good performance indicator is cru-
cial (Knowles & Corne, 2002). Hansen and Jaszkiewicz (1998) are Similarly, assuming a greater indicator value is preferable, a perfor-
the first to introduce a mathematical framework to evaluate the mance indicator I :  → R is strictly monotonic if and only if
performance of indicators according to the comparison of meth-  
for all YN1 , YN2 ∈ , YN1  YN2 ⇒ I YN1 > I (YN2 ).
ods. In their work, they define what can be considered as a good
measure to evaluate the quality of Pareto front. This work has been Once the notion of performance indicator is defined, the defini-
extended in Knowles and Corne (2002), Zitzler et al. (2008, 2003). tion of comparison method can be introduced.
We next define the notion of an approximation.
Definition 8 (Comparison method Zitzler et al., 2003). Let YN1 , YN2 ∈
Definition 5 (Pareto set approximation Zitzler et al. (2008)). A set
of decision vectors XN in the feasible set is called a Pareto set ap-
 be two Pareto front approximations, I = (I1 , I2 , . . . , Ik ) a combi-
nation of performance indicators and E : Rk × Rk → {true, f alse}
proximation if no element of this set is weakly dominated by any
a Boolean function taking two vectors of size k as arguments. If all
other. The image of such a set in the objective space is called a
Ii for i = 1, 2, . . . , k are unary, the comparison method CI,E (YN1 , YN2 ) is
Pareto front approximation denoted YN = F (XN ) ⊆ Rm . The set of all
defined as a Boolean function by the following formula:
Pareto set approximations is denoted  and the set of all Pareto
front approximations is denoted .
 
CI,E (YN1 , YN2 ) = E I (YN1 ), I (YN2 )
By definition, the Pareto front approximation corresponding to
where for all YN ∈ , I (YN ) = (I1 (YN ), I2 (YN ), . . . , Ik (YN )).
a given Pareto set approximation possesses only distinct elements,
If every Ii for i = 1, 2, . . . , k is binary, the comparison method
i.e. two different elements of the Pareto set approximation cannot
CI,E (YN1 , YN2 ) is defined as a Boolean function by
map to the same point in the objective space. Consequently, for all
XN ∈  , |XN | = |YN |.  
CI,E (YN1 , YN2 ) = E I (YN1 , YN2 ), I (YN2 , YN1 )
Remark. We use the terms Pareto set approximation and Pareto
front approximation in the remaining of the paper. where for all YN1 , YN2 ∈ , I (YN1 , YN2 ) = (I1 (YN1 , YN2 ), I2 (YN1 , YN2 ), . . . ,
Ik (YN1 , YN2 )).
Zitzler et al. (2003) propose an extension of the relation order
for objective vectors to Pareto front approximations. They are sum- If I is composed
 of a single indicator I0 , we adopt the notation
marized in Table 2. CI0 ,E YN1 , YN2 instead of CI,E (YN1 , YN2 ).

401
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Informally, a comparison method is a true/false answer to: 3. A classification of performance indicators

“Is a Pareto front approximation better than another one according
to the combination of performance indicators I?” A simple compari- We classify performance indicators into the four following
son method is the following: given an unary performance indicator groups (Jiang et al., 2014; Okabe et al., 2003; Riquelme et al.,
I and two Pareto front approximations YN1 , YN2 ∈ , 2015):
  
if the proposition CI,E (YN1 , YN2 ) = I (YN1 ) > I (YN2 ) is true, then • Cardinality indicators 3.1: Quantify the number of non-
YN1 is said to be better than YN2 according to the indicator I, as- dominated points generated by an algorithm.
suming a greater indicator scalar value is preferable. • Convergence indicators 3.2: Quantify how close a set of non-
dominated points is from the Pareto front in the objective
To compare several Pareto front approximations, one can be in- space.
terested in defining comparison methods that capture the Pareto • Distribution and spread indicators 3.3: Quantify the distribu-
dominance in the objective space, i.e. given two Pareto front ap- tion of a Pareto front approximation. Coverage measures how
proximations YN1 , YN2 ∈  provided by Algorithms 1 and 2, well every region of the objective space is represented. Spread
1 dominates/strictly dominates/is better than YN2 ⇒ focuses on the aspect that points should be far away from
 YN 1weakly  each other (typically this drives them to the boundary). The
CI,E (YN , YN2 ) is true .
More precisely, good comparison methods should always be difference is discussed in Emmerich, Deutz, and Kruisselbrink
compliant with the - relation (Zitzler et al., 2003). On a given (2013a) and Kidd, Lusby, and Larsen (2020). Extent refers to
problem, Algorithm 1 should not be considered as less performant a more precise property, i.e. if the Pareto front approximation
than Algorithm 2 if YN1 is better than YN2 . The following definition contains the extreme points of the Pareto front. Uniformity only
summarizes these points: considers how well the points are equally spaced (Faulkenberg
& Wiecek, 20 09; Sayın, 20 0 0). Spread and uniformity properties
constitute the diversity of a Pareto front approximation (Jiang
Definition 9 (Compatibility and completeness Zitzler et al.,
et al., 2014).
2003). Let R be an arbitrary binary relation on Pareto front
• Convergence and distribution indicators 3.4: Capture both the
approximations (typically, R ∈ {≺, ≺≺, , }). The comparison
properties of convergence and distribution.
method CI,E is denoted as R-compatible if either for any YN1 , YN2
Pareto front approximations, we have:
3.1. Cardinality indicators
(
CI,E YN1 , YN2 )⇒ YN1 R YN2
These indicators focus on the number of non-dominated points
or for any YN1 , YN2 Pareto front approximations, we have: generated by a given algorithm. Some of them require the knowl-
edge of the Pareto front.
CI,E (YN1 , YN2 ) ⇒ YN2 R YN1 .
3.1.1. Overall Non-dominated vector generation (ONVG)
(Van Veldhuizen, 1999)
The comparison method is denoted as R-complete if either for
ONV G returns the number of elements in the Pareto front ap-
any YN1 , YN2 Pareto front approximations, we have:
proximation generated by the algorithm:

YN1 R YN2 ⇒ CI,E (YN1 , YN2 ) for all YN ∈ , ONV G(YN ) = |YN |.
This indicator is to be maximized. Nonetheless, this is not a perti-
or for any YN1 , YN2 Pareto front approximations, we have:
nent measure. For example, consider a Pareto front approximation
YN1 composed of one million non-dominated points and a Pareto
YN2 R YN1 ⇒ CI,E (YN1 , YN2 ). front approximation YN2 with only one point, such as this point
dominates all the other points of YN1 . YN1 outperforms YN2 for this
For any Pareto front approximations YN1 , YN2 ∈ , there are no indicator but YN2 is clearly better than YN1 (Knowles & Corne, 2002).
combination I of unary performance indicators such that YN1 
YN2 ⇔ CI,E (YN1 , YN2 ) (Zitzler et al., 2003). 3.1.2. Overall Non-dominated vector generation ratio (ONVGR)
The mathematical properties of the performance indicators (Van Veldhuizen, 1999)
mentioned in this survey are summarized in Tables 3, 4 and 5 in ONV GR represents the ratio of a number of elements in the
the appendices. Pareto front approximation with respect to a Pareto optimal solu-
tion set. Formally,
Remark. Throughout the rest of the paper, the following nota-
|YN |
tions will be used. A discrete representation of the Pareto front is ONV GR(YN ; YP ) =
denoted by YP ⊆ YP , called the Pareto optimal solution set (Okabe |YP |
et al., 2003). where |YP | is the cardinality of a Pareto optimal solution set and
The Pareto front approximation at iteration k will be denoted by |YN | the number of points of the Pareto front approximation. A
YN (k ). In many cases, the Pareto front is unknown. The user needs higher value is considered to be better. Note that this indicator is
to specify a set of objective vectors in the objective space, called just ONV G (3.1.1) divided by a scalar. Consequently, it suffers from
a reference set and denoted by YR ⊆ Rm . Note that a Pareto front the same drawbacks as the previous indicator.
(approximated or not) contains only feasible objective vectors, i.e.
each element of a Pareto front approximation belongs to Y. It im- 3.1.3. Generational indicators (GNVG, GNVGR and NVA)
plies that if an algorithm does not find any feasible points then (Van Veldhuizen, 1999)
YN (k ) is empty. For the following definitions to apply, we impose GNV G(YN ; k ) (generational non-dominated vector generation)
that the iteration counter k is set to 0 at the iteration where a first returns the number of non-dominated points |YN (k )| generated at
feasible point has been found. iteration k for a given iterative algorithm. GNV GR(YN ; YP , k ) (gen-
erational non-dominated vector generation ratio) is the ratio of

402
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

non-dominated points |YN (k )| generated at iteration k over the car- it gives some unreliable results. Moreover, it does not give an indi-
dinality of YP where YP is a set of points from the Pareto front. cation of “how much” a Pareto front approximation strictly domi-
NVA(YN ; k ) (non-dominated vector addition) represents the varia- nates another.
tion of non-dominated points in the objective space generated be- Similarly to the C-metric, given a reference set YR in the ob-
tween successive iterations. It is given by: jective space, the C2R metric (Ratio of non-dominated points by the
reference set) introduced in Hansen and Jaszkiewicz (1998) is given
NVA(YN ; k ) = |YN (k )| − |YN (k − 1 )| for k > 0.
by:
These indicators can be used to follow the evolution of the gen-
|{y ∈ YN : there does not exist r ∈ YR such that r ≤ y}|
eration of non-dominated points along iterations of a given algo- C2R (YN ; YR ) = .
|YN |
rithm. It seems difficult to use them as a stopping criterion as the
number of non-dominated points can evolve drastically between A higher C2R value is considered to be better. This indicator has the
two iterations. same drawbacks as the C-metric.

3.1.4. Error ratio (ER) (Van Veldhuizen, 1999) 3.1.6. Mutual domination rate (MDR) (Martí, García, Berlanga, &
This indicator considers the number of non-dominated objec- Molina, 2016)
tive vectors which belong to the Pareto front. It is given by the The authors of Martí et al. (2016) use this performance indica-
following formula: tor in combination with a Kalman filter to monitor the progress
of evolutionary algorithms along iterations and thus providing
1 
E (YN ) = 1Rm \YP (y ) a stopping criterion. At a given iteration k, MDR captures how
|YN | y∈YN many non-dominated points at iteration k − 1 are dominated by
non-dominated points generated at iteration k and reciprocally.
where for all y ∈ Rm ,
 Given two Pareto front approximations YN1 and YN2 , the function
0 if y belongs to the Pareto front. (YN1 , YN2 ) returns the set of elements of YN1 that are dominated
1Rm \YP (y ) =
1 otherwise. by at least one element of YN2 . Formally,

The lower the indicator value, the better it is considered. |(YN (k − 1 ), YN (k ) )| |(YN (k ), YN (k − 1 ) )|
MDR(YN ; k ) = −
In Van Veldhuizen (1999), the author does not mention the |YN (k − 1 )| |YN (k )|
presence of rounding errors in their indicator. A suggestion should
where YN (k ) is the Pareto front approximation generated at it-
be to consider an external accuracy parameter  , quantifying the
eration k. If MDR(YN ; k ) = 1, the set of non-dominated points
belonging of an element of the Pareto front approximation to the
at iteration k totally dominates its predecessor at iteration k −
Pareto front with  near to correct rounding errors.
1. If MDR(YN ; k ) = 0, no significant progress has been observed.
This indicator requires the analytical expression of the Pareto
MDR(YN ; k ) = −1 is the worst case, as it results in a total loss of
front. Consequently, an user can only use it on analytical bench-
domination at the current iteration.
mark tests. Moreover, this indicator depends mostly on the cardi-
Cardinality indicators have a main drawback. They fail to quan-
nality of the Pareto front approximation, which can misguide inter-
tify how well-distributed the Pareto front approximation is, or to
pretations. Knowles and Corne (2002) illustrate this drawback with
quantify how it converges during the course of an algorithm.
the following example. Consider two Pareto front approximations.
The first one has 100 elements, one in the Pareto front and the
3.2. Convergence indicators
others close to it. Its error ratio is equal to 0.99. The second one
has only two elements, one in the Pareto front, the other far from
Most of these measures require the knowledge of the Pareto
it. Its ratio is equal to 0.5. It is obvious that the first Pareto front
Front to be evaluated. They capture the degree of proximity be-
approximation is better, even if its error ratio is bad. However, it is
tween a Pareto front and its approximation.
straightforward to compute.
Similarly to the error ratio measure, Hansen and Jaszkiewicz
3.2.1. Generational distance (GD) (Van Veldhuizen, 1999)
(1998) defines the C1R metric (called also ratio of the reference
The GD indicator captures the average distance between each
points). Given a reference set YR (chosen by the user) in the objec-
element of a Pareto front approximation and its closest neighbor
tive space, it is the ratio of the number of objectives vectors found
in a discrete representation of the Pareto front. This indicator is
in YR over the cardinality of the reference set YR . A higher value is
given by the following formula:
desirable.
1
p
1 
3.1.5. C-metric or coverage of two sets (C) (Zitzler, 1999) GD(YN ; YP ) = min y − y1 2 p

Let YN1 and YN2 be two Pareto front approximations. The C-metric |YN | y1 ∈YN
y2 ∈YP

captures the proportion of points in a Pareto front approximation

where |YN | is the number of points in a Pareto front approxima-
YN2 weakly dominated by the Pareto front approximation YN1 . This
tion and YP ⊆ YP a discrete representation of the Pareto front. Gen-
binary performance indicator maps the ordered pair (YN1 , YN2 ) to the
erally, p = 2. In this case, it is similar to the M1 -measure defined
interval [0, 1] and is defined by:
  in Zitzler et al. (20 0 0). When p = 1, it is equivalent to the γ -
 y2 ∈ YN2 , there exists y1 ∈ YN1 such that y1  y2  metric defined in Deb, Agrawal, Pratap, and Meyarivan (20 0 0). For
C( YN1 , YN2 )=   .
Y 2  all these indicators, a lower value is considered to be better.
N
GD is straightforward to compute but very sensitive to the
If C (YN1 , YN2 ) = 1, all the elements of YN2 are dominated by (or number of points found by a given algorithm. In fact, if the algo-
equal to) the elements of YN1 . If C (YN1 , YN2 ) = 0, no element of YN2 is rithm identifies a single point in the Pareto front, the generational
weakly dominated by an element of YN1 . Both orderings have to be distance will equal 0. An algorithm can then miss an entire portion
computed, as C (YN1 , YN2 ) is not always equal to 1 − C (YN2 , YN1 ). of the Pareto front without being penalized by this indicator. This
Knowles and Corne (2002) point out the limits of this measure. measure favors algorithms returning a few non-dominated points
If C (YN1 , YN2 ) = 1 and if C (YN2 , YN1 ) = 1, the two sets are incompara- close to the Pareto front versus those giving a more distributed
ble. If the distribution of the sets or the cardinality is not the same, representation of the Pareto front. As suggested by Collette and

403
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Siarry (2011), it could be used as a stopping criteria. A slight varia- 3.2.5. Progress metric (Pg ) (Van Veldhuizen, 1999)
tion of the generational distance GD(YN (k ), YN (k + 1 ) ) between two This indicator introduced in Bäck (1996) measures the progres-
successive iterations, as long as the algorithm is running, could sion of the Pareto front approximation given by an algorithm to-
mean a convergence towards the Pareto front. It can be applied wards the Pareto front in function of the number of iterations for
on continuous and discontinuous Pareto front approximations. a given objective function i. It is defined by:

 min yi
 y∈YN (0)
3.2.2. Standard deviation from the generational distance (ST DGD) Pg (YN ; i, k ) = ln  .
(Van Veldhuizen, 1999) min yi
y∈YN (k )
It measures the deformation of the Pareto front approxima-
tion according to a discrete representation of the Pareto front. It In Van Veldhuizen (1999), the author modifies this metric to take
is given by the following formula: into account whole Pareto front approximations:

1 
2
GD(YN (0 ); YP )
ST DGD(YN ; YP ) = min y1 − y2 − GD(YN ; YP ) . RPg (YN ; YP , k ) = ln
|YN | 1 y2 ∈YP GD(YN (k ); YP )
y ∈YN
where GD(YN (k ); YP ) is the generational distance (3.2.1) of the
The same critics as with the generational distance apply. Pareto front approximation YN at iteration k.
Pg is not always defined, for example when values of
miny∈YN (0 ) yi or miny∈YN (k ) yi are negative or null. As GD is always
3.2.3. Seven points average distance (SPAD) (Schott, 1995)
positive, RPg is well defined, but it requires the knowledge of the
As it is not practical to obtain the Pareto front, an alternative is
Pareto front.
to use a reference set YR in the objective space. The SPAD indicator
Pg , when it exists, provides an estimation of the speed of con-
defined for biobjective optimization problems uses a reference set
vergence of the associated algorithm. RPg captures only the vari-
composed of seven points:
ations of the generational distance along the number of itera-
 
h l tions. The drawbacks of the generational distance do not apply in
YR = max f1 (x ), max f2 (x ) . this case. Finally, a bad measure of progression does not necessar-
3 x∈X 3 x∈X
0≤h,l≤3
ily mean that an algorithm performs poorly. Some methods less
SPAD captures the average distance of the elements of the refer- deeply explore the objective space, but reach the Pareto front after
ence set YR to their closest neighbor in the Pareto front approxi- a more important number of iterations.
mation. Formally,
3.2.6.  -indicator (I ) (Zitzler et al., 2003)
1  An objective vector y1 ∈ Rm is  -dominating, for  > 0, an ob-
SPAD(YN ; YR ) = min y − r .
|YR | y∈YN
r∈YR jective vector y2 ∈ Rm if:

A lower value is considered to be better. for all i = 1, 2, . . . , m, y1i ≤  y2i .

This indicator raises same critics as above. Note that the com- The multiplicative  -indicator for two Pareto front approximations
putational cost to solve the single-objective problems maxx∈X fi (x ) YN1 and YN2 is defined as the minimum factor one has to multiply
for i = 1, 2 is not negligible. Also, the points in the reference set a Pareto front approximation to make it weakly dominate another
can fail to capture the whole form of the Pareto front. Its limita- one. It is given by
tion to two objectives is also an inconvenient. Nonetheless, it does
not require the knowledge of Pareto front. I× (YN1 , YN2 )

= inf y2 ∈ YN2 : ∃y1 ∈ YN1 such that y1 is  -dominating y2 .
 >0
3.2.4. Maximum Pareto front error (MP F E) (Van Veldhuizen, 1999)
It can be calculated the following way:
This indicator defined in Van Veldhuizen (1999) is another mea-
sure that evaluates the distance between a discrete representation y1i
of the Pareto front and the Pareto front approximation obtained I× (YN1 , YN2 ) = max min max .
y2 ∈YN2 y1 ∈YN1 1≤i≤m y2i
by a given algorithm. It corresponds to the largest minimal dis-
tance between elements of the Pareto front approximation and Given a discrete representation of the Pareto front YP , the unary
their closest neighbors belonging to the Pareto front. This indica- metric I× (YN ; YP ) (with a semicolon) is defined as I× (YP , YN ).
tor is to be minimized. It is expressed with the following formula Similarly, Zitzler et al. (2003) define an additive  -indicator
(generally, p = 2 ): based on the following additive  -domination. It is said that an
objective vector y1 is additively  -dominating an objective vector
1
m 
  p y2 for  > 0 if for all i = 1, 2, . . . , m, y1i ≤  + y2i . This indicator is
MP F E (YN ; YP ) = max min y1i − y2i  p . then calculated by:
y1 ∈YN y2 ∈YP
i=1
I+ (YN1 , YN2 ) = max min max y1i − y2i .
y2 ∈YN2 y1 ∈YN1 1≤i≤m
When YN ⊆ YP , the value MP F E (YN ; YP ) is zero. The indicator is
not relevant, as pointed out in Knowles and Corne (2002). Con- A value inferior to 1 (respectively 0) for the binary multiplica-
sider two Pareto fronts approximations in which the first possesses tive  -indicator (respectively additive  -indicator) indicates that YN1
only one element in the Pareto front and the second has ten ele- weakly dominates YN2 .
ments: nine of them belong to the Pareto front and one is at some Binary additive and multiplicative  -indicators possess desirable
positive distance from it. As MP F E considers only largest minimal properties. They are Pareto compliant and compatible (Zitzler et al.,
distances, it favors the first Pareto front approximation. But the 2003), do not require the knowledge of the Pareto front, and repre-
second is clearly better. However, it is straightforward and cheap sent natural extensions for approximation schemes in optimization
to compute and may be used on continuous and discontinuous theory. However, the main problem with the  -indicator is that
problems. its evaluation value involves only one particular element in each

404
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

 
Pareto front approximation, which can misguide quality compar- where d1 y j , YN \ {y j } = miny∈Y \{y j } y − y j 1 is the l1 distance
N
ison between different Pareto front approximations. Furthermore,
of y j ∈ YN to the set YN \ {y j } and d¯ is the mean of all d1 (y j , YN \
the  -indicator focuses only on one objective when comparing dif-
{y j } ) for j = 1, 2, . . . , |YN |.
ferent objective vectors, as noticed in Li and Yao (2017). For ex-
This method cannot account for holes in the Pareto front ap-
ample, consider y1 = (0, 1, 1 ) and y2 = (1, 0, 0 ) in a tri-objective
proximation as it takes into account the distance between an ob-
maximization problem. Although y1 performs better than y2 in two
jective vector and its closest neighbor. The major issue with this
different objectives, the additive  -indicators are identical:
    indicator is it gives some limited information when points given
I+ {y1 }, {y2 } = I+ {y2 }, {y1 } . by the algorithm are clearly separated, but spread into multiple
groups. On the contrary, it is straightforward to compute.
On the contrary, it is straightforward to compute. It can be used
for continuous and discontinuous approximations of Pareto fronts.

3.2.7. Degree of approximation (DOA) (Dilettoso, Rizzo, & Salerno, 3.3.2. Delta indexes ( ,  and  ) (Deb et al., 2000; Zhou, Jin,
2017) Zhang, Sendhoff, & Tsang, 2006)
By taking into account the dominance relation in the objective Deb et al. (20 0 0) introduce the  index for biobjective prob-
space, DOA captures an average degree of proximity from a discrete lems, which captures the variation of distance between consecu-
representation of the Pareto front to a Pareto front approximation. tive elements of the Pareto front approximation into the biobjec-
A lower value is considered to be better. This indicator is proved tive space. Formally,
to be ≺-complete (see Definition 9). It aims to compare algorithms    
when the Pareto fronts are known.
|Y
N |−1 dc y j , YN \ {y j } − d¯c 
 (Y )=
Given y2 an objective vector belonging to YP , the set D (y2 , YN ) N
|YN | − 1
j=1
in the objective space is defined as the subset of points belonging
to the Pareto front approximation YN dominated by the objective  
where dc y j , YN \ {y j } is the Euclidean distance between consec-
vector y2 . The minimum Euclidean distance between y2 ∈ YP and
D (y2 , YN ) (which may be empty) is computed with utive elements of the Pareto front approximation YN , and d¯c the
 
 mean of the dc y j , YN \ {y j } for j = 1, 2, . . . , |YN | − 1. As this indi-
min 2
y −y 1
if |D (y , YN )| > 0
2
cator considers Euclidean distances between consecutive objective
d (y2 , YN ) = y1 ∈D (y2 ,YN )
∞ if |D (y2 , YN )| = 0. vectors, it can be misleading if the Pareto front approximation is
  piecewise continuous. The  index does not generalize to more
Similarly, d + y2 , YN \ D (y2 , YN ) is defined for y2 ∈ YP by consider- than 2 objectives, as it uses lexicographic order
  in the biobjective
ing the set of points of YN that do not belong to D (y2 , YN ) as: objective space to compute the dc y j , YN \ {y j } . In addition, it does
  not consider the extent of the Pareto front approximation, i.e. dis-
d+ y2 , YN \ D (y2 , YN )
 tances between extreme points of the Pareto front.
min ( y1 − y2 )+ if |YN \ D (y2 , YN )| > 0 The  index is an indicator derived from the  index to take
= y1 ∈YN \D (y2 ,YN ) into account the extent of the Pareto front approximation for biob-
∞ if |YN \ D (y2 , YN )| = 0 jective problems:
where (y1 − y2 )+ = (max(0, y1i − y2i ))i=1,2,...,m . 2 |YN |−1    
The DOA indicator is finally given by min yi, − y + dc y j , YN \ {y j } − d¯c 
i=1 y∈YN j=1
1     (YN ; YP ) = 2
DOA(YN ; YP ) = min d (y2 , YN ), d+ y2 , YN \ D (y2 , YN ) . min yk, − y + (|YN | − 1 )d¯c
|YP | 2 y ∈YP
k=1 y∈YN

The value of DOA does not depend on the number of points of where miny∈YN yi, − y for i = 1, 2 are the Euclidean distances be-
YP , i.e. if |YP |  |YN | (Dilettoso et al., 2017). In fact, this indica- tween the extreme solutions of the Pareto front (i.e. yi, = F (xi, ) ∈
tor partitions YN into subsets in which each element is domi- YP where xi, is solution to the ith single-objective problem) and
nated by a point y ∈ YP . Its computational cost is quite low (in the boundary solutions of the Pareto front approximation. The
O (m |YN | × |YP | )). It can be used for discontinuous and continuous other notations remain the same as before. This indicator requires
approximations of Pareto fronts. the resolution of each single-objective optimization problem. This
indicator is extended to Pareto fronts with more than two objec-
3.3. Distribution and spread indicators tives by Zhou et al. (2006) to the generalized  -index:
m |YN |  
According to Custódio et al. (2011), “the spread metrics try to
d2 (yi, , YN ) + d2 (y j , YN \ {y j } ) − d¯2 
measure the extents of the spread achieved in a computed Pareto front i=1 j=1
 (YN ; YP ) = m
approximation”. They are not really useful to evaluate the conver- d2 (yi, , YN ) + |YN | d¯2
gence of an algorithm, or at comparing algorithms, but rather i=1

the distribution of the points along Pareto front approximations.

where d2 (yi, , YN ) = miny∈YN yi, − y with yi, = F (xi, ) ∈ YP the
They only make sense when the Pareto set is composed of several
extreme objective vector corresponding to xi, solution to the ith
solutions corresponding to distinct objective vectors.
single-objective problem and d2 (y j , YN \ {y j } ) = miny∈Y \{y j } y j −
N
3.3.1. Spacing (SP ) (Schott, 1995) y the minimal Euclidean distance between two points of the
The SP indicator captures the variation of the distance between Pareto front approximation. d¯2 is the mean of all d2 (y j , YN \ {y j } )
elements of a Pareto front approximation. A lower value is consid- for j = 1, 2, . . . , |YN |. As it considers the shortest distances between
ered to be better. This indicator is computed with elements of the Pareto front approximation, the  index suffers
 from the same drawbacks as the spacing metric. Moreover, it re-

 1 
N
 |Y |  2 quires the knowledge of the extreme solutions of the Pareto front.
SP (YN ) =  d¯ − d1 y j , YN \ {y j } For these three indicators, a lower value is considered to be
|YN | − 1 j=1 better.

405
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Fig. 2. Illustration of the  metric for a biobjective problem (inspired by Custódio et al., 2011).

3.3.3. Two measures proposed by Custódio et al. (2011) ( and ) 3.3.4. Hole relative size (HRS) (Collette & Siarry, 2005)
Given a Pareto set approximation XN = {x1 , x2 , . . . , xN } to which This indicator identifies the largest hole in a Pareto front ap-
two additional “extreme” points indexed by 0 and N + 1 are proximation for a biobjective problem. It is given by
added, for each objective i for i = 1, 2, . . . , m, elements x j for j =
0, 1, . . . , N + 1 of the Pareto set approximation are sorted so that,
HRS(YN ) = (1/d¯) max dj
j=1,2,...,|YN |−1

fi (x0 ) ≤ fi (x1 ) ≤ fi (x2 ) ≤ . . . ≤ fi (xN+1 ). where YN is a Pareto front approximation whose elements have
been sorted in ascendant order according to the first objective,
Custódio et al. (2011) introduce the following metric  > 0 defined d j = y j − y j+1 2 is the l2 distance between the two adjacent ob-
by: jective vectors y j ∈ YN and y j+1 ∈ YN and d¯ the mean of all d j for
j = 1, 2, . . . , |YN | − 1.
(YN ) = max max δi, j A lower indicator value is desirable. As it takes into account
i∈{1,2,...,m} j∈{0,1,...,N}
holes in the objective space, this indicator is more adapted to con-
tinuous Pareto front approximations.
where δi, j = fi (x j+1 ) − fi (x j ) and YN = F (XN ). When considering a
biobjective problem (m = 2), the metric reduces to consider the
maximum distance in the infinity norm between two consecutive 3.3.5. Zitzler’s metrics M2 and M3 (Zitzler et al., 2000)
points in the Pareto front approximation as it is shown in Fig. 2. The M2 metric returns a value in the interval [0; |YN |] where YN
To take into account the extent of the Pareto front approxima- is the Pareto front approximation. It reflects the number of sub-
tion, the authors of Custódio et al. (2011) define the following in- sets of the Pareto front approximation YN of a certain radius (σ ). A
dicator by higher value is considered to be better. Its expression is given by
 N−1   1 
δi,0 + δi,N + δi, j − δi  M2 (YN ; σ ) = |{y1 ∈ YN , y2 − y1 > σ }|.
(YN ) = max
j=1 |YN | − 1 2y ∈YN
i∈{1,2,...,m} δi,0 + δi,N + (N − 1 )δi
If M2 (YN ; σ ) = |YN |, it means that for each objective vector, no
other objective vector within the distance σ can be found. It is
where δi , for i = 1, 2, . . . , m, is the mean of all distances δi, j for j =
straightforward to compute but it can be difficult to interpret.
1, 2 . . . , N − 1.
The authors of Tan, Lee, and Khor (2002) introduce the Uniform
The  and  indicators do not use the closest distance between
distribution indicator, based too on the search of niches of size σ ,
two points in the objective space. Consequently, they do not have
given by
the same drawbacks as the spacing metric. However, the δi, j dis-
tance captures holes in the Pareto front if this one is piecewise 1
UD(YN ; σ ) =
discontinuous. For these two indicators, a lower value is desirable. 1 + Dnc (YN , σ )
These two metrics are more adapted to continuous Pareto front ap-
proximations. where Dnc (YN , σ ) is the standard deviation of the number of niches
around all the points of the Pareto front approximation YN defined
Remark. The authors of Custódio et al. (2011) suggest two ways as
 ⎛ ⎞
to compute extreme points x0 and xN+1 . For benchmark tests, the 
 |YN |
 |YN |
2
Pareto front is known and extreme points correspond to the ones  1 1 
of the Pareto set. Otherwise, the  and  indicators use the ex- Dnc (YN , σ ) =  ⎝ nc (y j , σ ) − nc (yl , σ ) ⎠
|YN | − 1 j=1
|YN | l=1
treme points of the Pareto set approximation XN .

406
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

 
with nc (y j , σ ) = {y ∈ YN , y − y j < σ } − 1. The UD indicator is to 2. For both sets, compute the average distances dl2 (YN1 ) and
be minimized.
dl2 (YN2 ) between neighbor points given by
Finally, the M3 metric defined by Zitzler et al. (20 0 0), considers
the extent of the front: 1 
 dl2 (YN ) = dl2 (y, YN \ {y} ).

m |YN | y∈Y
M3 (YN ) =
N
max max yj −y .
j∈{1,2,...,|YN |} y∈YN \{y j }
i=1 3. For each set, compute the spacing measures SP l (YN1 ) and
A higher value is considered to be better. The metric only takes M3 SP l (YN2 ) given by
into account the extremal points of the computed Pareto front ap- 
   2
proximation. Consequently, it is sufficient for two different algo- 
rithms to have the same extremal points to be considered as equiv-   1 − ψ dl2 (y, YN \ {y} ), dl2 (YN )
SP l (YN ) = 
alent according to this metric. It can be used on continuous and
y∈Y
|YN | − 1
discontinuous approximations of Pareto fronts as it only gives in- N

formation on the extent of the Pareto front. a

if a > b
with ψ (a, b) = b
3.3.6. Uniformity (δ ) (Sayın, 2000) b
otherwise.
a
This is the minimal distance between two points of the Pareto 4. If SP l (YN1 ) < SP l (YN2 ), then YN1 has better uniformity
front approximation. This measure is straightforward to compute than YN2 and reciprocally. If SP l (YN1 ) = SP l (YN2 ) and l ≥
and easy to understand. However, it does not really provide perti- min (|YN1 | − 1, |YN2 | − 1 ) then YN1 has the same uniformity as
nent information on the distribution of the points along the Pareto YN2 . Else if SP l (YN1 ) = SP l (YN2 ) and l < min (|YN1 | − 1, |YN2 | − 1 ),
front approximation. then increment l by 1, and recompute the previous steps
by removing the smallest distance dl2 (y1 , YN1 \ {y1 } ) and
3.3.7. Evenness (ξ ) (Messac & Mattson, 2004) dl2 (y2 , YN2 \ {y2 } ) until the end.
The ξ -evenness indicator captures the uniformity quality of
a Pareto front approximation by integrating distance values be- The value of the binary uniformity indicator is difficult to inter-
tween its elements into a coefficient of variation. More specifi- pret but can be computed easily. It does not take into account the
cally, two scalar values are associated to each element y ∈ YN of extreme points of the Pareto front.
the Pareto front approximation. dl (y, YN \ {y} ) is the minimum Eu-
clidean distance between objective vector y and its closest neigh-
bor in the objective space. du (y, YN \ {y} ) is the maximal Euclidean 3.3.9. U-measure (U) (Leung & Wang, 2003)
distance between an element y ∈ YN and another element of YN This indicator measures the uniformity of a Pareto front approx-
such that no other point of YN is within the (hyper)sphere of di- imation based on distance between its elements according to each
ameter du (y, YN \ {y} ). ξ is then defined as objective. For each objective vector in the Pareto front approxima-
σ tion YN , Euclidean distance to their nearest neighbors with respect
ξ (YN ) = D to each objective axis is determined, as well as distances of refer-
D
ence objectives (objective vectors corresponding to solutions of the
where σD and D  are respectively the standard deviation and the single-objective optimization problems or determined by the user)
mean of the set of minimum distances dl (y, YN \ {y} ) and maxi- to their nearest neighbors. Let χ be the set of distances between
mum diameters du (y, YN \ {y} ) for each element y of YN . The clos- nearest neighbors and χ̄ 0 the set of distances between reference
est ξ is to 0, the better the uniformity is. objective vectors and their closest neighbor. Small variability in
It can be considered as a coefficient of variation. It is straight- the set χ would reflect uniform distribution, and values close to
forward to compute. In the case of continuous Pareto front, it can 0 for the set χ̄ 0 would reflect good coverage properties. For ease
account for holes in the Pareto front approximation. of computation, each distance in χ̄ 0 is summed up with the aver-
Reference Ghosh and Chakraborty (2015) also defines the even- age value of the distances in χ , resulting in the new set χ̄ . The
ness as U-measure captures the discrepancy among the scalar elements of
max min y1 − y2 the set χ̄ and is given by
E (YN ) =
y1 ∈YN y2 ∈YN \{y1 }
.  
1   d 
min min y1 − y2 U (YN ) =  − 1
y1 ∈YN y2 ∈YN \{y1 } |χ̄ |  dideal
d ∈χ̄
The lower the value, the better the distribution with a lower bound

E (YN ) = 1. where dideal = |χ̄1 | d .
d ∈χ̄
d
− 1 can be interpreted as the percentage deviation from
3.3.8. Binary uniformity (SPl ) (Meng, Zhang, & Liu, 2005) dideal
Contrary to others indicators, this indicator aims to compare the ideal distance if it is multiplied by 100%. The U-measure is
the uniformity of two Pareto front approximations. This indicator then the mean of this ratio along all elements of the set χ̄ . A small
is inspired by the wavelet theory. U indicator value can be interpreted as a better uniformity.
Let YN1 and YN2 be two Pareto front approximations. The algo- It attempts to quantify the uniformity of found points along the
rithm is decomposed in several steps: Pareto front approximation.
Set l = 1. The same problems as for the previous indicators can be raised.
Especially, the formula works only if there are several points.
1. For each element y1 ∈ YN1 and y2 ∈ YN2 , compute the respec- Moreover, this indicator can take time to compute when comput-
tive distances to their closest neighbor dl2 (y1 , YN1 \ {y1 } ) and ing the minimal distances. As for the spacing metric (3.3.1), this
dl2 (y2 , YN2 \ {y2 } ) given by last one does not account for holes in the Pareto front approxima-
tion as it takes only into account closest neighbors. It is then more
dl2 (y, YN \ {y} ) = v min y − yv . pertinent on continuous Pareto front approximations.
y ∈YN \{y}

407
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

|YN |−1  j 2 |YN |−1 j

with σi = |Y 1|−2 di − di , μi = |Y 1|−1 di where
N j=1 N j=1
|YN | is the number of non-dominated objective vectors, yI and yN
j
are respectively the ideal and nadir objective vectors. di is the dis-
tance of the jth interval between two adjacent solutions corre-
sponding to the ith objective, σi and μi are the standard deviation
σ
and mean of the distances relative to the ith objective, and μi is
i
the coefficient of variance relative to the ith objective.
A smaller DM indicates better distributed solutions. It takes into
account the extent and distribution of the points along the Pareto
Fig. 3. An example showing the weaknesses of the spacing metric (inspired
front approximation. However, it requires the nadir and ideal ob-
by Zheng et al., 2017): the spacing metric ignores the gap drawn in dashed lines. jective vectors, which may be computationally expensive. As it ac-
counts for holes, this indicator is more relevant for continuous
Pareto front approximations.
3.3.10. Overall Pareto spread (OS) (Wu & Azarm, 2000)
This indicator only captures the extent of the front covered by 3.3.13. Uniform assessment metric (ID ) (Li, Zheng, & Xiao, 2008)
the Pareto front approximation. The larger the better it is. It is The ID indicator measures the variation of distances between el-
given by ements of a Pareto front approximation based on the construction
 
  of a minimum spanning tree. The indicator value is comprised be-
max y − min y  tween 0 and 1. The closest to 1, the better. Let YN be a Pareto front

 y∈YN
m i
y∈YN
i

OS(YN ) =   approximation such that |YN | > 2. The computation of this indica-
i=1
yi I − yi M  tor is decomposed into several steps:

where  yM is an approximation of the maximum objective vector 1. A minimum spanning tree TG covering all the elements of
(objective vector composed of the maxima of each single-objective YN based on the Euclidean distance in the objective space is
optimization problem assuming they exist) or the maximum ob- built.
jective vector and yI an approximation of the ideal objective vector 2. Each element y ∈ YN has at least one neighbor in the span-
or the ideal objective vector. ning set, i.e a vertex adjacent to y. Let NTG (y ) be the set of
This is an indicator for which the values are among the values 0 adjacent vertices to y in the spanning tree TG .
and 1. The maximum and ideal objective vectors need to be com- For each yv ∈ NTG (y ), we define a “neighborhood” (Li et al.,
puted for more precise results, resulting in 2 m single-objective 2008)
problems to solve. The indicator does not take into account the 
Nyv (y ) = y2 ∈ YN , y2 − y ≤ yv − y
distribution of points along the Pareto front approximation.
which corresponds to the subset of YN contained in the
3.3.11. Outer diameter (IOD ) (Zitzler et al., 2008) closed ball of radius yv − y and centered in y. Note that
Analogously to the overall Pareto spread metric, the outer di- {y, yv } ∈ Nyv (y ). The neighborhoods that contain only two el-
ameter indicator returns the maximum distance along all objec- ements, i.e. y and yv , are not considered.
tive dimensions pondered by weights w ∈ Rm 3. For all y ∈ YN and yv ∈ NTG (y ), a distribution relation is de-
+ chosen by the user.
A higher indicator value is desirable. It is given by: fined by
⎧
⎨0 if |Nyv (y )| = 2,
IOD (YN ) = max wi max yi − min yi .  y − y2
ψ (y, yv ) = otherwise.
1≤i≤m y∈YN y∈YN
⎩ y − yv
y2 ∈Nyv (y )\{y}
The weights can be used to impose an order on criteria im-
portance relatively to the modeling of a specific problem but it 4. There are 2|YN | − 2 neighborhoods. Among them, Nr corre-
is not mandatory. Although this indicator is cheap to compute, it sponds to the number of neighborhoods that only contain
only takes into account the extent of the Pareto front approxima- two elements. The uniform assessment metric is then de-
tion. By the way, it can result in an information loss of the extent fined by
of the Pareto front approximation, as it focuses only on the largest 1  
distance along a single dimension. ID (YN ) = ψ (y, yv )
2|YN | − Nr − 2
y∈YN yv ∈NT (y )
G

3.3.12. Distribution metric (DM) (Zheng, Yang, Xu, & Hu, 2017) which corresponds to the mean of the distribution relation
This indicator introduced by Zheng et al. (2017) aims to cor- for neighborhoods containing more than two elements.
rect several drawbacks of the spacing measure (Schott, 1995) and
add some information about the extent of the Pareto front. As it is This indicator does not require external parameters. Due to the
mentioned, the “spacing metric does not adopt normalized distance, definition of the neighborhood, it takes into account holes in the
which may result in a bias conclusion, especially when the orders of Pareto front. Indeed, contrary to the spacing metric, it does not
magnitudes of the objectives differ considerably”. Moreover, it cannot consider only closest distances between objective vectors.
account for holes in the Pareto front, as it considers only closest
neighbors. An example pointing out the drawbacks of the spacing 3.3.14. Extension measure (EX) (Meng et al., 2005)
metric (3.3.1) is given in Fig. 3. This indicator aims to measure the extent of the Pareto front
The DM indicator is given by approximation. It is given by
  
m   yI − yN 
1  σi 1 
m
DM (YN ) =
i i
EX (YN ; YP ) = d2 (yi, , YN )2
|YN | μii=1
max yi − min yi m
y∈YN y∈YN i=1

408
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

where d2 (yi, , YN ) is the minimal Euclidean distance between the

objective vector corresponding to the solution to the ith single-
objective problem and the set of non-dominated points obtained
by a given algorithm in the objective space.
This indicator requires the resolution of m single-objective op-
timization problems. It penalizes well-distributed Pareto front ap-
proximations neglecting the extreme values. It is straightforward
to compute.

3.3.15. Diversity indicator based on reference vectors (DIR) (Cai, Sun,

& Fan, 2018)
Fig. 4. The intersection of H (S ) and D (S ) for a biobjective minimization problem.
As its name indicates, this indicator uses reference vectors in
the objective space to measure the diversity of a Pareto front
approximation. The lower this indicator is, the better. Let YR =
be the set of objective vectors dominated by y. Given a Pareto front
{r1 , r2 , . . . , r|YR | } be a set of uniformly generated reference vec-
approximation YN , D (YN ) is designed as the dominated space by
tors in Rm . For each element of a Pareto front approximation
the set YN and is defined as
y ∈ YN , the closeness between y and the reference vector r j , for
#
j = 1, 2, . . . , |YR |, is given by D (YN ) = D ( y ).
( r j )T ( y − yI ) y∈YN
angle(r j , y ) = arccos .
rj y − yI Let y j, be the jth outer point of the Pareto front approximation YN
If a reference vector rj
is the closest to an element y of YN rela- defined by: for all i = 1, 2, . . . , m,
tively to the closeness metric, it is said that y “covers the reference 
max yi if i = j,
vector r j ” (Cai et al., 2018). The coverage vector c of size |YN | repre- y∈YN
( yij, )=
sents for each y ∈ YN the number of reference vectors that y covers. min yi otherwise.
y∈YN
DIR is the normalized standard deviation of the coverage vector c,
defined as We introduce the hypercube H (YN ) = {y ∈ Rm : y = yI +
 m i, − yI ), a ∈ [0, 1]} where yI is the ideal point. The
a ( y
|YR | "
|YN |
 i=1 i i
1
DIR(YN ; YR ) = (ci − c̄ )2 ÷ |YN | − 1 Laumanns metric is defined as the ratio of the Lebesgue measure
|YN | i=1
|YN | of the intersection of D and H, with the Lebesgue measure of H:
where c̄ is the mean of all ci for i = 1, 2, . . . , |YN |. It is intu- λm (D (YN ) ∩ H (YN )))
itive to understand and cheap to compute (in O (m |YN | × |YR | )
IL (YN ) =
λm (H (YN ))
Cai et al., 2018). It captures both the distribution and the spread-
ing. Nonetheless, it requires the knowledge of the ideal point. The where λm (A ) is the m-dimensional Lebesgue measure of the
number of reference vectors to choose (at least greater than |YN | bounded set A ⊂ Rm . The metric returns a value between 0 and
to be more pertinent) equally plays an important role. It can be 1. The higher the better. An illustration is given in Fig. 4.
biased when the Pareto front is piecewise continuous. This indicator is biased in favor of convex and ex-
tended fronts. Moreover, its computational complexity in
m
3.3.16. The Riesz s-energy indicator (Es ) (Falcón-Cardona, Coello, & O (|YN | 3 poly log |YN | ) Chan (2013) explodes when the objective
Emmerich, 2019; Hardin & Saff, 2004) space dimension increases: in fact, it is similar to the hypervolume
The Riesz s-energy indicator (Hardin & Saff, 2004) aims at indicator (3.4.7) when the reference point is chosen such as  yN .
quantifying a good distribution of points in d-dimensional mani- Similarly, the convex hull surface indicator (Zhao et al., 2018)
folds. Given a Pareto front approximation YN , this indicator is de- measures the volume of the convex hull formed by the Pareto front
fined as: approximation and a reference point r ∈ Rm dominated by all the
  1 elements of the Pareto front approximation. The greater the value
Es (YN ) = of this indicator is, the better. Computational cost increases expo-
y1 − y2 s
y1 ∈YN y2 ∈YN \{y1 } nentially with the number of objectives. As it only considers points
where s > 0 is a fixed external parameter which controls the de- on the boundaries of the convex hull, it is more pertinent to use it
gree of uniformity of the elements of YN . on convex Pareto front approximations.
An uniformly distributed Pareto front approximation must have
a minimal Riesz s-energy value. In Hardin and Saff (2005), it is 3.3.18. Other distribution indicators
proved that configurations of points in a rectifiable d−dimensional Some other indicators are mentioned in this subsection. They
manifold that have minimum Riesz s-energy possess asymptoti- require external parameters chosen by the user that can be crucial
cally uniformly distribution properties for s ≥ d. Moreover, s does to their performance. The reader can consult the provided refer-
not depend on the shape of the manifold (Hardin & Saff, 2005). ences.
Use of the Riesz s-energy indicator to assess generation of an
uniformly distributed Pareto front approximation can be found 1. Entropy measure (Farhang-Mehr & Azarm, 2004): For each
in Falcón-Cardona et al. (2019). element of YN , an influential function (a Gaussian function
centered in y for y ∈ YN ) is defined, which enables the cre-
3.3.17. Laumanns metric (IL ) (Laumanns, Günter, & Schwefel, 1999; ation of a density function considered as the sum of influ-
Laumanns, Zitzler, & Thiele, 2000) ential functions for each element y ∈ YN . Peaks and valleys
The Laumanns metric measures the normalized volume of the in the objective space are considered as places where in-
objective space dominated by a Pareto front approximation and formation can be measured. A “good” Pareto front approx-
bounded above by an approximated nadir objective  vector. Given imation should have an uniform density function in the ob-
a vector y in the objective space Rm , let D (y ) = y2 ∈ Rm , y ≤ y2 jective space. The subset of the objective space bounded by

409
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

the nadir and ideal objective vectors is firstly normalized, When an element of the Pareto front approximation YN does
then divided into boxes, whose the number is decided by not belong to the set of nearest points to the Pareto optimal solu-
the user. Based on this discretization of the objective space, tion set YP , it is ignored by the IGD indicator. The authors of Tian,
the indicator is computed using the values of the density Zhang, Cheng, and Jin (2016) propose a variant of the IGD indicator,
function for each center of each box and the Shannon for- named IGD–NS, which takes into account these non-contributed el-
mula of entropy (Shannon, 2001). The higher the value, the ements in an Euclidean distance-based indicator. Let
better.  
2. Cluster CLμ and Number of Distinct Choices NDCμ (Wu & YNC = y ∈ YN : ∀y ∈ YP , y − y
2 2
= min y − y 1 2
y1 ∈YN
Azarm, 20 0 0): Given two respective good (ideal objective
vector) and bad (maximum objective vector assuming it ex- be the set of non-contributed elements of YN . The IGD–NS indicator
ists) objective vectors yI and yM , the objective space (pre- is defined by
liminary normalized) is divided into hyperboxes of size μ  
(∈ (0; 1]). NDCμ is defined as the number of hyperboxes IGD–NS(YN ; YP ) = min y1 − y2 + min y1 − y2 .
y1 ∈YN y1 ∈YNC
containing elements of the Pareto front approximation. For y2 ∈YP y2 ∈YP
this indicator, a higher value is desirable. CLμ is then defined A lower indicator value is desirable.
|YN |
as CLμ (YN ) = NDCμ (YN )
. A higher value of the CLUμ indicator
is the consequence of a more clustered distribution of the el- 3.4.2. Averaged Hausdorff distance ( p ) (Schutze, Esquivel, Lara, &
ements of a Pareto front approximation and so a lower value Coello, 2012)
is considered to be better. In Schutze et al. (2012), the authors combine GD (3.2.1) and
3. Sigma diversity metrics σ and σ (Mostaghim & Teich, 2005): IGD (3.4.1) into a new indicator, called the averaged Hausdorff dis-
The objective space is divided into zones delimited by uni- tance  p defined by
formly distributed reference lines starting from the origin
whose the number equals |YN |. The indicator value is the ra-  p (YN ; YP ) = max {GD p (YN ; YP ), IGD p (YN ; YP )}
tio of the number of objective vectors that are sufficiently where GD p and IGD p are slightly modified versions of the GD and
close to the reference lines according to the Euclidean norm IGD indicators defined as
with a threshold d chosen by the user, over the total number 1
p p
of reference lines. The higher the value, the better. 1 
4. Diversity comparison indicator DCI (Li, Yang, & Liu, 2014): It GD p (YN ; YP ) = min y1 − y2
|YN | 1
y ∈YN
y2 ∈YP
is a k-ary spread indicator. The zone of interest in the objec-
tive space delimited by lower and upper bounds is divided and
into a number of hyperboxes. For each Pareto front approx- 1
imation, a contribution coefficient is computed relatively to p p
1 
the hyperboxes where non-dominated points are found. For IGD p (YN ; YP ) = min y − y 1 2
.
each Pareto front approximation, DCI returns the mean of
| P| 2
Y
y ∈YP
y1 ∈YN

contribution coefficients relatively to all hyperboxes of in-

This indicator is to be minimized. It is straightforward to compute
terest. A variant is the M–DI indicator (Asafuddoula, Ray, &
and to understand. On the contrary, it requires the knowledge of
Singh, 2015) (Modified Diversity Indicator) which considers a
the Pareto front. The authors of Schutze et al. (2012) introduce this
distributed reference set in the objective space instead of the
new indicator to correct the drawbacks of the GD and IGD indi-
set of non-dominated points from the union of the k Pareto
cators. It can be used to compare continuous and discontinuous
front approximations.
approximations of Pareto fronts.
A drawback of these indicators is the choice of external parame- In Vargas and Bogoya (2018), the authors propose an exten-
ters (d threshold, μ size, number of hyperboxes) that can wrongly sion of the averaged Hausdorff distance indicator, called the p, q-
favor Pareto front approximations over others. σ and CLμ can be averaged distance  p,q for p, q ∈ R \ {0}. Given two finite sets
considered as cardinal indicators too and therefore suffer from the of objective vectors Y 1 ⊂ Rm , Y 2 ⊂ Rm , the generational distance
same drawbacks as the above cardinal indicators. GD p,q for p, q ∈ R \ {0} is defined as
⎛ p ⎞ 1p
q
3.4. Convergence and distribution indicators 1  1 
GD p,q (Y 1 , Y 2 ) = ⎝ 1 y1 − y2 q ⎠ .
|Y | 1 1 y ∈Y
|Y 2 | 2 2
y ∈Y
These indicators are of two types: some enable to compare sev-
eral Pareto approximations in term of distribution and Pareto dom- $
When p < 0 or q < 0, GD p,q exists if and only if Y 1 Y 2 = ∅. Given
inance. The others give a value that capture distribution, spreading
a Pareto front approximation YN and a discrete representation of
and convergence at the same time.
the Pareto front YP ⊆ Y p , the p, q-averaged distance indicator is de-
fined as
3.4.1. Inverted generational distance (IGD) (Coello & Cortés, 2005)
IGD has a quite similar form than GD. It captures the average  p,q (YN ; YP ) = max (GD p,q (YN , YP \ YN ), GD p,q (YP , YN \ YP )).
minimal distance from an element of a discrete representation of As the averaged Hausdorff distance, this indicator is not Pareto
the Pareto front to the closest point in the Pareto front approxima- compliant (Vargas & Bogoya, 2018). However, once that the values
tion. It is given by of p and q are selected, it is straightforward to compute and to
1 understand.
p p
1 
IGD(YN ; YP ) = min y − y1 2
.
|YP | y2 ∈YP
y1 ∈YN 3.4.3. Modified inverted generational distance (IGD+ ) (Ishibuchi,
Masuda, Tanigaki, & Nojima, 2015)
Generally, p = 2. A lower value is considered to be better. Pros and Although the GD (3.2.1) and IGD (3.4.1) indicators are com-
cons are the same as for the GD indicator (3.2.1). monly used due to their low computational cost (Riquelme et al.,

410
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

2015), one of their major drawbacks is that they are non mono- The authors of Hansen and Jaszkiewicz (1998) recommend to
tone (Ishibuchi et al., 2015). The  p indicator (3.4.2) has the same use the utility set U∞ = (uw )w∈W of weighted Tchebycheff utility
problem (Schutze et al., 2012). functions, with
Also, the authors of Ishibuchi et al. (2015) propose a slightly
different version of the IGD indicator named IGD+ integrating the uw (y ) = − max (wi |yi − ri | )
i=1,2,...,m
dominance relation computable in O (m |YN | × |YP | ) where YP is a
fixed Pareto optimal solution set. It is weakly Pareto compliant, for y ∈ Rm where r is a reference objective vector chosen so that
i.e. : any objective vector of the feasible objective set does not dominate
r (or as an approximation of the ideal point Brockhoff, Wagner, &
for all YN1 , YN2 ∈  such that YN1 YN2 , IGD+ (YN1 ; YP ) ≤ IGD+ (YN2 ; YP ). Trautmann, 2012; Brockhoff, Wagner, & Trautmann, 2015; Zitzler
The IGD+ indicator is defined by et al., 2008) and w ∈ W a weight vector such that for all w ∈ W
and i = 1, 2, . . . , m,
1 
IGD+ (YN ; YP ) = min (y1 − y2 )+ 
m
|YP | y2 ∈Y y1 ∈YN
wi ≥ 0 and wi = 1.
P

i=1
where ( y1 − y2 )
+ = (max(0, y1i − y2i ))i=1,2,...,m .
As opposed to the IGD indicator, IGD+ takes
into account the Zitzler et al. (2008) suggest using the set of augmented weighted
dominance relation between an element of YP and an element of Tchebycheff utility functions defined by
YN when computing their Euclidean distance. A reference set YR
can also be used instead of YP : the authors of Ishibuchi, Masuda, 
m
uw ( y ) = − max wi |yi − ri | + ρ | ( yi − ri |
Tanigaki, and Nojima (2014) analyze the choice of such reference i=1,2,...,m
i=1
sets. This indicator can be used with discontinuous and continu-
ous Pareto fronts. where ρ is a sufficiently small positive real number.
Similarly to IGD+ , given a reference set YR ⊂ Rm , As given in Brockhoff et al. (2012), for m = 2 objectives, W can
Dist1R (Czyzzak & Jaszkiewicz, 1998) is given by be chosen such that:
1  1. W = {(0, 1 ), ( k−1
1 1
), ( k−1
2 2
), . . . , ( 1, 0 )}
Dist1R (YN ; YR ) = min max {0, wi (yi − ri )} , 1− k−1
, 1− k−1
is a
|YR | y∈YN i=1,2,...,m
r∈YR set of k weights uniformly distributed in the space [0; 1]2 .
tan ϕ
2. W = {( 1+tan
1
ϕ , 1+tan ϕ ), ϕ ∈ k } where k = {0, 2(kπ−1) ,
with wi a relative weight assigned to objective i. For all these indi-
2π
cators, a lower value is desirable. 2(k−1 )
, . . . , π2
} is a set of weights uniformly distributed over
the trigonometric circle.
3.4.4. R1 and R2 indicators (Hansen & Jaszkiewicz, 1998)
The IR2 indicator (Brockhoff et al., 2012) is an unary indicator
Let YN1 and YN2 be two Pareto front approximations, U a set of
derived from R2 defined as (in the case of weighted Tchebycheff
utility functions u : Rm → R mapping each point in the objective
utility functions)
space into a measure of utility, and p a probability distribution on
the set U. To each u ∈ U are associated u (YN1 ) = maxy∈Y 1 u(y ) and 1 
& '
N IR2 (YN ; W ) = min max (wi |yi − ri | ) .
u (YN2 ) = maxy∈Y 2 u(y ). The two indicators measure to which ex- |W | y∈YN i=1,2,...,m
w∈W
N
tent YN1 is better than YN2 over the set of utility functions U. The R1 The lower this index, the better.
indicator is given by As Knowles and Corne (2002) remark, “the application of R2 de-
%
pends up on the assumption that it is meaningful to add the val-
R1 (YN1 , YN2 ; U, p) = C (YN1 , YN2 , u ) p(u ) du
u∈U
ues of different utility functions from the set U. This simply means
that each utility function in U must be appropriately scaled with
where
 respect to the others and its relative importance”. R-indicators are
1 if u (YN1 ) > u (YN2 ), only monotonic, i.e. I (YN1 ) ≥ I (YN2 ) in case YN1 weakly dominates YN2 .
C( YN1 , YN2 , u )= 1/2 if u (YN1 ) = u (YN2 ), They do not require important computations as the number of
0 if u (YN1 ) < u (YN2 ). objectives increases. The reference point has to be chosen care-
fully. Studies concerning the properties of the R2 indicator can be
The R2 indicator defined as
    found in Brockhoff et al. (2012), Brockhoff et al. (2015) and Wagner,
R2 (YN1 , YN2 ; U, p) = E u (YN1 ) − E u (YN2 ) Trautmann, and Brockhoff (2013).
%   1 
= u (YN ) − u (YN2 ) p(u )du.
u∈U 3.4.5. G-metric (Lizarraga-Lizarraga, Hernandez-Aguirre, &
Botello-Rionda, 2008)
is the expected difference in the utility of a Pareto front approx-
This indicator enables to compare k Pareto front approximations
imation YN1 with another one YN2 . In practice, these two indicators
based on two criteria: the repartition of their elements and the
use a discrete and finite set U of utility functions associated with
level of domination in the objective space. It is compliant with the
an uniform distribution over U (Zitzler et al., 2008). The two indi-
weak dominance relation as defined above. Its computation de-
cators can then be rewritten as
composes into several steps: given k Pareto front approximations
1  (YN1 , YN2 , . . . , YNk ):
R1 (YN1 , YN2 ; U ) = C (YN1 , YN2 , u ) and
|U | u∈U
1. Scale the values of the objective vectors corresponding to
1   1
R2 (YN1 , YN2 ; U ) = u YN − u (YN2 ). the images of the decision vectors in the k sets, i.e. ex-
|U | u∈U tract the non-dominated objective vectors from the union
#k j
If R2 (YN1 , YN2 ; U ) > 0, then YN1 is considered as better than YN2 . Else if YN , then normalize all objective vectors according to
j=1
R2 (YN1 , YN2 ; U ) ≥ 0, YN1 is considered as not worse than YN2 . the extreme values of the objective vectors of this set.

411
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

2. Group the Pareto front approximations according to their de-

gree of dominance. In level L1 will be put all Pareto front ap-
proximations which are not dominated by the union of the
k Pareto front approximations; remove them from the con-
sidered Pareto front approximations; then in L2 , will be put
the Pareto front approximations which are not dominated by
the union of the remaining Pareto front approximations, and
so on.
3. For each level of dominance Lq for q = 1, 2, . . . , Q , where Q
is the
# number of levels, dominated points belonging in the
set Y are removed. Each objective vector in each set
q N
YN ∈L
of the same level possesses a zone of influence. It is a ball Fig. 5. Illustration of the hypervolume indicator for a biobjective problem.
of radius ρ centered in this last one. The radius ρ consid-
ers distances between neighbors objective vectors (Leung & 3.4.7. Hyperarea/hypervolume metrics (HV ) (Zitzler, 1999)
Wang, 2003) from the k Pareto front approximations. For Named also S-metric, the hypervolume indicator is described as
each Pareto front approximation belonging to the same level the volume of the space in the objective space dominated by the
of dominance, a measure of dispersion is computed. This last Pareto front approximation YN and delimited from above by a ref-
one takes into account the zone of influence that the union erence objective vector r ∈ Rm such that for all y ∈ YN , y ≤ r. The
of non-dominated elements of the set cover in the objective hypervolume indicator is given by
space. The smaller the value, the closer the points are.
4. The G-metric associated to an Pareto front approximation is #
the summation of the dispersion measure of this set and the HV (YN ; r ) = λm [y, r]
largest dispersion measure of Pareto front approximations y∈YN
of lower dominance degree for each level. The bigger, the
better. where λm is the m-dimensional Lebesgue measure. An illustration
is given in Fig. 5 for the biobjective case (m = 2). This indicator is
The computation cost is quite important (in O (k3 × to be maximized.
max j=1,2,...,k |YN |2 ) Lizarraga-Lizarraga et al., 2008) but the cost
j
If the Pareto front is known, the Hyperarea ratio is given by
can be decreased when one considers a small number of Pareto
front approximations. Note that this indicator highly depends on HV (YN ; r )
HR(YN , YP ; r ) = .
the computation of the radius ρ when defining zones of influence. HV (YP ; r )
This indicator can be used for continuous and discontinuous Pareto
fronts, especially to compare two Pareto front approximations, in The greater the ratio is (converges toward 1), the better the ap-
terms of dominance and distribution in the objective space. proximation is.
The hypervolume indicator and some closely related metrics are
the only known unary indicators to be strictly monotonic (Falcón-
3.4.6. Performance comparison indicator (PCI) (Li, Yang, & Liu, 2015)
Cardona, Emmerich, & Coello, 2019; Friedrich, Bringmann, Voß, &
The Performance Comparison indicator (Li et al., 2015) PCI was
Igel, 2011; Zitzler, Brockhoff, & Thiele, 2007; Zitzler et al., 2008),
conceived to rectify the main drawback of the  -indicator. PCI en-
i.e. if a Pareto front approximation YN1 is better than another Pareto
ables to compare k Pareto front approximations taking into ac-
front approximation YN2 , HV (YN1 ; r ) > HV (YN2 ; r ) assuming that all
count their level of dominance and their distribution in the objec-
elements of the two Pareto front approximations are in the interior
tive space. PCI uses a reference set composed of all non-dominated
of the region which dominates the reference point. The best known
points taken from the union of the k Pareto front approximations. m
complexity upper cost is O (|YN | 3 poly log |YN | ) (Chan, 2013). To the
Using extreme values of the reference set, all objective vectors of
best of our knowledge, no implementation of this algorithm is
the k Pareto front approximations are normalized. Then PCI divides
available. Practically, efficient implementations of the exact hyper-
the set into different clusters based on a distance threshold σ . PCI
volume indicator can be found in Beume, Fonseca, Lopez-Ibanez,
estimates the minimum distance move of the points in the Pareto
Paquete, and Vahrenhold (2009), Jaszkiewicz (2018), Lacour, Klam-
front approximations to weakly dominate all points in the clusters.
roth, and Fonseca (2017), Russo and Francisco (2014), Russo and
A lower value reflects better closeness of the Pareto front approxi-
Francisco (2016) and While, Bradstreet, and Barone (2012). The sec-
mation to the reference set.
ond drawback is the choice of the reference point, as illustrated in
This indicator possesses a quadratic computational cost and is
Fig. 6. A practical guide to specify the reference point can be found
Pareto compliant when only two Pareto front approximations are
in Ishibuchi, Imada, Setoguchi, and Nojima (2018).
considered (Li et al., 2015). The authors propose the following
For the biobjective case, it was shown (Auger, Bader, Brockhoff,
choice for the threshold, i.e.
& Zitzler, 2009) that the optimal distribution of non-dominated
1 points which maximizes the hypervolume indicator depends on
σ≈ " the slope of the Pareto front. Consequently, if the Pareto front is
m−1
|YR |(m − 1 )! − (m/2 )
highly nonlinear, a non-uniform Pareto front approximation may
where |YR | is the size of the reference set, which enables this indi- have a higher hypervolume value according to another incompa-
cator to be external-parameter-free. rable Pareto front approximation. Other theoretical results can be
The recent binary dominance move indicator (Li & Yao, 2017) found in Bringmann and Friedrich (2010a, 2010b, 2013). Due to
DoM is a generalization of the PCI indicator, as it computes the its properties, it is widely used in the evolutionary community in
minimum distance move from one Pareto front approximation to the search of potential interesting new points or to compare algo-
weakly dominate another. A polynomial algorithm is proposed in Li rithms.
and Yao (2017) in the biobjective case. To the best of our knowl- Similarly, Zitzler (1999) introduces the Difference D of two sets
edge, no extension of the DoM indicator to more objectives has YN1 and YN2 . D(YN1 , YN2 ) enables to measure the size of the area dom-
been proposed yet. inated by YN1 but not by YN2 in the objective space.

412
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Fig. 6. The relative value of the hypervolume metric depends on the chosen reference point r 1 or r 2 (inspired by Knowles & Corne, 2002). On the top, two non-dominated
YN1 and YN2 sets are shown, with HV (YN1 ; r 1 ) > HV (YN2 ; r 1 ). On the bottom, HV (YN2 ; r 2 ) > HV (YN1 ; r 2 ).

The Hyperarea Difference was suggested by Wu and Azarm volume indicators based on set monotonic and desirability func-
(20 0 0) to compensate the lack of information about the theoret- tions (Emmerich, Deutz, & Yevseyeva, 2014).
ical Pareto front. Given a good objective vector 
yI and a bad objec-
tive vector yM , we can approximate the size of the area dominated
by the Pareto front (or circumvent the objective space by a rect-
angle). The Hyperarea Difference is just the normalization of the
3.4.8. Hypervolume Sharpe-Ratio indicator IHSR (Yevseyeva, Guerreiro,
dominated space by the Pareto front approximation in the objec-
Emmerich, & Fonseca, 2014)
tive space over the given rectangle.
The conception of this indicator proposed by Yevseyeva et al.
More recently, a pondered hypervolume by weights indicator
(2014) lies on an analogy between the financial Portfolio Selection
was introduced by Zitzler et al. (2007) to give a preference of
problem (Markowitz, 1952) and the quality of a Pareto front ap-
an objective according to another. More volume indicators can be
proximation. This analogy interprets the elements of the approxi-
found in Wu and Azarm (20 0 0). Some other authors (Jiang, Yang,
mation set as assets with expected returns. An investor must al-
& Li, 2016) (for biobjective optimization problems) suggest to com-
locate capital to maximize the expected return of the assembled
pute the hypervolume defined by a reference point and the projec-
portfolio while minimizing the expected variance (associated to
tion of the points belonging to the Pareto front approximation on
risk).
the line delimited by the two extreme points. This measure en-
To solve the financial Portfolio Selection problem, it is com-
ables to better estimate the distribution of the points along the
mon to look for a strategy which maximizes the financial per-
Pareto front (in fact, it can be shown that for a linear Pareto front,
formance index known as reward-to-volatility ratio or Sharpe ra-
an uniform distribution of points maximizes the hypervolume in-
tio (Cornuéjols, Peña, & Tütüncü, 2018). Let A = {a1 , a2 , . . . , a|A| } be
dicator: see Auger, Bader, and Brockhoff (2010); Auger et al. (2009,
a non-empty set of assets, let r a ∈ R|A| be the expected return of
2012); Shukla, Doll, and Schmeck (2014) for more details about the
these assets and Q ∈ R|A|×|A| the covariance matrix of asset returns.
properties of the hypervolume indicator). A logarithmic version of
The Sharpe-Ratio maximization problem is defined as
the hypervolume indicator called the logarithmic hypervolume in-
dicator (Friedrich et al., 2011) is defined by
(ra )T z − r f
max hA ( z ) = "
# z∈[0,1]|A| zT Qz
log HV (YN ; r ) = λm [log y, log r] |A|
y∈YN

such that zi = 1
with the same notations as previously. Note that this indicator can i=1

only be used with positive vectors in Rm . Finally, we can mention a

generalization of the hypervolume indicator called the cone-based where r f ∈ R is the return of a riskless asset. This non-linear
hypervolume indicator that was introduced recently by Emmerich, problem which may be difficult to solve can be transformed
Deutz, Kruisselbrink, and Shukla (2013b) and an extension of the (see Cornuéjols et al., 2018 for more details) into the following
hypervolume indicator to reference point free weighted hyper- convex problem (under the condition that Q be symmetric posi-

413
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

tive) authors conceive diversity indicators based on diversity preference

relations in the feasible set. In Deb and Tiwari (2008), the authors
min gA (w ) = wT Qw
w∈R + |A| define diversity crowding-distance indicator in the decision space.
|A|
 Indicators to take into account diversity both in the decision space
such that (ria − r f )wi = 1. and the objective space can be found in Shir, Preuss, Naujoks, and
i=1 Emmerich (2009) and Ulrich, Bader, and Zitzler (2010b). Finally,
Theoptimal strategy z of the first problem is given by z = in Emmerich et al. (2013a), the authors present some measures to
|A|  qualify approximation sets in level set approximations, which are
w / w
i=1 i
where w is the solution of the constrained
subsets of the feasible set.
quadratic problem defined above. Sayin Sayın (20 0 0) states that the decision maker is firstly in-
The authors of Yevseyeva et al. (2014) define the Hypervolume terested by the quality of the best trade-off solutions found in the
Sharpe-Ratio indicator as objective space as long as corresponding decision variables sat-
IHSR (YN ; yl , yu ) = max hYN (z ) isfy the constraints. Furthermore, the number of objectives is usu-
srz∈Z ally smaller than the number of variables, which makes the Pareto
where ⊂ [0, 1]|YN | is the set of feasible strategies of the Sharpe
Z sr front approximation easier to study/visualize.
ratio maximization problem and yl ∈ Rm and yu ∈ Rm two refer-
ence points such that yl < yu . The definition of the covariance ma- 4. Some usages of performance indicators
trix and the asset returns inspired by the hypervolume indicator
are given by This section focuses on four applications of performance indi-
  (m   cators: comparison of algorithms for multiobjective optimization,
|Y | λm [yl , yu ] ∩ [y j , +∞[ k=1 max yuk − max(ykj , ylk ), 0 embedding performance indicators in multiobjective optimization
rj N =   = (m algorithms, definition of stopping criteria, and the use of relevant
λm [yl , yu ] k=1 (yk − yk )
u l
distribution and spread indicators for assessing the diversity char-
and acterization of a Pareto front approximation.
 
λm [yl , yu ] ∩ [yi , +∞[∩[y j , +∞[
Qi, j =   4.1. Comparison of algorithms
λm [yl , yu ]
(m  u  The first use of performance indicators is to evaluate the per-
k=1 max yk − max (yk , yk , yk ), 0
i j l
= (m formance of algorithms on a multiobjective problem. In single-
k=1 (yk − yk )
u l
objective optimization, the most used graphical tools to compare
for i, j ∈ {1, 2, . . . , |YN |} where λm is the m-dimensional Lebesgue algorithms include performance profiles (Dolan & Moré, 2002) and
measure. The riskless asset value is set to r f = 0. The greater the data profiles (Moré & Wild, 2009) (see also Beiranvand, Hare, &
indicator value is, the better. Lucet, 2017 for a detailed survey on the tools to compare single-
The IHSR indicator has desirable properties: providing that yu  optimization algorithms). More specifically, let S be a set of solvers
y < yl for all y ∈ YN , the IHSR indicator is proved to be mono- and P the set of benchmarking problems. Let t p,s > 0 be a perfor-
tonic (Guerreiro & Fonseca, 2016; 2020), but not strictly mono- mance measure of solver s ∈ S on problem p ∈ P: the lower, the
tonic (Guerreiro & Fonseca, 2020). Other theoretical results can be better. Performance and data profiles combine performance mea-
found in Guerreiro and Fonseca (2016, 2020). Due to its relation sures of solvers t p,s to enable a general graphic representation of
with a financial model, it is interpretative. However, it is sensitive the performance of each solver relatively to each other on the set
to the choice of the reference point yu (Guerreiro & Fonseca, 2020). of benchmarking problems P.
Its main drawback is its computational cost, directly linked to the To the best of our knowledge, Custódio et al. (2011) are the first
resolution of the quadratic formulation of the Sharpe ratio max- to use data and performance profiles for multiobjective optimiza-
imization problem. Assuming the associated correlation matrix Q tion. For each
# problem p ∈ P, they build a Pareto front approxima-
p p,s
is symmetric positive, this indicator can be computed in O (|YN |3 ) tion YN = YN composed of the union of all Pareto front ap-
s∈S
p,s
operations (for theoretical complexity results, the reader is invited proximations YN generated by each solver s ∈ S for the problem
to refer to Nesterov and Nemirovskii (1994); see also (Nocedal p. All dominated points are then removed. Pareto front approxima-
& Wright, 2006, chapter 16)). Practically, existing quadratic con- tions and relative Pareto optimal solution sets are then compared
strained solvers can efficiently compute the indicator value for a using cardinality and  and  metrics proposed by Custódio et al.
given Pareto front approximation (see Nocedal & Wright, 2006, (2011).
chapter 16 for a list of quadratic solvers). One of the critics we can make with this approach is the
use of distribution and cardinality indicators that do not cap-
3.5. Quality assessment of Pareto set approximations in decision ture order relations between two different sets. The choice of
space (weakly) monotonic indicators or (≺-complete / ≺-compatible) -
complete / -compatible comparisons methods is more appropri-
By definition, performance indicators enable to characterize ated in this context (Hansen & Jaszkiewicz, 1998; Knowles & Corne,
properties of Pareto front approximations with respect to their di- 2002; Zitzler et al., 2008; Zitzler et al., 2003). Among them, domi-
versity, spread and convergence. Moreover, it is possible to build nance move (3.4.6), G-metric (3.4.5), binary  -indicator (3.2.6), Hy-
indicators which assess the quality of Pareto set approximations pervolume Sharpe-Ratio indicator (3.4.8) and volume-space met-
in the decision space, i.e. to design mappings I :  → R. Some rics (3.4.7) have properties corresponding to these criteria. Math-
research works explore the design of such indicators. For exam- ematical proofs can be found in Auger et al. (2009), Brockhoff
ple, Zitzler et al. (20 0 0) propose the M1 , M2 and M3 indicators et al. (2012, 2015), Guerreiro and Fonseca (2016, 2020), Knowles
to respectively assess convergence, diversity and extension prop- and Corne (2002), Li and Yao (2017), Lizarraga-Lizarraga et al.
erties of Pareto set approximations. In Sayın (20 0 0), the author (2008) and Zitzler et al. (2003)) and are synthesized in Appendices.
suggests using a cardinality indicator that returns the number of An example of performance profile using the hypervolume indica-
non-dominated points |XN |, coverage indicator or uniformity indi- tor (3.4.7) can be found in Liuzzi, Lucidi, and Rinaldi (2016). The
cator in the feasible set. In Ulrich, Bader, and Thiele (2010a), the use of performance indicators such as GD (3.2.1) or IGD (3.4.1) as

414
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

it is done in Al-Dujaili and Suresh (2016) and Brockhoff et al. long to the Pareto front and maximize (or minimize) the perfor-
(2015) is not a pertinent choice due to their inability to cap- mance indicator for a given multiobjective problem. Their study
ture dominance relation. Instead, we suggest to use their weakly enables to understand bias of considered indicators and analyze
monotonic counterpart IGD+ (3.4.3) or DOA (3.2.7), that can be the behavior of bounded size indicator-based algorithms. The first
cheaper to compute than for example the hypervolume indica- ones were done for the hypervolume indicator and some of its
tor when the number of objectives is high. It is equally possible variants in the biobjective case (Auger et al., 2009; 2012; Em-
to build Pareto-compliant indicators by considering a combination merich et al., 2013b) then extended to more objectives in Auger,
of weakly Pareto compliant indicators with at least one strictly Bader, and Brockhoff (2010) and Shukla et al. (2014). Theoretical
Pareto compliant indicator as it is proposed in Falcón-Cardona results for the R2 indicator (Brockhoff et al., 2012), the  p in-
et al. (2019). dicator (Rudolph, Schütze, Grimme, Domínguez-Medina, & Traut-
The attainment function (da Fonseca, Fonseca, & Hall, 2001) mann, 2016) or the Hypervolume Sharpe-Ratio indicator (Guerreiro
is another tool for the performance assessment of multiobjective & Fonseca, 2020) for the biobjective case exist too.
(stochastic) solvers. Assuming a multiobjective solver has produced
j
k Pareto front approximations YN for j = 1, 2, . . . , k on a given prob- 4.3. Stopping criteria of multiobjective algorithms
lem, the empirical attainment function α : Rm → [0, 1] is defined
as To generate a Pareto front approximation, two approaches are
currently considered. The first category, named as scalarization
1 
k
α (y ) = 1{YNj {y}}. methods, consists in aggregating the objective functions and to
k solve a series of single-objective problems. Surveys about scalar-
j=1
ization algorithms can be found for example in Wiecek, Ehrgott,
For a given y ∈ Rm , the attainment function estimates the prob- and Engau (2016). The second class, designed as a posteriori ar-
ability that the multiobjective solver reaches (in term of domi- ticulations of preferences (Custódio et al., 2011) methods, aims at
nance in the objective space) the objective vector y. The interested obtaining the whole Pareto front without combining any objective
reader can refer to Brockhoff, Auger, Hansen, and Tušar (2017), function in a single-objective framework. Evolutionary algorithms,
da Fonseca et al. (2001), Fonseca, da Fonseca, and Paquete (2005), Bayesian optimization methods (Emmerich et al., 2016) or deter-
Fonseca, Guerreiro, López-Ibáñez, and Paquete (2011) and Zitzler ministic algorithms such as DMS (Custódio et al., 2011) belong to
et al. (2008) for additional information. this category.
For scalarization methods, under some assumptions, solutions
4.2. Embedding performance indicators in multiobjective optimization to single-objective problems can be proved to belong to the Pareto
algorithms front or a local one. So, defining stopping criteria results in choos-
ing the number of single-objective problems to solve via the choice
Performance indicators are able to quantify properties a good of parameters and a single-objective stopping criterion for each of
Pareto front approximation should possess. It is then logical to in- them. Stopping at a predetermined number of function evaluations
corporate them into multiobjective optimization methods. By op- is often used in the context of blackbox optimization (Audet et al.,
timizing directly the indicator, one can hope to obtain approxi- 2008). The use of performance indicators also is not relevant.
mations of the Pareto front satisfying demanding properties. For A posteriori methods consider a set of points in the objective
these last years, the evolutionary multiobjective community has space (a population) that is brought to head for the Pareto front
frequently adopted this approach. along iterations. Basically, a maximum number of evaluations is
For example, performance indicators such as R2 (Trautmann, still given as a stopping criterion but it remains crucial to give
Wagner, & Brockhoff, 2013) or HV (Beume, Naujoks, & Emmerich, an estimation to how far from a (local) Pareto front the approxi-
2007), I (Zitzler & Künzli, 2004), IGD+ (Falcón-Cardona et al., mation set is. For multiobjective Bayesian optimization (Emmerich
2019; Lopez & Coello, 2016), IGD–NS (Tian et al., 2016) are used in et al., 2016), the goal is to find at next iteration the point that max-
selection mechanisms in evolutionary algorithms. The reader is in- imizes the hyperarea difference between old non-dominated set of
vited to consult the recent survey (Falcón-Cardona & Coello, 2020) points and the new one. The performance indicator is directly em-
for more information on indicator-based multiobjective evolution- bedded into the algorithm and could be used as a stopping crite-
ary algorithms. Similarly, the  -indicator (Custódio et al., 2011) en- rion. For evolutionary algorithms, surveys on stopping criteria for
ables to identify holes in the Pareto front approximation around multiobjective optimization can be found in Martí et al. (2016) and
which the algorithm can explore to improve diversity. In global Wagner, Trautmann, and Martí (2011). The approach is to measure
stochastic optimization, some methods integrate hypervolume in- the progression of the current population combining performance
dicator (Bradford, Schweidtmann, & Lapkin, 2018; Emmerich, Yang, indicators (hypervolume, MDR, etc.) and statistic tools (Kalman fil-
Deutz, Wang, & Fonseca, 2016; Feliot, Bect, & Vazquez, 2017) and ter Martí et al., 2016, χ 2 -variance test Wagner, Trautmann, and
its variants (Feliot, Bect, & Vazquez, 2019) or R2 (Deutz, Emmerich, Naujoks (2009), etc.) These last ones enable to detect a stationary
& Yang, 2019) to better explore the decision space. In Akhtar and state reached by the evolving population.
Shoemaker (2016), the authors use Radial Basis models and the hy- We believe that the use of monotonic performance indicators
pervolume indicator to identify next promising points to evaluate. or binary ones that capture the dominance property seems to be
In Al-Dujaili and Suresh (2018), the authors propose a multiobjec- the most efficient one in the years to come to follow the behavior
tive optimistic algorithm using the additive  -indicator (3.2.6) and of population-based algorithms along iterations.
analyse its link with the weighted Tchebysheff approach.
The transformation of a multiobjective optimization problem 4.4. Distribution and spread
into a single-objective quality indicator based problem implies a
loss of information. Indeed, the choice of a specific performance The choice of spread and distribution indicators has only a
indicator reflects the personal preferences of the decision user. It sense when one wants to measure the distribution of points in the
is then important to understand the bias of this choice on the so- objective space, no matter how close from the Pareto front the ap-
lution set found. Given a performance indicator, the concept of op- proximated set is. Spread and distribution metrics can put forward
timal μ-distribution (Auger et al., 2009) refers to the study of the global properties (for example statistics on the distribution of the
optimal distributions of non-dominated points of size μ which be- points or extent of the front) or local properties such as the largest

415
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

distance between closest non-dominated points that can be used sidered until now as the most relevant. The hypervolume indica-
to conduct search such as  indicator. Typically, the construction tor possesses good mathematical properties, it can capture domi-
of a distribution or spread indicator requires two steps. The first nance properties and distribution and does not require the knowl-
consists in defining a distance between two points in the objec- edge of the Pareto front. Empirical studies (Jiang et al., 2014; Ok-
tive space. Many distribution indicators in the literature use min- abe et al., 2003) have confirmed its efficiency compared to other
imum Euclidean or Manhattan distance between points such as performance indicators. That is why it has been deeply used in
the SP metric (3.3.1), the  index (3.3.2), HRS (3.3.4), and so on. the evolutionary community (Riquelme et al., 2015). However, it
The DM (3.3.12) and  -metric (3.3.3) indicators use a “sorting dis- has some limitations: the exponential cost as the number of ob-
tance”; ID (3.3.13) a “neighborhood distance” based on a spanning jectives increases and the choice of the reference point. To com-
tree, and so on. Once this is done, many of the existing distribu- pare algorithms, it can be replaced with other indicators capturing
tion indicators are built by using statistic tools on this distance: lower dominance relation such as dominance move, G-metric, bi-
mean ( (3.3.2), U measure (3.3.9), DM (3.3.12) for example), mean nary  -indicator, Hypervolume Sharpe-Ratio indicator, modified in-
square (SP (3.3.1), Dnc (3.3.5)), and so on. verted generated distance or degree of approximation whose com-
To use a distribution or spread indicator, it should satisfy the putational cost is less important.
following properties: Future research can focus on the discovery of new performance
indicators that correct some drawbacks of the hypervolume indi-
1. The support of scaled functions, which enables to compare
cator but keeps its good properties, and the integration of perfor-
all objectives in an equivalent way (DM (3.3.12), OS (3.3.10),
mance indicators directly into algorithms for multiobjective opti-
IOD (3.3.11),  (3.3.3),  (3.3.3)).
mization.
2. For piecewise continuous or discontinuous Pareto front ap-
proximations, a good distribution indicator should not be
Acknowledgments
based on the distance between closest neighbors, as it can
hide some holes (Zheng et al., 2017). Some indicators pos-
This research was financed by Le Digabel’s NSERC discov-
sess this property such as DM (3.3.12),  (3.3.3),  (3.3.3),
ery grant RGPIN-2018-05286, and also by the NSERC CRD RD-
Es (3.3.16) or evenness indicators (3.3.7).
CPJ 490744-15 grant and an InnovÉÉ grant, both in collaboration
3. Distribution and spread performance indicators should not
with Hydro-Québec and Rio Tinto. We also would like to thank the
be based on external parameters, such as Zitzler’s metric
anonymous reviewers for their suggestions and comments which
M2 (3.3.5), UD (3.3.5), or entropy measure (3.3.18).
helped to greatly improve this work.
4. An easy interpretation: a value returned by an indicator has
to be “intuitive” to understand. For example, the binary uni-
formity (3.3.8) is extremely difficult to interpret and should Appendix A. A summary of performance indicators
not be used. This remark applies for all types of performance
indicators. Table 3 draws a summary of all indicators described in
Section 3. Most of complexity cost indications for computing in-
One could directly include spread control parameters in the
dicators are drawn from Jiang et al. (2014). YP ⊆ YP corresponds to
design of new algorithms. The Normal Boundary Intersection
the Pareto optimal solution set and YN is a Pareto front approxima-
method (Das & Dennis, Jr., 1998) controls the spread of a Pareto
tion returned by a given algorithm. The symbol “✗” indicates that
front approximation. This method is also used in the context of
the performance indicator does not satisfy the monotony property.
blackbox optimization (Audet, Savard, & Zghal, 2010).
The “-” symbol corresponds to binary indicators, for which mono-
tonicity has no meaning.
5. Discussion

In this work, we give a review of performance indicators for the Appendix B. Compatibility and completeness
quality of Pareto front approximations in multiobjective optimiza-
tion, as well as some usages of these indicators. Tables 4 and 5 summarize compatibility and completeness
The most important application of performance indicators is to properties.
allow comparison and analysis of results of different algorithms. In Only the strongest relationships are kept. Some of them are
this optic, among all these indicators, the hypervolume indicator drawn from Zitzler et al. (2003). All spread and distribution indica-
and its binary counterpart, the hyperarea difference can be con- tors are not compatible with Pareto front approximation relations.

416
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Table 3
A summary of performance indicators.

Category Performance indicators Sect. Symbol Parameters Comparison sets Computational complexity Monotone

Cardinality C-metric/Two sets Coverage Zitzler and Thiele (1998) 3.1.5 C None Binary indicator O (m |YN1 | × |YN2 | ) -
3.1 Error ratio Van Veldhuizen (1999) 3.1.4 ER None Pareto front YP Low ✗
Generational non dominated vector 3.1.3 GNVG None None Low ✗
generation Van Veldhuizen and Lamont (2000)
Generational non dominated vector generation 3.1.3 GNVGR None Pareto front YP Low ✗
ratio Van Veldhuizen and Lamont (2000)
Mutual domination rate Martí et al. (2016) 3.1.3 MDR None None Low ✗
Nondominated vector additional Van Veldhuizen and 3.1.3 NVA None None Low ✗
Lamont (2000)
Overall nondominated vector generation Van Veldhuizen 3.1.1 ONVG None None Low ✗
(1999)
Overall nondominated vector generation 3.1.2 ONVGR None Pareto front YP Low ✗
ratio Van Veldhuizen (1999)
Ratio of non-dominated points by the reference 3.1.5 C2R None Reference set YR O (m |YN | × |YR | ) ✗
set Hansen and Jaszkiewicz (1998)
Ratio of the reference points Hansen and Jaszkiewicz 3.1.4 C1R None Reference set YR O (m |YN | × |YR | ) ✗
(1998)
Convergence Degree of Approximation Dilettoso et al. (2017) 3.2.7 DOA None Pareto front YP O (m |YN | × |YP | ) Not
strictly
3.2  -family Zitzler et al. (2003) 3.2.6 I None Pareto front YP O (m |YN | × |YP | ) Not
strictly
Generational distance Van Veldhuizen (1999) 3.2.1 GD None Pareto front YP O (m |YN | × |YP | ) ✗
γ -metric Deb et al. (2000) 3.2.1 γ None Pareto front YP O (m |YN | × |YP | ) ✗
Maximum Pareto front error Van Veldhuizen (1999) 3.2.4 MPF E None Pareto front YP O (m |YN | × |YP | ) ✗
M1 -metric Zitzler et al. (2000) 3.2.1 M1 None Pareto front YP O (m |YN | × |YP | ) ✗
Progression metric Van Veldhuizen (1999) 3.2.5 - None None O (m |YN | ) ✗
Seven points average distance Schott (1995) 3.2.3 SPAD None Reference set YR O (m |YN | ) ✗
Standard deviation from the Generational 3.2.2 ST DGD None Pareto front YP O (m |YN | × |YP | ) ✗
distance Van Veldhuizen (1999)
Distribution Cluster Wu and Azarm (2000) 3.3.18 CLμ A parameter None High ✗
μ
and spread -index Deb et al. (2000) 3.3.2  None Pareto front YP O (m |YN |2 + m |YN | × |YP | ) ✗
3.3  -index Deb et al. (2000) 3.3.2  None None O (m |YN |2 ) ✗
 spread metric Zhou et al. (2006) 3.3.2  None Pareto front YP O (m |YN |2 + m |YN | × |YP | ) ✗
Distribution metric Zheng et al. (2017) 3.3.12 DM None None O (m |YN |2 )  ✗
Diversity comparison indicator Li et al. (2014) 3.3.18 DCI A parameter k-ary indicator O m (k |YNmax | )2 ✗
div comparing
YN1 , YN2 , . . . , YNk
non-dominated
sets
Diversity indicator Cai et al. (2018) 3.3.15 DIR None Reference set YR O (m |YN | × |YR | ) ✗
Entropy metric Farhang-Mehr and Azarm (2004) 3.3.18 - A parameter None High ✗
grids
Evenness Messac and Mattson (2004) 3.3.7 ξ None None O (m |YN |2 ) ✗
Extension Meng et al. (2005) 3.3.14 EX None Pareto front YP O (m |YN | × |YP | ) ✗
 -metric Custódio et al. (2011) 3.3.3  None None O (m |YN |2 ) ✗
Hole Relative Size Collette and Siarry (2005) 3.3.4 HRS None None O (m |YN |2 ) ✗
O (|YN | 3 poly log |YN | )
m
Laumanns metric Laumanns et al. (2000) 3.3.17 - None None ✗
Modified Diversity indicator Asafuddoula et al. (2015) 3.3.18 M –DI A parameter Reference set YR O (m |YN |2 × |YR | )
δ
M2 -metric Zitzler et al. (2000) 3.3.5 M2 Niche radius None O (m |YN |2 ) ✗
σ
M3 -metric
Zitzler et al. (2000) 3.3.5 M3 None None O (m |YN | ) 2
✗
Number of distinct choices Wu and Azarm (2000) 3.3.18 NDCμ A parameter None High ✗
μ
Outer diameter Zitzler et al. (2008) 3.3.11 IOD None None O (m |YN | ) ✗
Overall Pareto Spread Wu and Azarm (2000) 3.3.10 OS None y˜I and y˜M O (m |YN | ) ✗
Riesz S-energy Hardin and Saff (2004) 3.3.16 ES A parameter None O (m |YN |2 ) ✗
s
Sigma diversity metric Mostaghim and Teich (2005) 3.3.18 σ A parameter None High ✗
lines
Spacing Schott (1995) 3.3.1 SP None None O (m |YN |2 ) ✗
U-measure Leung and Wang (2003) 3.3.9 U None None O (m |YN |2 ) ✗
Uniform assessment metric Li et al. (2008) 3.3.13 ID None None O (m |YN |2 ) ✗
Uniform distribution Tan et al. (2002) 3.3.5 UD Niche radius None O (m |YN |2 ) ✗
σ
Uniformity Sayın (2000) 3.3.6 δ None None O (m |YN |2 ) ✗
Uniformity Meng et al. (2005) 3.3.8 - None Binary Quadratic ✗
(continued on next page)

417
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Table 3 (continued)

Category Performance indicators Sect. Symbol Parameters Comparison sets Computational complexity Monotone

Convergence Averaged Hausdorff distance Schutze et al. (2012) 3.4.2 q None Pareto front YP O (m |YN | × |YP | ) ✗
and
Angle γ and O (|YN | poly log |YN | )
m
distribution Cone-based hypervolume Emmerich et al. (2013b) 3.4.7 - None 3 Strictly
Reference
point r
3.4 Dominance move Li and Yao (2017) 3.4.6 DoM None Binary indicator O (|YN | log |YN | ) -
O (|YN | 3 poly log |YN | )
m
D-metric/Difference coverage of two sets Zitzler (1999) 3.4.7 - Reference Binary indicator -
point r
DR -metric Czyzzak and Jaszkiewicz (1998) 3.4.3 - None Reference set YR O (m |YN | × |YR | ) Not
strictly
O (|YN | 3 poly log |YN | )
m
Hyperarea difference Wu and Azarm (2000) 3.4.7 HD Reference None Strictly
point r
O (|YN | 3 poly log |YN | )
m
Hypervolume indicator (or S-metric) Zitzler et al. (2000) 3.4.7 HV Reference None Strictly
point r
Hypervolume Sharpe-ratio indicator Yevseyeva et al. 3.4.8 IHSR None Polynomial Not
(2014) Reference points strictly
yl and yu
Inverted generational distance Coello and Cortés (2005) 3.4.1 IGD None Pareto front YP O (m |YN | × |YP | ) ✗
Inverted generation distance with non contributed 3.4.1 IGD–NS None Pareto front YP O (m |YN | × |YP | ) ✗
solutions detection Tian et al. (2016)
G-metric Lizarraga-Lizarraga et al. (2008) 3.4.5 - None k-ary indicator O (k3 |YNmax |2 ) Not
comparing strictly
YN1 , YN2 , . . . , YNk
non-dominated
sets
O (|YN | 3 poly log |YN | )
m
Logarithmic hypervolume indicator Friedrich et al. (2011) 3.4.7 log HV Reference None Strictly
point r
Modified inverted generational distance Ishibuchi et al. 3.4.3 IGD +
None Pareto front YP O (m |YN | × |YP | ) Not
(2015) strictly
Performance comparison indicator Li et al. (2015) 3.4.6 PCI σ distance k-ary indicator Quadratic Not
comparing strictly
YN1 , YN2 , . . . , YNk
non-dominated
sets
p, q-averaged distance Vargas and Bogoya (2018) 3.4.2  p,q None Pareto front YP Quadratic ✗
R-metric Hansen and Jaszkiewicz (1998) 3.4.4 YR A set W of Reference set YR O (m |YN | × |YR | × |W | ) Not
weights strictly
vectors

Table 4
Compatibility and completeness of unary performance indicators.

Category Performance indicators Sect. Symbol Boolean function Compatible Complete

Cardinality Error ratio Van Veldhuizen (1999) 3.1.4 ER ER(YN1 ) < ER(YN2 ) ✗ ✗
3.1 Generational non dominated vector 3.1.3 GNVG - - -
generation Van Veldhuizen and Lamont (2000)
Generational non dominated vector generation 3.1.3 GNVGR - - -
ratio Van Veldhuizen and Lamont (2000)
Mutual domination rate Martí et al. (2016) 3.1.6 MDR - - -
Nondominated vector additional Van Veldhuizen 3.1.3 NVA - - -
and Lamont (2000)
Overall nondominated vector 3.1.1 ONVG ONVG(YN1 ) > ONVG(YN2 ) ✗ ✗
generation Van Veldhuizen (1999)
Overall nondominated vector generation 3.1.2 ONVGR ONVGR(YN1 ; YP ) > ONVGR(YN2 ; YP ) ✗ ✗
ratio Van Veldhuizen (1999)
Ratio of non-dominated points by the reference 3.1.5 C2R (
C2R YN1 ; YR )> (
C2R YN2 ; YR ) ✗ ✗
set Hansen and Jaszkiewicz (1998)
Ratio of the reference points Hansen and 3.1.4 C1R C1R (YN1 ; YR ) > C1R (YN2 ; YR ) ✗ ✗
Jaszkiewicz (1998)
Convergence Degree of Approximation Dilettoso et al. (2017) 3.2.7 DOA DOA(YN1 ; YP ) < DOA(YN2 ; YP ) Not better than ≺
3.2 Generational distance Van Veldhuizen (1999) 3.2.1 GD GD (YN1 ; YP ) < GD (YN2 ; YP ) ✗ ✗
γ -metric Deb et al. (2000) 3.2.1 γ γ (YN1 ; YP ) < γ (YN2 ; YP ) ✗ ✗
Maximum Pareto front error Van Veldhuizen 3.2.4 MPF E MPF E (YN1 ; YP ) < MPF E (YN2 ; YP ) ✗ ✗
(1999)
M1 -metric Zitzler et al. (2000) 3.2.1 M1 M1 (YN1 ; YP ) < M1 (YN2 ; YP ) ✗ ✗
Progression metric Van Veldhuizen (1999) 3.2.5 - - - -
Seven points average distance Schott (1995) 3.2.3 SPAD SPAD (YN1 ; YP ) < SPAD (YN2 ; YP ) ✗ ✗
Standard deviation from the Generational 3.2.2 ST DGD - - -
distance Van Veldhuizen (1999)
Distribution Cluster Wu and Azarm (2000) 3.3.18 CLμ - - -
and spread -index Deb et al. (2000) 3.3.2  (YN1 ; YP ) < (YN2 ; YP ) ✗ ✗
(continued on next page)

418
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

Table 4 (continued)

Category Performance indicators Sect. Symbol Boolean function Compatible Complete

3.3  -index Deb et al. (2000) 3.3.2   (YN1 ) <  (YN2 ) ✗ ✗

 spread metric Zhou et al. (2006) 3.3.2   (YN1 ; YP ) <  (YN2 ; YP ) ✗ ✗
Distribution metric Zheng et al. (2017) 3.3.12 DM DM (YN1 ) < DM (YN2 ) ✗ ✗
Diversity indicator Cai et al. (2018) 3.3.15 DIR DIR(YN1 ) < DIR(YN2 ) ✗ ✗
Entropy metric Farhang-Mehr and Azarm (2004) 3.3.18 - - - -
Evenness Messac and Mattson (2004) 3.3.7 ξ ξ (YN1 ) < ξ (YN2 ) ✗ ✗
Extension Meng et al. (2005) 3.3.14 EX EX (YN1 ; YP ) < EX (YN2 ; YP ) ✗ ✗
 -metric Custódio et al. (2011) 3.3.3  (YN1 ) < (YN2 ) ✗ ✗
Hole Relative Size Collette and Siarry (2005) 3.3.4 HRS HRS (YN1 ) < HRS (YN2 ) ✗ ✗
Laumanns metric Laumanns et al. (2000) 3.3.17 - IL (YN1 ) > IL (YN2 ) ✗ ✗
Modified Diversity indicator Asafuddoula et al. 3.3.18 M –DI M –DI (YN1 ; YR ) > M –DI (YN2 ; YR ) ✗ ✗
(2015)
M2 -metric Zitzler et al. (2000) 3.3.5 M2 M2 (YN1 ; σ ) > M2 (YN2 ; σ ) ✗ ✗
M3 -metric Zitzler et al. (2000) 3.3.5 M3 M3 (YN1 ) > M3 (YN2 ) ✗ ✗
Number of distinct choices Wu and Azarm (2000) 3.3.18 NDCμ NDCμ (YN1 ) > NDCμ (YN2 ) ✗ ✗
Outer diameter Zitzler et al. (2008) 3.3.11 IOD IOD (YN1 ) > IOD (YN2 ) ✗ ✗
Overall Pareto Spread Wu and Azarm (2000) 3.3.10 OS OS (YN1 ) > OS (YN2 ) ✗ ✗
Riesz S-energy Hardin and Saff (2004) 3.3.16 ES ES (YN1 ) < ES (YN2 ) ✗ ✗
Sigma diversity metric Mostaghim and Teich 3.3.18 σ σ (YN1 ; d ) > σ (YN2 ; d ) ✗ ✗
(2005)
Spacing Schott (1995) 3.3.1 SP SP (YN1 ) < SP (YN2 ) ✗ ✗
U-measure Leung and Wang (2003) 3.3.9 U U (YN1 ) < U (YN2 ) ✗ ✗
Uniform assessment metric Li et al. (2008) 3.3.13 ID ID (YN1 ) > ID (YN2 ) ✗ ✗
Uniform distribution Tan et al. (2002) 3.3.5 UD UD (YN1 ; σ ) < UD(YN2 ; σ ) ✗ ✗
Uniformity Sayın (2000) 3.3.6 δ δ (YN1 ) < δ (YN2 ) ✗ ✗
Convergence Averaged Hausdorff distance Schutze et al. (2012) 3.4.2 q q (YN1 ; YP ) < q (YN2 ; YP ) ✗ ✗
and Cone-based hypervolume Emmerich et al. (2013b) 3.4.7 - χ (YN1 ) > χ (YN2 ) Not better than 
distribution 3.4 DR -metric Czyzzak and Jaszkiewicz (1998) 3.4.3 - DR (YN1 ; YR ) < DR (YN2 ; YR ) Not better than ≺≺
Hyperarea difference Wu and Azarm (2000) 3.4.7 HD HD (YN1 ) < HD(YN2 ) Not better than 
Hypervolume indicator (or S-metric) Zitzler et al. 3.4.7 HV HV (YN1 ; r ) > HV (YN2 ; r ) Not better than 
(2000)
Hypervolume Sharpe-ratio indicator Yevseyeva 3.4.8 IHSR IHSR (YN1 ; yl , yu ) > IHSR (YN2 ; yl , yu ) Not better than ≺
et al. (2014)
Inverted generational distance Coello and Cortés 3.4.1 IGD IGD ( YN1 , P ) < IGD(B, P ) ✗ ✗
(2005)
Inverted generation distance with non contributed 3.4.1 IGD–NS IGD–NS (YN1 ; YP ) < IGD–NS (YN2 ; YP ) ✗ ✗
solutions detection Tian et al. (2016)
Logarithmic hypervolume indicator Friedrich 3.4.7 log HV log HV (YN1 ; r ) > log HV (YN2 ; r ) Not better than 
et al. (2011)
Modified inverted generational distance Ishibuchi 3.4.3 IGD +
IGD (+
YN1 ; YP ) < IGD (+
YN2 ; YP ) Not better than
et al. (2015)
p, q-averaged distance Vargas and Bogoya (2018) 3.4.2  p,q  ( p,q YN ; YP
1
)< ( p,q YN ; YP
1
) ✗ ✗

Table 5
Compatibility and completeness of binary performance indicators (inspired by Zitzler et al. (2003)): a - means there is no comparison method which is complete and
compatible for the given relation, a ✗ that the indicator is not even monotone.

Category Performance indicators Sect. Symbol Relation

 =

C (YN1 , YN2 ) = 1 C (YN1 , YN2 ) = 1 C (YN1 , YN2 ) > 1

Cardinality 3.1 C-metric/Two sets Coverage Zitzler and Thiele 3.1.5 C C (YN1 , YN2 ) = 1
C (YN2 , YN1 ) < 1 C (YN2 , YN1 ) = 1 C (YN2 , YN1 ) > 1
(1998)
I (YN1 , YN2 ) ≤ 0 I (YN1 , YN2 ) = 0 I (YN1 , YN2 ) > 0
Convergence 3.2 Additive  -indicator Zitzler et al. (2003) 3.2.6 I I (YN1 , YN2 ) ≤ 0
I (YN2 , YN1 ) > 0 I (YN2 , YN1 ) = 0 I (YN2 , YN1 ) > 0
Distribution and Diversity comparison indicator Li et al. (2014) 3.3.18 DCI ✗ ✗ ✗ ✗
spread
3.3 Uniformity Meng et al. (2005) 3.3.8 – ✗ ✗ ✗ ✗
DoM (YN1 , YN2 ) = 0 DoM (YN1 , YN2 ) = 0 DoM (YN1 , YN2 ) = 0 DoM (YN1 , YN2 ) > 0
Convergence and Dominance move Li and Yao (2017) 3.4.6 DoM
DoM (YN2 , YN1 ) > 0 DoM (YN2 , YN1 ) ≥ 0 DoM (YN2 , YN1 ) = 0 DoM (YN1 , YN2 ) > 0
D (YN1 , YN2 ) > 0 D(YN1 , YN2 ) ≥ 0 D(YN1 , YN2 ) = 0 D(YN1 , YN2 ) > 0
distribution 3.4 D-metric/Difference coverage of two sets Zitzler 3.4.7 –
D (YN2 , YN1 ) = 0 D (YN2 , YN1 ) = 0 D (YN2 , YN1 ) = 0 D (YN2 , YN1 ) > 0
(1999)
G-metric Lizarraga-Lizarraga et al. (2008) 3.4.5 – – – - -
Performance comparison indicator Li et al. (2015) 3.4.6 PCI – – - -
R-metric Hansen and Jaszkiewicz (1998) 3.4.4 R – – - -

419
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

References Coello, C., & Cortés, N. (2005). Solving multiobjective optimization problems using
an artificial immune system. Genetic Programming and Evolvable Machines, 6(2),
Akhtar, T., & Shoemaker, C. (2016). Multi objective optimization of computation- 163–190. https://doi.org/10.1007/s10710- 005- 6164- x.
ally expensive multi-modal functions with RBF surrogates and multi-rule se- Collette, Y., & Siarry, P. (2005). Three new metrics to measure the convergence of
lection. Journal of Global Optimization, 64(1), 17–32. https://doi.org/10.1007/ metaheuristics towards the Pareto frontier and the aesthetic of a set of solutions
s10898-015-0270-y. in biobjective optimization. Computers and Operations Research, 32(4), 773–792.
Al-Dujaili, A., & Suresh, S. (2016). BMOBench: Black-box multi-objective optimiza- https://doi.org/10.1016/j.cor.2003.08.017.
tion benchmarking platform. Technical Report. arXiv. Collette, Y., & Siarry, P. (2011). Multiobjective optimization: Principles and case studies.
Al-Dujaili, A., & Suresh, S. (2018). Revisiting norm optimization for multi-objective Springer. https://doi.org/10.1007/978- 3- 662- 08883- 8.
black-box problems: A finite-time analysis. Journal of Global Optimization, 73(3), Cornuéjols, G., Peña, J., & Tütüncü, R. (2018). Optimization methods in finance. Cam-
659–673. https://doi.org/10.1007/s10898- 018- 0709- z. bridge University Press. https://doi.org/10.1017/9781107297340.
Asafuddoula, M., Ray, T., & Singh, H. (2015). Characterizing Pareto front approxima- Custódio, A., Emmerich, M., & Madeira, J. (2012). Recent developments in derivative-
tions in many-objective optimization. In Proceedings of the 2015 annual confer- free multiobjective optimization. Computational Technology Reviews, 5(1), 1–30.
ence on genetic and evolutionary computation (pp. 607–614). New York, NY, USA: https://doi.org/10.4203/ctr.5.1.
ACM. https://doi.org/10.1145/2739480.2754701. Custódio, A., Madeira, J., Vaz, A., & Vicente, L. (2011). Direct multisearch for mul-
Audet, C., Savard, G., & Zghal, W. (2008). Multiobjective optimization through a se- tiobjective optimization. SIAM Journal on Optimization, 21(3), 1109–1140. https:
ries of single-objective formulations. SIAM Journal on Optimization, 19(1), 188– //doi.org/10.1137/10079731X.
210. https://doi.org/10.1137/060677513. Czyzzak, P., & Jaszkiewicz, A. (1998). Pareto simulated annealing–a meta-
Audet, C., Savard, G., & Zghal, W. (2010). A mesh adaptive direct search algorithm heuristic technique for multiple-objective combinatorial optimization. Jour-
for multiobjective optimization. European Journal of Operational Research, 204(3), nal of Multi-Criteria Decision Analysis, 7(1), 34–47. https://doi.org/10.1002/(sici)
545–556. https://doi.org/10.1016/j.ejor.2009.11.010. 1099- 1360(199801)7:1<34::aid- mcda1613.0.co;2- 6.
Auger, A., Bader, J., & Brockhoff, D. (2010). Theoretically investigating optimal μ- Dächert, K., Klamroth, K., Lacour, R., & Vanderpooten, D. (2017). Efficient compu-
distributions for the hypervolume indicator: First results for three objectives. In tation of the search region in multi-objective optimization. European Journal of
R. Schaefer, C. Cotta, J. Kołodziej, & G. Rudolph (Eds.), Parallel problem solving Operational Research, 260(3), 841–855. https://doi.org/10.1016/j.ejor.2016.05.029.
from nature, PPSN XI (pp. 586–596). Berlin, Heidelberg: Springer. https://doi.org/ Das, I., & DennisJr. , J. (1998). Normal-boundary intersection: A new method for
10.1007/978- 3- 642- 15844- 5_59. generating the Pareto surface in nonlinear multicriteria optimization prob-
Auger, A., Bader, J., Brockhoff, D., & Zitzler, E. (2009). Theory of the hypervolume lems. SIAM Journal on Optimization, 8(3), 631–657. https://doi.org/10.1137/
indicator: Optimal μ-distributions and the choice of the reference point. In S1052623496307510.
Proceedings of the tenth ACM SIGEVO workshop on foundations of genetic algo- Deb, K. (2001). Multi-objective optimization using evolutionary algorithms. New York,
rithms (pp. 87–102). New York, NY, USA: ACM. https://doi.org/10.1145/1527125. NY, USA: John Wiley and Sons.
1527138. Deb, K., Agrawal, S., Pratap, A., & Meyarivan, T. (20 0 0). A fast elitist non-
Auger, A., Bader, J., Brockhoff, D., & Zitzler, E. (2012). Hypervolume-based multiob- dominated sorting genetic algorithm for multi-objective optimization: NSGA-
jective optimization: Theoretical foundations and practical implications. Theo- II. In M. Schoenauer, K. Deb, G. Rudolph, X. Yao, E. Lutton, J. Merelo, & H.-
retical Computer Science, 425, 75–103. https://doi.org/10.1016/j.tcs.2011.03.012. P. H. P. Schwefel (Eds.), Parallel problem solving from nature PPSN VI (pp. 849–
Bäck, T. (1996). Evolutionary algorithms in theory and practice: Evolution strategies, 858). Berlin, Heidelberg: Springer. https://doi.org/10.1007/3- 540- 45356- 3_83.
evolutionary programming, genetic algorithms. New York, NY, USA: Oxford Uni- Deb, K., & Tiwari, S. (2008). Omni-optimizer: A generic evolutionary algorithm
versity Press. for single and multi-objective optimization. European Journal of Operational Re-
Beiranvand, V., Hare, W., & Lucet, Y. (2017). Best practices for comparing optimiza- search, 185(3), 1062–1087. https://doi.org/10.1016/j.ejor.2006.06.042.
tion algorithms. Optimization and Engineering, 18(4), 815–848. https://doi.org/10. Deutz, A., Emmerich, M., & Yang, K. (2019). The expected R2-indicator improvement
1007/s11081- 017- 9366- 1. for multi-objective Bayesian optimization. In K. Deb, E. Goodman, C. Coello,
Beume, N., Fonseca, C., Lopez-Ibanez, M., Paquete, L., & Vahrenhold, J. (2009). On K. Klamroth, K. Miettinen, S. Mostaghim, & P. Reed (Eds.), Evolutionary multi-
the complexity of computing the hypervolume indicator. IEEE Transactions on criterion optimization (pp. 359–370). Cham: Springer. https://doi.org/10.1007/
Evolutionary Computation, 13(5), 1075–1082. https://doi.org/10.1109/tevc.2009. 978- 3- 030- 12598- 1_29.
2015575. Dilettoso, E., Rizzo, S., & Salerno, N. (2017). A weakly Pareto compliant quality in-
Beume, N., Naujoks, B., & Emmerich, M. (2007). SMS-EMOA: Multiobjective selec- dicator. Mathematical and Computational Applications, 22(1). https://doi.org/10.
tion based on dominated hypervolume. European Journal of Operational Research, 3390/mca22010025.
181(3), 1653–1669. https://doi.org/10.1016/j.ejor.20 06.08.0 08. Dolan, E., & Moré, J. (2002). Benchmarking optimization software with perfor-
Bradford, E., Schweidtmann, A., & Lapkin, A. (2018). Efficient multiobjective opti- mance profiles. Mathematical Programming, 91(2), 201–213. https://doi.org/10.
mization employing Gaussian processes, spectral sampling and a genetic al- 1007/s101070100263.
gorithm. Journal of Global Optimization, 71(2), 407–438. https://doi.org/10.1007/ Ehrgott, M. (2005). Multicriteria optimization, volume 491 of lecture notes in eco-
s10898- 018- 0609- 2. nomics and mathematical systems (2nd). Berlin: Springer.
Branke, J., Deb, K., Miettinen, K., & Slowiński, R. (2008). Multiobjective optimization: Emmerich, M., Deutz, A., & Kruisselbrink, J. (2013a). On Quality Indicators for Black-
Interactive and evolutionary approaches: 5252. Springer. https://doi.org/10.1007/ -Box Level Set Approximation (pp. 157–185)). Berlin, Heidelberg: Springer.
978- 3- 540- 88908- 3. Emmerich, M., Deutz, A., Kruisselbrink, J., & Shukla, P. (2013b). Cone-based hy-
Bringmann, K., & Friedrich, T. (2010a). Approximating the volume of unions and pervolume indicators: Construction, properties, and efficient computation. In
intersections of high-dimensional geometric objects. Computational Geometry, R. Purshouse, P. Fleming, C. Fonseca, S. Greco, & J. Shaw (Eds.), Evolutionary
43(6), 601–610. https://doi.org/10.1016/j.comgeo.2010.03.004. multi-criterion optimization (pp. 111–127). Berlin, Heidelberg: Springer. https:
Bringmann, K., & Friedrich, T. (2010b). Tight bounds for the approximation ratio of //doi.org/10.1007/978- 3- 642- 37140- 0_12.
the hypervolume indicator. In R. Schaefer, C. Cotta, J. Kołodziej, & G. Rudolph Emmerich, M., Deutz, A., & Yevseyeva, I. (2014). On Reference Point Free Weighted
(Eds.), Parallel problem solving from nature, PPSN XI (pp. 607–616). Berlin, Hei- Hypervolume Indicators based on Desirability Functions and their Probabilistic
delberg: Springer. Interpretation. Procedia Technology, 16, 532–541. https://doi.org/10.1016/j.protcy.
Bringmann, K., & Friedrich, T. (2013). Approximation quality of the hypervolume in- 2014.10.001.
dicator. Artificial Intelligence, 195, 265–290. https://doi.org/10.1016/j.artint.2012. Emmerich, M., Yang, K., Deutz, A., Wang, H., & Fonseca, C. (2016). A Multicriteria
09.005. Generalization of Bayesian Global Optimization (pp. 229–242)). Cham: Springer.
Brockhoff, D., Auger, A., Hansen, N., & Tušar, T. (2017). Quantitative performance Falcón-Cardona, J., & Coello, C. (2020). Indicator-based multi-objective evolutionary
assessment of multiobjective optimizers: The average runtime attainment func- algorithms: A comprehensive survey. ACM Computing Surveys, 53(2), 1–35. https:
tion. In H. Trautmann, G. Rudolph, K. Klamroth, O. Schütze, M. Wiecek, Y. Jin, & //doi.org/10.1145/3376916.
C. Grimme (Eds.), Evolutionary multi-criterion optimization (pp. 103–119). Cham: Falcón-Cardona, J., Coello, C., & Emmerich, M. (2019). CRI-EMOA: A Pareto-front
Springer. https://doi.org/10.1007/978- 3- 319- 54157- 0_8. shape invariant evolutionary multi-objective algorithm. In K. Deb, E. Goodman,
Brockhoff, D., Wagner, T., & Trautmann, H. (2012). On the properties of the R2 in- C. Coello, K. Klamroth, K. Miettinen, S. Mostaghim, & P. Reed (Eds.), Evolutionary
dicator. In Proceedings of the 14th annual conference on genetic and evolutionary multi-criterion optimization (pp. 307–318). Cham: Springer.
computation (pp. 465–472). New York, NY, USA: ACM. https://doi.org/10.1145/ Falcón-Cardona, J., Emmerich, M., & Coello, C. (2019). On the Construction of Pareto-
2330163.2330230. Compliant Quality Indicators. In Proceedings of the Genetic and Evolutionary Com-
Brockhoff, D., Wagner, T., & Trautmann, H. (2015). 2 indicator-based multiobjective putation Conference Companion (pp. 2024–2027). ACM. https://doi.org/10.1145/
search. Evolutionary Computation, 23(3), 369–395. https://doi.org/10.1162/EVCO_ 3319619.3326902.
a_00135. Farhang-Mehr, A., & Azarm, S. (2004). An information-theoretic entropy metric for
Cai, X., Sun, H., & Fan, Z. (2018). A diversity indicator based on reference vectors assessing multi-objective optimization solution set quality. Journal of Mechanical
for many-objective optimization. Information Sciences, 430–431, 467–486. https: Design, 125(4), 655–663. https://doi.org/10.1115/1.1623186.
//doi.org/10.1016/j.ins.2017.11.051. Faulkenberg, S., & Wiecek, M. (2009). On the quality of discrete representations in
Chan, T. (2013). Klee’s measure problem made easy. In IEEE 54th annual symposium multiple objective programming. Optimization and Engineering, 11(3), 423–440.
on foundations of computer science (pp. 410–419). IEEE. https://doi.org/10.1109/ https://doi.org/10.1007/s11081- 009- 9099- x.
FOCS.2013.51. Feliot, P., Bect, J., & Vazquez, E. (2017). A Bayesian approach to constrained single-
Cheng, S., Shi, Y., & Qin, Q. (2012). On the performance metrics of multiob- and multi-objective optimization. Journal of Global Optimization, 67(1–2), 97–
jective optimization. In Y. Tan, Y. Shi, & Z. Ji (Eds.), Advances in swarm in- 133. https://doi.org/10.1007/s10898- 016- 0427- 3.
telligence (pp. 504–512). Berlin, Heidelberg: Springer. https://doi.org/10.1007/ Feliot, P., Bect, J., & Vazquez, E. (2019). User preferences in Bayesian multi-objective
978- 3- 642- 30976- 2_61. optimization: The expected weighted hypervolume improvement criterion. In

420
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

G. Nicosia, P. Pardalos, G. Giuffrida, R. Umeton, & V. Sciacca (Eds.), Machine Li, M., & Yao, X. (2019). Quality evaluation of solution sets in multiobjective optimi-
learning, optimization, and data science (pp. 533–544). Cham: Springer. https: sation: A survey. ACM Computing Surveys, 52(2), 26:1–26:38. https://doi.org/10.
//doi.org/10.1007/978- 3- 030- 13709- 0_45. 1145/3300148.
Fonseca, C., da Fonseca, V., & Paquete, L. (2005). Exploring the performance of Li, M., Zheng, J., & Xiao, G. (2008). Uniformity assessment for evolutionary multi-
stochastic multiobjective optimisers with the second-order attainment function. objective optimization. In IEEE congress on evolutionary computation (IEEE World
In C. Coello, A. H. Aguirre, & E. Zitzler (Eds.), Evolutionary multi-criterion op- Congress on Computational Intelligence) (pp. 625–632). https://doi.org/10.1109/
timization (pp. 250–264). Berlin, Heidelberg: Springer. https://doi.org/10.1007/ CEC.2008.4630861.
978- 3- 540- 31880- 4_18. Liuzzi, G., Lucidi, S., & Rinaldi, F. (2016). A derivative-free approach to constrained
Fonseca, C., Guerreiro, A., López-Ibáñez, M., & Paquete, L. (2011). On the computa- multiobjective nonsmooth optimization. SIAM Journal on Optimization, 26(4),
tion of the empirical attainment function. In R. Takahashi, K. Deb, E. Wanner, 2744–2774. https://doi.org/10.1137/15M1037810.
& S. Greco (Eds.), Evolutionary multi-criterion optimization (pp. 106–120). Berlin, Lizarraga-Lizarraga, G., Hernandez-Aguirre, A., & Botello-Rionda, S. (2008). G-metric:
Heidelberg: Springer. https://doi.org/10.1007/978- 3- 642- 19893- 9_8. An M-ary quality indicator for the evaluation of non-dominated sets. In Pro-
da Fonseca, V., Fonseca, C., & Hall, A. (2001). Inferential performance assess- ceedings of the 10th annual conference on genetic and evolutionary computa-
ment of stochastic optimisers and the attainment function. In E. Zitzler, tion (pp. 665–672). New York, NY, USA: ACM. https://doi.org/10.1145/1389095.
L. Thiele, K. C. A. C. Deb, & D. Corne (Eds.), Evolutionary multi-criterion op- 1389227.
timization (pp. 213–225). Berlin, Heidelberg: Springer. https://doi.org/10.1007/ Lopez, E., & Coello, C. (2016). IGD+-EMOA: A multi-objective evolutionary algo-
3- 540- 44719- 9_15. rithm based on IGD+. In 2016 IEEE Congress on Evolutionary Computation (CEC)
Friedrich, T., Bringmann, K., Voß, T., & Igel, C. (2011). The logarithmic hypervolume (pp. 999–1006). https://doi.org/10.1109/CEC.2016.7743898.
indicator. In Proceedings of the 11th workshop proceedings on foundations of ge- Markowitz, H. (1952). Portfolio selection. Journal of Finance, 7(1), 77–91.
netic algorithms (pp. 81–92). New York, NY, USA: ACM. https://doi.org/10.1145/ Martí, L., García, J., Berlanga, A., & Molina, J. (2016). A stopping criterion for multi-
1967654.1967662. objective optimization evolutionary algorithms. Information Sciences, 367–368,
Ghosh, D., & Chakraborty, D. (2015). A direction based classical method to obtain 700–718. https://doi.org/10.1016/j.ins.2016.07.025.
complete Pareto set of multi-criteria optimization problems. Opsearch, 52(2), Meng, H., Zhang, X., & Liu, S. (2005). New quality measures for multiobjective
340–366. https://doi.org/10.1007/s12597- 014- 0178- 1. programming. In L. Wang, K. Chen, & Y. Ong (Eds.), Advances in natural com-
Guerreiro, A., & Fonseca, C. (2016). Hypervolume sharpe-ratio indicator: Formaliza- putation (pp. 1044–1048). Berlin, Heidelberg: Springer. https://doi.org/10.1007/
tion and first theoretical results. In Parallel problem solving from nature – PPSN 11539117_143.
XIV (pp. 814–823). https://doi.org/10.1007/978- 3- 319- 45823- 6_76. Messac, A., & Mattson, C. (2004). Normal constraint method with guarantee of
Guerreiro, A., & Fonseca, C. (2020). An analysis of the hypervolume sharpe-ratio even representation of complete Pareto frontier. AIAA Journal, 42(10), 2101–2111.
indicator. European Journal of Operational Research, 283(2), 614–629. https://doi. https://doi.org/10.2514/1.8977.
org/10.1016/j.ejor.2019.11.023. Moré, J., & Wild, S. (2009). Benchmarking derivative-free optimization algorithms.
Hansen, M., & Jaszkiewicz, A. (1998). Evaluating the quality of approximations to the SIAM Journal on Optimization, 20(1), 172–191. https://doi.org/10.1137/080724083.
non-dominated set. IMM, Department of Mathematical Modelling. Mostaghim, S., & Teich, J. (2005). A new approach on many objective diversity
Hardin, D., & Saff, E. (2004). Discretizing manifolds via minimum energy points. measurement. In J. Branke, K. Deb, K. Miettinen, & R. Steuer (Eds.), Practi-
Notices of the AMS, 51(10), 1186–1194. cal approaches to multi-objective optimization. In Dagstuhl Seminar Proceedings.
Hardin, D., & Saff, E. (2005). Minimal Riesz energy point configurations for rectifi- Dagstuhl, Germany: Internationales Begegnungs- und Forschungszentrum für
able d-dimensional manifolds. Advances in Mathematics, 193(1), 174–204. https: Informatik (IBFI), Schloss Dagstuhl, Germany.
//doi.org/10.1016/j.aim.20 04.05.0 06. Nesterov, Y., & Nemirovskii, A. (1994). Interior-point polynomial algorithms in convex
Ishibuchi, H., Imada, R., Setoguchi, Y., & Nojima, Y. (2018). How to specify a reference programming. Society for Industrial and Applied Mathematics. https://doi.org/10.
point in hypervolume calculation for fair performance comparison. Evolutionary 1137/1.9781611970791.
Computation, 26(3), 411–440. https://doi.org/10.1162/evco_a_00226. Nocedal, J., & Wright, S. (2006). Numerical optimization. Springer Series in Operations
Ishibuchi, H., Masuda, H., Tanigaki, Y., & Nojima, Y. (2014). Difficulties in specify- Research and Financial Engineering (2nd). Berlin: Springer.
ing reference points to calculate the inverted generational distance for many- Okabe, T., Jin, Y., & Sendhoff, B. (2003). A critical survey of performance indices
objective optimization problems. In IEEE symposium on computational intel- for multi-objective optimisation. In Evolutionary computation: 2 (pp. 878–885).
ligence in multi-criteria decision-making (pp. 170–177). https://doi.org/10.1109/ https://doi.org/10.1109/CEC.2003.1299759. Canberra, Australia
mcdm.2014.7007204. Riquelme, N., Lücken, C. V., & Baran, B. (2015). Performance metrics in multi-
Ishibuchi, H., Masuda, H., Tanigaki, Y., & Nojima, Y. (2015). Modified distance calcu- objective optimization. In Latin American computing conference (pp. 1–11). IEEE.
lation in generational distance and inverted generational distance. In A. Gaspar- https://doi.org/10.1109/CLEI.2015.7360024.
Cunha, C. H. Antunes, & C. Coello (Eds.), Evolutionary multi-criterion optimization Rudolph, G., Schütze, O., Grimme, C., Domínguez-Medina, C., & Traut-
(pp. 110–125). Cham: Springer. https://doi.org/10.1007/978- 3- 319- 15892- 1_8. mann, H. (2016). Optimal averaged Hausdorff archives for bi-objective problems:
Jaszkiewicz, A. (2018). Improved quick hypervolume algorithm. Computers and Op- theoretical and numerical results. Computational Optimization and Applications,
erations Research, 90, 72–83. https://doi.org/10.1016/j.cor.2017.09.016. 64(2), 589–618. https://doi.org/10.1007/s10589-015-9815-8.
Jiang, S., Ong, Y., Zhang, J., & Feng, L. (2014). Consistencies and contradictions of Russo, L., & Francisco, A. (2014). Quick hypervolume. IEEE Transactions on Evolution-
performance metrics in multiobjective optimization. IEEE Transactions on Cyber- ary Computation, 18(4), 481–502. https://doi.org/10.1109/TEVC.2013.2281525.
netics, 44(12), 2391–2404. https://doi.org/10.1109/tcyb.2014.2307319. Russo, L., & Francisco, A. (2016). Extending quick hypervolume. Journal of Heuristics,
Jiang, S., Yang, S., & Li, M. (2016). On the use of hypervolume for diversity measure- 22(3), 245–271. https://doi.org/10.1007/s10732- 016- 9309- 6.
ment of Pareto front approximations. In 2016 IEEE Symposium Series on Com- Sayın, S. (20 0 0). Measuring the quality of discrete representations of efficient sets
putational Intelligence (SSCI) (pp. 1–8). IEEE. https://doi.org/10.1109/SSCI.2016. in multiple objective mathematical programming. Mathematical Programming,
7850225. 87(3), 543–560. https://doi.org/10.1007/s101070 050 011.
Kidd, M., Lusby, R., & Larsen, J. (2020). Equidistant representations: Connecting cov- Schott, J. (1995). Fault tolerant design using single and multicriteria genetic algo-
erage and uniformity in discrete biobjective optimization. Computers and Oper- rithm optimization. Technical Report. Air force Institute Of Tech Wright-Patter-
ations Research, 117, 104872. https://doi.org/10.1016/j.cor.2019.104872. son AFB OH.
Knowles, J., & Corne, D. (2002). On metrics for comparing nondominated sets. In Schutze, O., Esquivel, X., Lara, A., & Coello, C. (2012). Using the averaged Hausdorff
Evolutionary computation: 1 (pp. 711–716). Honolulu, HI: IEEE. https://doi.org/ distance as a performance measure in evolutionary multiobjective optimization.
10.1109/CEC.2002.1007013. IEEE Transactions on Evolutionary Computation, 16(4), 504–522. https://doi.org/
Lacour, R., Klamroth, K., & Fonseca, C. (2017). A box decomposition algorithm to 10.1109/TEVC.2011.2161872.
compute the hyperVolume indicator. Computers and Operations Research, 79, Shannon, C. (2001). A mathematical theory of communication. ACM SIGMOBILE Mo-
347–360. https://doi.org/10.1016/j.cor.2016.06.021. bile Computing and Communications Review, 5(1), 3–55. https://doi.org/10.1145/
Laumanns, M., Günter, R., & Schwefel, H. (1999). Approximating the Pareto set: Con- 584091.584093.
cepts, diversity issues, and performance assessment. Secretary of the SFB 531. Shir, O., Preuss, M., Naujoks, B., & Emmerich, M. (2009). Enhancing decision space
Laumanns, M., Zitzler, E., & Thiele, L. (20 0 0). A unified model for multi-objective diversity in evolutionary multiobjective algorithms. In M. Ehrgott, C. Fonseca,
evolutionary algorithms with elitism. In Congress on evolutionary computation: X. Gandibleux, J.-K. Hao, & M. Sevaux (Eds.), Evolutionary multi-criterion op-
1 (pp. 46–53). https://doi.org/10.1109/CEC.20 0 0.870274. Piscataway, New Jersey, timization (pp. 95–109). Berlin, Heidelberg: Springer. https://doi.org/10.1007/
USA 978- 3- 642- 01020- 0_12.
Leung, Y., & Wang, Y. (2003). U-measure: a quality measure for multiobjective pro- Shukla, P., Doll, N., & Schmeck, H. (2014). A theoretical analysis of volume based
gramming. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems Pareto front approximations. In Proceedings of the 2014 annual conference on ge-
and Humans, 33(3), 337–343. https://doi.org/10.1109/TSMCA.2003.817059. netic and evolutionary computation (pp. 1415–1422). New York, NY, USA: ACM.
Li, M., Yang, S., & Liu, X. (2014). Diversity comparison of Pareto front approximations https://doi.org/10.1145/2576768.2598348.
in many-objective optimization. IEEE Transactions on Cybernetics, 44(12), 2568– Slowinski, R., & Teghem, J. (1990). Stochastic versus fuzzy approaches to multiobjective
2584. https://doi.org/10.1109/TCYB.2014.2310651. mathematical programming under uncertainty. Springer. https://doi.org/10.1007/
Li, M., Yang, S., & Liu, X. (2015). A performance comparison indicator for Pareto 978- 94- 009- 2111-5.
front approximations in many-objective optimization. In Proceedings of the 2015 Tan, K., Lee, T., & Khor, E. (2002). Evolutionary algorithms for multi-objective op-
annual conference on genetic and evolutionary computation (pp. 703–710). New timization: performance assessments and comparisons. Artificial Intelligence Re-
York, NY, USA: ACM. https://doi.org/10.1145/2739480.2754687. view, 17(4), 251–290. https://doi.org/10.1023/A:1015516501242.
Li, M., & Yao, X. (2017). Dominance move: A measure of comparing solution sets in Tian, Y., Zhang, X., Cheng, R., & Jin, Y. (2016). A multi-objective evolutionary algo-
multiobjective optimization. Technical Report. arXiv. rithm based on an enhanced inverted generational distance metric. In Proceed-

421
C. Audet, J. Bigeon, D. Cartier et al. European Journal of Operational Research 292 (2021) 397–422

ings of the 2016 IEEE congress on evolutionary computation (CEC’2016) (pp. 5222– Wu, J., & Azarm, S. (20 0 0). Metrics for quality assessment of a multiobjective design
5229). IEEE. https://doi.org/10.1109/CEC.2016.7748352. optimization solution set. Journal of Mechanical Design, 123(1), 18–25. https://
Trautmann, H., Wagner, T., & Brockhoff, D. (2013). R2-EMOA: Focused multiobjective doi.org/10.1115/1.1329875.
search using R2-indicator-based selection. In G. Nicosia, & P. Pardalos (Eds.), Yevseyeva, I., Guerreiro, A., Emmerich, M., & Fonseca, C. (2014). A portfolio optimiza-
Learning and intelligent optimization (pp. 70–74). Berlin, Heidelberg: Springer. tion approach to selection in multiobjective evolutionary algorithms. In Parallel
https://doi.org/10.1007/978- 3- 642- 44973- 4_8. problem solving from nature – PPSN XIII (pp. 672–681). https://doi.org/10.1007/
Ulrich, T., Bader, J., & Thiele, L. (2010a). Defining and optimizing indicator-based 978- 3- 319- 10762- 2_66.
diversity measures in multiobjective search. In R. Schaefer, C. Cotta, J. Kołodziej, Zhao, J., Jiao, L., Liu, F., Fernandes, V., Yevseyeva, I., Xia, S., & Emmerich, M. (2018).
& G. Rudolph (Eds.), Parallel problem solving from nature, PPSN XI (pp. 707–717). 3D fast convex-hull-based evolutionary multiobjective optimization algorithm.
Berlin, Heidelberg: Springer. https://doi.org/10.1007/978- 3- 642- 15844- 5_71. Applied Soft Computing, 67, 322–336. https://doi.org/10.1016/j.asoc.2018.03.005.
Ulrich, T., Bader, J., & Zitzler, E. (2010b). Integrating decision space diversity into Zheng, K., Yang, R., Xu, H., & Hu, J. (2017). A new distribution metric for comparing
hypervolume-based multiobjective search. In Proceedings of the 12th annual con- Pareto optimal solutions. Structural and Multidisciplinary Optimization, 55(1), 53–
ference on genetic and evolutionary computation (pp. 455–462). New York, NY, 62. https://doi.org/10.10 07/s0 0158-016-1469-3.
USA: ACM. https://doi.org/10.1145/1830483.1830569. Zhou, A., Jin, A., Zhang, Q., Sendhoff, B., & Tsang, E. (2006). Combining model-based
Van Veldhuizen, D. (1999). Multiobjective evolutionary algorithms: Classifications, and genetics-based offspring generation for multi-objective optimization using
analyses, and new innovations. Technical Report. School of Engineering of the a convergence criterion. In IEEE international conference on evolutionary compu-
Air Force Institute of Technology, Dayton, Ohio. tation (pp. 892–899). IEEE. https://doi.org/10.1109/CEC.2006.1688406.
Van Veldhuizen, D., & Lamont, G. (20 0 0). On measuring multiobjective evolution- Zitzler, E. (1999). Evolutionary algorithms for multiobjective optimization: Methods and
ary algorithm performance. In Proceedings of the 2000 congress on evolutionary applications. Zurich, Swiss Federal: Institute of Technology Ph.D. thesis..
computation: 1 (pp. 204–211). https://doi.org/10.1109/CEC.20 0 0.870296. Zitzler, E., Brockhoff, D., & Thiele, L. (2007). The hypervolume indicator revis-
Vargas, A., & Bogoya, J. (2018). A generalization of the averaged Hausdorff ited: on the design of Pareto-compliant indicators via weighted integration.
distance. Computación y Sistemas, 22(2), 331–345. https://doi.org/10.13053/ In S. Obayashi, K. Deb, C. Poloni, T. Hiroyasu, & T. Murata (Eds.), Evolutionary
CyS- 22- 2- 2950. multi-criterion optimization (pp. 862–876). Berlin, Heidelberg: Springer. https:
Wagner, T., Trautmann, H., & Brockhoff, D. (2013). Preference articulation by means //doi.org/10.1007/978- 3- 540- 70928- 2_64.
of the R2 indicator. In R. Purshouse, J. Fleming, C. Fonseca, S. Greco, & J. Shaw Zitzler, E., Deb, K., & Thiele, L. (20 0 0). Comparison of multiobjective evolutionary
(Eds.), Evolutionary multi-criterion optimization (pp. 81–95). Berlin, Heidelberg: algorithms: Empirical results. Evolutionary Computation, 8(2), 173–195. https://
Springer. https://doi.org/10.1007/978- 3- 642- 37140- 0_10. doi.org/10.1162/106365600568202.
Wagner, T., Trautmann, H., & Martí, L. (2011). A taxonomy of online stopping criteria Zitzler, E., Knowles, J., & Thiele, L. (2008). Quality assessment of Pareto set approx-
for multi-objective evolutionary algorithms. In R. Takahashi, K. Deb, E. Wanner, imations. In Multiobjective optimization (pp. 373–404). Springer, Berlin, Heidel-
& S. Greco (Eds.), Evolutionary multi-criterion optimization (pp. 16–30). Berlin, berg. https://doi.org/10.1007/978- 3- 540- 88908- 3_14.
Heidelberg: Springer. https://doi.org/10.1007/978- 3- 642- 19893- 9_2. Zitzler, E., & Künzli, S. (2004). Indicator-based selection in multiobjective search.
Wagner, T., Trautmann, H., & Naujoks, B. (2009). OCD: Online convergence detec- In X. Yao, E. Burke, J. Lozano, J. Smith, J. Merelo-Guervós, J. Bullinaria, . . . H.-
tion for evolutionary multi-objective algorithms based on statistical testing. In S. Schwefel (Eds.), Parallel problem solving from nature - PPSN VIII (pp. 832–842).
M. Ehrgott, C. Fonseca, X. Gandibleux, J.-K. Hao, & M. Sevaux (Eds.), Evolu- Berlin, Heidelberg: Springer. https://doi.org/10.1007/978- 3- 540- 30217- 9_84.
tionary multi-criterion optimization (pp. 198–215). Berlin, Heidelberg: Springer. Zitzler, E., & Thiele, L. (1998). Multiobjective optimization using evolutionary al-
https://doi.org/10.1007/978- 3- 642- 01020- 0_19. gorithms – A comparative case study. In A. Eiben, T. Bäck, M. Schoenauer, &
While, L., Bradstreet, L., & Barone, L. (2012). A fast way of calculating exact hy- H. Schwefel (Eds.), Parallel problem solving from nature – PPSN V (pp. 292–301).
pervolumes. IEEE Transactions on Evolutionary Computation, 16(1), 86–95. https: Berlin, Heidelberg: Springer. https://doi.org/10.10 07/BFb0 056872.
//doi.org/10.1109/TEVC.2010.2077298. Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C., & da Fonseca, V. (2003). Perfor-
Wiecek, M., Ehrgott, M., & Engau, A. (2016). Continuous multiobjective program- mance assessment of multiobjective optimizers: An analysis and review. IEEE
ming. In Multiple criteria decision analysis (pp. 739–815). Springer New York. Transactions on Evolutionary Computation, 7(2), 117–132. https://doi.org/10.1109/
https://doi.org/10.1007/978- 1- 4939- 3094- 4_18. tevc.2003.810758.

422