IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO.

XX, AUGUST 2020 1

Fast Greedy Subset Selection from Large Candidate

Solution Sets in Evolutionary Multi-objective
Optimization
Weiyu Chen, Hisao Ishibuchi, Fellow, IEEE, and Ke Shang

Abstract—Subset selection is an interesting and important topic optimal solution. A Pareto front of the multi-objective problem
in the field of evolutionary multi-objective optimization (EMO). is the set that contains the corresponding objective values of
arXiv:2102.00941v1 [cs.NE] 1 Feb 2021

Especially, in an EMO algorithm with an unbounded external all Pareto optimal solutions. If no solution in a solution set is
archive, subset selection is an essential post-processing procedure
to select a pre-specified number of solutions as the final result. In dominated by any other solutions, the solution set is referred
this paper, we discuss the efficiency of greedy subset selection for to as a non-dominated solution set.
the hypervolume, IGD and IGD+ indicators. Greedy algorithms The Pareto dominance relation has been extended to the
usually efficiently handle subset selection. However, when a large dominance relation between two sets [2]: A solution set A
number of solutions are given (e.g., subset selection from tens of dominates solution set B if and only if ∀b ∈ B, ∃a ∈ A,
thousands of solutions in an unbounded external archive), they
often become time-consuming. Our idea is to use the submodular such that a ≺ b. A better relation [2] is defined using the
property, which is known for the hypervolume indicator, to weak Pareto dominance relation between two non-dominated
improve their efficiency. First, we prove that the IGD and IGD+ solution sets A and B as follows: A is better than B if and
indicators are also submodular. Next, based on the submodular only if ∀b ∈ B, ∃a ∈ A, a  b and A 6= B.
property, we propose an efficient greedy inclusion algorithm for Since the number of Pareto optimal solutions can be very
each indicator. Then, we demonstrate through computational
experiments that the proposed algorithms are much faster than large for combinatorial optimization problems and infinity
the standard greedy subset selection algorithms. for continuous optimization problems, subset selection is an
essential topic in the EMO field. It is involved in many phases
Index Terms—Evolutionary multi-objective optimization, evo-
lutionary many-objective optimization, subset selection, perfor- of EMO algorithms. (i) In each generation, we need to select
mance indicators, submodularity. a pre-specified number of solutions from the current and
offspring populations for the next generation. (ii) After the
I. I NTRODUCTION execution of EMO algorithms, the final population is usually
presented to the decision-maker. However, if the decision-

E VOLUTIONARY multi-objective optimization (EMO)

algorithms aim to optimize m potentially conflicting
objectives concurrently. To compare solutions, we usually
maker does not want to examine all solutions in the final
population, we need to choose only a small number of rep-
resentative solutions for the decision-makers. (iii) Since many
use the Pareto-dominance relation [1]. In a multi-objective good solutions are discarded during the execution of EMO
minimization problem with m objectives fi (x), i = 1, 2, ..., m, algorithms [3], we can use an unbounded external archive
solution a dominates solution b (i.e., a ≺ b) if and only if ∀i ∈ (UEA) to store all non-dominated solutions examined during
{1, 2, . . . , m}, fi (a) ≤ fi (b) and ∃j ∈ {1, 2, . . . , m}, fj (a) < the execution of EMO algorithms. In this case, a pre-specified
fj (b). Its weaker version is defined by only the first con- number of solutions are selected from a large number of non-
dition: a weakly dominates b (i.e., a  b) if and only if dominated solutions in the archive.
∀i ∈ {1, 2, . . . , m}, fi (a) ≤ fi (b). This relation includes the Formally, a subset selection problem can be defined as
case where a and b are exactly the same in the m-dimensional follows:
objective space (i.e., fi (a) = fi (b) for i = 1, 2, ..., m).
A solution that is not dominated by any other feasible Definition 1 (Subset Selection Problem). Given an n-point
solutions of a multi-objective problem is called a Pareto set V ⊂ Rm , a performance indicator g : 2|V | → R and a
positive integer k (k < n), maximize g(S) subject to S ⊂ V
This paper is an expanded paper from the IEEE Congress on Evo- and |S| ≤ k.
lutionary Computation held on July 19-24, 2020 in Glasgow, UK. This
work was supported by National Natural Science Foundation of China Many subset selection algorithms have been proposed. They
(Grant No. 61876075), Guangdong Provincial Key Laboratory (Grant No.
2020B121201001), the Program for Guangdong Introducing Innovative can be classified into the following three categories: (i) exhaus-
and Enterpreneurial Teams (Grant No. 2017ZT07X386), Shenzhen Science tive search algorithms, (ii) evolutionary algorithms, and (iii)
and Technology Program (Grant No. KQTD2016112514355531), the Pro- greedy algorithms. Exhaustive search algorithms search for
gram for University Key Laboratory of Guangdong Province (Grant No.
2017KSYS008). the optimal solution subset. The performance indicator value
Corresponding author: Hisao Ishibuchi of the optimal solution subset is always called OPT. Since the
Weiyu Chen, Hisao Ishibuchi and Ke Shang are with the Department subset selection problem is NP-hard [4], [5], it is impractical to
of Computer Science and Engineering, Southern University of Science
and Technology, Shenzhen, China (e-mail: [email protected]; find the optimal solution subset unless the candidate solution
[email protected]; [email protected]). set is small. In practice, evolutionary algorithms and greedy
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 2

algorithms are usually used. better indicator values than the final population.
Greedy algorithms can be further divided into greedy inclu- Although subset selection from all the evaluated solutions
sion algorithms and greedy removal algorithms. The greedy in- is effective, it is time-consuming. When an EMO algorithm
clusion algorithms select solutions from V one by one. In each is applied to a many-objective problem, a huge number of
iteration, the solution that leads to the largest improvement of non-dominated solutions are usually included in the evaluated
the performance indicator g is selected until the required num- solutions. That is, the size of the candidate solution set for
ber of solutions are selected. If the performance indicator is subset selection is huge. For example, the size of the UEA
submodular, the obtained solution set can achieve a (1 − 1/e)- obtained by NSGA-II on the nine-objective car cab design
approximation (e is the natural constant) to the optimal subset problem [25] after 200 generations is around 20,000. In this
[6]. This means that the ratio of the performance indicator case, subset selection needs long computation time even when
value of the obtained solution set to the performance indicator greedy algorithms are used. The focus of this paper is to
value of the optimal solution set is not less than (1 − 1/e). decrease the computation time of greedy subset selection
In contrast to greedy inclusion algorithms, greedy removal algorithms, which improves the usefulness and applicability
algorithms discard one solution with the least improvement of of EMO algorithms in real-world scenarios by efficiently
the performance indicator g in each iteration. Although greedy selecting better solution sets than the final population.
removal algorithms are widely used and have shown good In this paper, we propose new greedy inclusion algorithms
performance [7], currently there is no theoretical guarantee for the hypervolume, IGD and IGD+ subset selection. The
for their approximation accuracy. submodularity [6] of the hypervolume, IGD and IGD+ indi-
When the required set size k is close to the size of V (i.e., cators is the key to make the proposed algorithms efficient
when the number of solutions to be removed is small), greedy and applicable to large candidate solution sets with many
removal algorithms are faster than greedy inclusion algorithms. objectives. Based on the submodularity of these indicators,
However, when k is relatively small in comparison with the we can reduce the unnecessary contribution calculations of
size of V , greedy removal algorithms are not efficient since solutions without changing the results (i.e., without changing
they need to remove a large number of solutions. the obtained solution sets) of the corresponding greedy subset
Evolutionary algorithms have also been applied to subset selection algorithms. Experimental results show that the pro-
selection problems [8], [9]. An arbitrary subset of V can be posed idea drastically improves the efficiency of greedy subset
encoded as a binary string b of length |V |. In the string, a one selection from large candidate solution sets of many-objective
“1” indicates the inclusion of the corresponding solution in the problems.
subset and a zero “0” indicates the exclusion of the solution. The followings are the main contributions of this paper.
A population of subsets (i.e., binary strings) is improved by
• We explain the submodular property of set functions,
an evolutionary algorithm. If the performance indicator g is
and prove that the IGD and IGD+ indicators have the
submodular, some algorithms such as POSS [8] have also been
submodular property. Based on our proof, the theoretical
proved to be (1 − 1/e)-approximation algorithms.
approximation ratio of the greedy algorithm based on
Subset selection can use different selection criteria (i.e.,
each indicator can be obtained.
different solution set evaluation criteria). From this point
• By exploiting the submodular property of the three indi-
of view, subset selection algorithms can be categorized as
cators (i.e., hypervolume, IGD, and IGD+), we propose
distance-based subset selection [10]–[12], clustering-based
an efficient greedy inclusion algorithm for subset selec-
subset selection [11], hypervolume subset selection [13]–[16],
tion based on each indicator.
IGD subset selection [17], IGD+ subset selection [17], and -
• We demonstrate that the computation time for subset
indicator subset selection [14], [18]. Some of these indicators
selection can be significantly decreased by the pro-
require a large computation load. For example, the calculation
posed algorithms through computational experiments on
of the hypervolume indicator is #P-hard with respect to the
frequently-used test problems and some real-world prob-
number of objectives [19].
lems.
Selecting the best solution from all the evaluated solutions
as the final solution is a standard choice in single-objective The remainder of this paper is arranged as follows. In
optimization. The best solution is usually stored and updated Section II, we explain hypervolume-based subset selection. In
during the execution of a single-objective algorithm. However, Section III, we discuss the use of the IGD and IGD+ indicators
in multi-objective optimization, it is not easy to store the best for subset selection. In Section IV, we propose an efficient
solution set since the quality of each solution is relatively greedy inclusion algorithm for each indicator. In Section V, the
measured based on its relation with other solutions. It is proposed algorithms are compared with the standard greedy
possible that the same solution is evaluated as the best in one algorithms. Finally, we conclude this paper in Section V.
generation and the worst in another generation. Experimental This paper is an extended version of our conference paper
results in [20] show that some good solutions can be discarded [26]. In our conference paper, we proposed an efficient greedy
during the evolution process (thus the final solution set is not inclusion algorithm using the submodular property of the
the best solution set). Therefore, recently, subset selection from hypervolume indicator. In this paper, we propose efficient
all the evaluated solutions has been used in some studies to algorithms for the IGD and IGD+ indicators after proving
find better solution sets than the final population [3], [10], their submodular property. More experimental results are also
[12], [17], [21]–[24]. The selected solution sets usually have reported in this paper in order to demonstrate the efficiency
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 3

of the proposed submodular property-based algorithms for the Bradstreet et al. [36] proposed a new calculation method to re-
three performance indicators. duce the amount of calculation. The hypervolume contribution
is calculated as
II. H YPERVOLUME S UBSET S ELECTION HV C(p, S) = HV ({p}) − HV (S 0 ), (4)
A. Hypervolume indicator and hypervolume contribution where
The hypervolume indicator [27]–[29] is a widely used S 0 = {limit(s, p)|s ∈ S}, (5)
metric to evaluate the diversity and convergence of a solution limit((s1 , ..., sm ), (p1 , ..., pm ))
set. It is defined as the size of the objective space which is (6)
= (worse(s1 , p1 ), ..., worse(sm , pm )).
covered by a set of non-dominated solutions and bounded by
a reference set R. Formally, the hypervolume of a solution set In this formulation worse (si , pi ) takes the larger value
S is defined as follows: for minimization problems. Compared to the straightforward
Z calculation method in (3), this method is much more efficient.
HV (S) := As (z)dz, (1) The hypervolume of one solution (i.e., HV ({p})) can be
z∈Rm easily calculated. We can also apply the previous mentioned
where m is the number of dimensions and As : Rm → {0, 1} HSO [30], [31], HOY [32]–[34] and WFG [35] to calculate
is the attainment function of S with respect to the reference the hypervolume of a reduced solution set S 0 (i.e., HV (S 0 )).
set R and can be written as Let us take Fig. 2 as an example. Suppose we want to
( calculate the hypervolume contribution of solution p to a
1 if ∃ s ∈ S, r ∈ R : f (s)  z  r, solution set S = {a, b, c, d, e}. First, for each solution in
As (z) = (2)
0 otherwise. S, we replace each of its objective values with the corre-
sponding value from solution p if the value of p is larger
Calculating the hypervolume of a solution set is a #P-hard (i.e., we calculate limit(a, p), ..., limit(e, p)). This leads to
problem [19]. A number of algorithms have been proposed S 0 = {a0 , b0 , c0 , d0 , e0 }. After the replacement, e0 is dominated
to quickly calculate the exact hypervolume such as Hyper- by d0 . Thus e0 can be removed from S 0 since e0 has no
volume by Slicing Objectives (HSO) [30], [31], Hypervolume contribution to the hypervolume of S 0 . Similarly, a0 and b0 can
by Overmars and Yap (HOY) [32]–[34], and Walking Fish also be removed from S 0 . Then, we calculate the hypervolume
Group (WFG) [35]. Among those algorithms, WFG has been of S 0 (i.e., the area of the gray region in Fig. 2) and subtract
generally accepted as the fastest one. The hypervolume con- it from the hypervolume of solution p. The remaining yellow
tribution is defined based on the hypervolume indicator. The part is the hypervolume contribution of solution p.
hypervolume contribution of a solution p to a set S is

HV C(p, S) = HV (S ∪ {p}) − HV (S). (3) r

Fig. 1 illustrates the hypervolume of a solution set and the a'

a HV ( S ')
hypervolume contribution of a solution to the solution set in
two dimensions. The grey region is the hypervolume of the b
solution set S = {a, b, c, d, e} and the yellow region is the
c
hypervolume contribution of a solution p to S. d' e'
p
r d
e
0 Minimize f1
a
HV ( S )
Fig. 2. Illustration of the efficient hypervolume contribution computation
b method.

c
p B. Hypervolume Subset Selection Problem
d
The hypervolume subset selection problem (HSSP) [13]
e uses the hypervolume indicator as the solution selection crite-
0
Minimize f1 ria (i.e., the hypervolume indicator is used as the performance
indicator g in Definition 1). The HSSP aims to select a pre-
Fig. 1. The hypervolume of the solution set S = {a, b, c, d, e} and the
hypervolume contribution of p to the solution set S for a two-objective specified number of solutions from a given candidate solution
minimization problem. set to maximize the hypervolume of the selected solutions.
For two-objective problems, HSSP can be solved with time
Note that calculating the hypervolume contribution based complexity of O(nk + nlogn) [18] and O((n − k)k + nlogn)
on its definition in (3) requires hypervolume calculation twice, [15]. For multi-objective problems with three or more objec-
which is not very efficient. Bringmann and Friedrich [19] and tives, HSSP is an NP-hard problem [37], it is impractical to
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 4

try to find the exact optimal solution set when the size of update the hypervolume contributions efficiently in three and
the candidate set is large and/or the dimensionality of the four dimensions. Using their algorithm, the time complexity of
objective space is high. In practice, some greedy heuristic hypervolume-based greedy removal in three and four dimen-
algorithms and genetic algorithms are employed to obtain an sions can be reduced to O(n(n−k)+nlogn) and O(n2 (n−k))
approximated optimal solution set. respectively.
In a hypervolume-based EMO algorithm called FV-MOEA
C. Greedy Hypervolume Subset Selection proposed by Jiang et al. [41], an efficient hypervolume con-
tribution update method applicable to any dimension was pro-
Hypervolume greedy inclusion selects solutions from V one posed. The main idea of their method is that the hypervolume
by one. In each iteration, the solution that has the largest contribution of a solution is only associated with a small
hypervolume contribution to the selected solution set is se- number of its neighboring solutions rather than all solutions in
lected until the required number of solutions are selected. The the solution set. Let us suppose that one solution sj have just
pseudocode of greedy inclusion is shown in Algorithm 1. Since been removed from the solution set S, the main process of the
hypervolume is a submodular indicator [38], hypervolume hypervolume contribution update method in [41] is shown in
greedy inclusion algorithms provide a (1−1/e)-approximation Algorithm 2.
to HSSP [6].
Hypervolume greedy removal algorithms discard one solu-
Algorithm 2 Hypervolume Contribution Update
tion with the least hypervolume contribution to the current
solution set in each iteration. Unlike greedy inclusion, greedy Input: HV C (The hypervolume contribution of each solu-
removal has no approximation guarantee. It can obtain an tion in S), sj (The newly removed solution)
arbitrary bad solution subset [39]. However, in practice, it Output: HV C (The updated hypervolume contribution of
usually leads to good approximations. each solution in S)
Although the greedy algorithm has polynomial complexity, 1: for each sk ∈ S do
it still takes a long computation time when the candidate 2: w = worse(sk , sj )
set is large or the dimension is high. A lot of algorithms 3: W = {limit(t, w)|t ∈ S}
are proposed to accelerate the naı̈ve greedy algorithm. For 4: HV C(sk ) = HV C(sk ) + HV ({w}) − HV (W )
example, to accelerate the hypervolume-based greedy removal 5: end for
algorithm, Incremental Hypervolume by Slicing Objectives
(IHSO*) [36] and Incremental WFG (IWFG) [40] were pro- The worse and limit operations in Algorithm 2 are the
posed to identify the solution with the least hypervolume same as those in Section II.A. Let us explain the basic idea
contribution quickly. Some experimental results show that of Algorithm 2 using Fig. 3. When we have a solution set
these methods can significantly accelerate greedy removal S = {a, b, c, d, e} in Fig. 3, the hypervolume contribution
algorithms. of solution c is the blue area. When solution b is removed,
the hypervolume contribution of c is updated as follows.
Algorithm 1 Greedy Inclusion Hypervolume Subset Selection The worse solution w in line 2 of Algorithm 2 has the
Input: V (A set of non-dominated solutions), k (Solution maximum objective values of solutions b and c. In line 3,
subset size) firstly the limit operator changes solutions a, d and e to
Output: S (The selected subset from V ) a0 , d0 and e0 . Next, the dominated solution e0 is removed.
1: if |V | < k then Then the solution set W = {a0 , d0 } is obtained. In line 4,
2: S=V the hypervolume contribution of c is updated by adding the
3: else term HV ({w}) − HV (W ) to its original value (i.e., the blue
4: S=∅ region in Fig. 3). The added term is the joint hypervolume
5: while |S| < k do contribution of solutions b and c (i.e., the yellow region in
6: for each si in V \ S do Fig. 3). In this way, the hypervolume contribution of each
7: Calculate the hypervolume contribution of si to S solution is updated.
8: end for Since the limit process reduces the number of non-
9: p = Solution in V \ S with the largest hypervolume dominated solutions, this updated method greatly improves
contribution the speed of hypervolume-based greedy removal algorithms.
10: S = S ∪ {p} Algorithm 2 in [41] is the fastest known algorithm to update
11: end while the hypervolume contribution in any dimension.
12: end if

III. IGD AND IGD+ S UBSET S ELECTION

Hypervolume-based greedy inclusion/removal algorithms
can be accelerated by updating hypervolume contributions The Inverted Generational Distance (IGD) indicator [42]
instead of recalculating them in each iteration (i.e., by utilizing is another widely-used indicator to evaluate solution sets. It
the calculation results in the previous iteration instead of calculates the average Euclidean distance from each reference
calculating hypervolume contributions in each iteration inde- point to the nearest solution. Formally, the IGD of a solution
pendently). Guerreiro et al. [16] proposed an algorithm to set S with respect to a reference set R can be calculated using
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 5

r as the Euclidean distance. For solution a and reference point

α, since a2 − α2 is smaller than zero, we only calculate the
a' positive part (i.e., a1 − α1 ). Thus, the IGD+ distance between
a HV (W )
a and α is the green dash line between a and α. In a similar
way, we can obtain the IGD+ distance from each reference
w
b point to each solution and find the minimum IGD+ distance
d' e'
from each reference point to the solution set S (i.e., green
dash lines in Fig. 4). The IGD+ of S is the mean value of
c
these distances.
d e
0 Minimize f1 

Fig. 3. Illustration of the hypervolume contribution update method in FV- a

MOEA. In this figure, it is assumed that point b has just been removed and
the hypervolume contribution of point c is to be updated.
b

the following formula: c
v 
u d
1 X uX
IGD(S, R) = min t (si − ri )2 . (7) 
|R| s∈S
i=1 0
r∈R Minimize f1
The IGD indicator can evaluate the convergence and diver-
Fig. 4. Illustration of the calculation of IGD and IGD+.
sity of a solution set in polynomial time complexity. However,
one of the disadvantages of the IGD indicator is that it is
not Pareto compliant [43]. This means that a solution set A
which dominates another solution set B may have a worse A. IGD/IGD+ Subset Selection Problem
IGD value than B. The Inverted Generational Distance plus
(IGD+) indicator [43] overcomes this disadvantage. Instead of In contrast to hypervolume, the smaller IGD or IGD+ value
using Euclidean distance to calculate the difference between indicates a better solution set. Therefore, the IGD subset
a solution and a reference point, the IGD+ indicator uses selection problem takes the minus IGD of a solution set as
a different distance called IGD+ distance. It is based on the performance indicator g in Definition 1 while the IGD+
the dominance relation. The formula to calculate the IGD+ subset selection problem takes the minus IGD+ of a solution
distance between a solution s and a reference point r is as set. Similar to HSSP, it is impractical to obtain the optimal
follows (for minimization problems): subset unless the candidate solution set size is small.
v Although IGD is a popular indicator to evaluate the per-
um
uX formance of an EMO algorithm, the IGD subset selection
+
D (s, r) = t (max{0, si − ri })2 . (8) problem is seldom studied. The reason is that calculating
i=1 the IGD needs a reference point set R. When evaluating an
Based on the definition of the IGD+ distance, the formula algorithm on an artificial test problem, we can use the points
to calculate the IGD+ value of a solution set S with respect on the true PF of the test problem to evaluate the solution set
to a reference set R is: (since the true PF is known). However, in general, the true PF
v
um of the problem is unknown, which brings difficulties to the
+ 1 X uX calculation of IGD [44].
IGD (S, R) = min t (max{0, si − ri })2 . (9) One approach is to generate reference points from the
|R| s∈S
i=1
r∈R
estimated PF as in IGD-based EMO algorithms [45], [46].
Fig. 4 illustrates the calculation of IGD and IGD+ of a Firstly, the extreme points in the current solution set are found.
solution set. In this example, the solution set S has three Next, based on these extreme points, we specify a hyperplane
solutions a, b and c (i.e., red points in the figure). The reference in the objective space. Then, we uniformly sample reference
set R has four points α, β, γ, δ (i.e., blue points in the figure). points from the hyperplane for IGD/IGD+ calculation.
To calculate the IGD value of S, firstly, we need to find the Another approach is to use the whole candidate solution
minimum Euclidean distance from each reference point in R set as the reference point set. In EMO algorithms with a
to the solutions in the solution set S. They are shown as black bounded or unbounded external archive, some or all non-
lines in Fig. 4). Then, the IGD of S is the mean value of dominated solutions among the evaluated solutions during
these distances. To calculate IGD+ of S in Fig. 4, we should their execution are stored externally. In this paper, we assume
calculate the IGD+ distance between each reference point and such an EMO algorithm, and we use all the stored non-
each solution. Since solution b is dominated by the reference dominated solutions in the external archive as the reference
point β, bi − βi is large or equal to zero for all objectives. point set for IGD/IGD+ calculation. As shown in Table III in
Therefore, the IGD+ distance from solution β to b is the same Section V for some real-world problems, tens of thousands
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 6

of non-dominated solutions are found by a single run of an Algorithm 3. In this algorithm, mean(·) calculates the mean
EMO algorithm. However, it is often unrealistic to show such value of each item in an array and min(·, ·) takes the smaller
a large number of solutions to the decision maker. Thus, subset one between two items that have the same index in the two
selection is used to choose a pre-specified number of solutions. arrays. Algorithm 3 is for IGD subset selection. It can be easily
Subset selection is also needed in performance comparison
of EMO algorithms to compare them using solution sets of Algorithm 3 Greedy Inclusion IGD Subset Selection
the same size. In the proposed IGD/IGD+ subset selection Input: V (A set of non-dominated solutions), k (Solution
algorithms, all the obtained non-dominated solutions (i.e., all subset size)
candidate solutions) are used as the reference point set. Output: S (The selected subset from V )
The following are two advantages of IGD and IGD+ subset 1: if |V | < k then
selection over hypervolume subset selection: (i) When the 2: S=V
number of dimensions is large, calculating hypervolume is 3: else
very time-consuming since hypervolume calculation is #P- 4: S=∅
hard in the number of objectives [19]. However, the increase 5: smin = Solution in V \S with the smallest IGD(si , V )
in the computation load for IGD and IGD+ is linear with 6: S = S ∪ {smin }
respect to the number of objectives. (ii) The behavior of 7: D = Euclidean distance from each reference point to
the hypervolume indicator is difficult to explain. Choosing smin
different reference points can result in totally different optimal 8: while |S| < k do
distributions [47]. The optimal distribution of solutions on PFs 9: for each si in V \ S do
with different shapes can have a significant difference. Besides, 10: D0 = Euclidean distance from each reference point
the hypervolume optimal µ distributions in three or higher to si
dimensions are still unknown. In contrast to hypervolume, IGD 11: ci = mean(D) − mean(min(D, D0 ))
and IGD+ subset selection can be formulated as a problem to 12: end for
minimize the expected loss function [17]. From this point of 13: p = Solution in V \ S with the largest ci
view, IGD subset selection and IGD+ subset selection have a 14: D0 = Euclidean distance from each reference point
clear meaning for the decision-maker. to p
15: D = min(D, D0 )
B. IGD/IGD+ Subset Selection Algorithms 16: S = S ∪ {p}
17: end while
Similar to HSSP, we can use greedy inclusion algorithms, 18: end if
greedy removal algorithms and evolutionary algorithms to find
an approximate solution to the IGD and IGD+ subset selection
problems. In greedy algorithms, we always need to calculate changed to the algorithm for IGD+ subset selection by replac-
the IGD improvement of a solution a to the current solution ing the Euclidean distance with the IGD+ distance. Besides,
set S with respect to the reference set R. According to the we can also use the efficient IGD improvement calculation
definition of IGD, the IGD improvement of solution a to S method in IGD and IGD+ greedy removal algorithms.
can be calculated by IGD(S, R)–IGD(S ∪{a}, R). However,
this is not efficient since IGD improvement calculation needs IV. T HE P ROPOSED L AZY G REEDY A LGORITHMS
to calculate IGD twice. The time complexity of IGD improve-
A. Submodularity of Indicators
ment calculation is O(m|S||R|) for an m-objective problem
with a candidate solution set S and a reference point set R. The core idea of the proposed algorithms is to exploit
In this paper, we use a more efficient calculation method. the submodular property of the hypervolume, IGD and IGD+
When calculating IGD of a solution set, we also use an array indicators. Submodular function is a kind of set function that
D to store the distance from each reference point to the nearest satisfies diminishing returns property. Formally, the definition
solution in the solution set S. When we want to calculate the of a submodular function is as follows [6].
IGD improvement of a new solution a to S, we only need
Definition 2 (Submodular Function). A real-valued function
to calculate the distance from each reference point to a and
z(V) defined on the set of all subsets of V that satisfies
store them in a new distance array D0 . Then, we compare each
z(S1 ∪ {p}) − z(S1 ) ≤ z(S2 ∪ {p}) − z(S2 ), S2 ⊂ S1 ⊂ V,
item in D and D0 . If the item in D0 is smaller than D, we
p ∈ V − S1 is called a submodular set function.
replace the item in D with the corresponding item in D0 . The
new IGD of the solution set S ∪ {a} is the average value of Note that the submodular property is different from the non-
each item in the new array D. Finally, we subtract the new decreasing property of set functions. To further illustrate the
IGD from the original IGD to obtain the IGD improvement. difference between them, we show three types of set functions
The proposed method can reduce the complexity of IGD in Fig. 5. The hypervolume indicator is similar to z2 , which
improvement calculation to O(m|R|), which can significantly is non-decreasing (i.e., HV (Y ) ≥ HV (X), if X ⊆ Y ) and
decrease the computation time. submodular (i.e., HV C(p, X) ≥ HV C(p, Y ), if X ⊆ Y ).
The details of the IGD greedy inclusion algorithm with the z1 is submodular but it is not non-decreasing. z3 is non-
proposed efficient IGD improvement calculation are shown in decreasing but it is non-submodular.
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 7

z (V ) In
z3 1 X 1 X
∆dxi + ∆dyi
|R| |R|
i∈I1 \(I1 ∩I2 ) i∈I2 \(I1 ∩I2 )
1 X
z2 + max{∆dxi , ∆dyi },
|R|
i∈(I1 ∩I2 )

z1 if i ∈ (I1 ∩ I2 ), only the between ∆dxi and ∆dyi

P larger xone P
will be added while in i∈I1 ∆di + i∈I2 ∆dyi all ∆dxi and
  A1  A2  A3  ...  An − 2  An −1  An
∆dyi will be added together.
Fig. 5. Illustration of three types of set functions. Hence,
1 X 1 X
TABLE I ∆dxi + ∆dyi
S UMMARY OF N OTATIONS
|R| |R|
i∈I1 \(I1 ∩I2 ) i∈I2 \(I1 ∩I2 )
1 X X X
Notation Description + max{∆dxi , ∆dyi } ≤ ∆dxi + ∆dyi .
|R|
i∈(I1 ∩I2 ) i∈I1 i∈I2
A, B, X Solution sets
R Reference set 1 X
x, y Solutions that do not belong to A − IGD(A ∪ {y}) − (−IGD(A)) = ∆dyi .
|R|
i Index of the reference point in the reference set i∈I2
I1 , I 2 Sets of reference point indexes whose closest distance
to the solution set are changed after adding x and y,
y
respectively IGD(A ∪ {x} ∪ {y}) − (−IGD(A ∪ {x}))
∆dx
i , ∆di Change of the closest distance between reference point
1 X 1 X
i and the solution set by adding x and y, respectively = ∆dxi + ∆dyi
|R| |R|
i∈I1 \(I1 ∩I2 ) i∈I2 \(I1 ∩I2 )
1 X 1 X
The hypervolume indicator has been proved to be non- + max{∆dxi , ∆dyi } − ∆dxi
|R| |R|
i∈(I1 ∩I2 ) i∈I2
decreasing submodular [38]. Now, we show that the minus
1 X 1 X 1 X
IGD indicator and the minus IGD+ indicator are both non- ≤ ∆dxi + ∆dyi − ∆dxi
decreasing submodular. The minus here means to take the |R| |R| |R|
i∈I1 i∈I2 i∈I1
opposite number of the indicator value. 1 X
= ∆dyi .
Theorem 1. The minus IGD indicator is non-decreasing |R|
i∈I2
submodular.
Therefore, −IGD(A ∪ {y}) − (−IGD(A)) ≥ IGD(A ∪
Proof: Let us consider a solution set A, a reference set {x} ∪ {y}) − (−IGD(A ∪ {x})).
R and two solutions x and y such that x ∈ / A and y ∈ / A. Then, by mathematical induction, we can obtain that
By adding a solution x to A, the nearest distance from some −IGD(A ∪ {y}) − (−IGD(A)) ≥ IGD(A ∪ X ∪ {y}) −
reference point to A will be changed. Let us denote the set (−IGD(A ∪ X)),where X is a solution set and X ∩ A = ∅.
of indexes of these points as I1 , and the difference between If we let B = A ∪ X, the above formula can be rewritten
the original distance and the new distance as ∆dxi , i ∈ I1 . as −IGD(A ∪ {y}) − (−IGD(A)) ≥ IGD(B ∪ {y}) −
Similarly, by adding a solution y, the set of indexes of (−IGD(B)), A ⊂ B, which is the same as the definition of
reference points whose nearest distance are changed is denoted the submodular function. Hence, the minus IGD indicator is
as I2 , and the corresponding distance differences are denoted non-decreasing submodular.
as ∆dyi , i ∈ I2 . The notations used in the proof are summaries
Theorem 2. The minus IGD+ indicator is non-decreasing
in Table 1.
submodular.
According toPthe definition of IGD, IGD(A ∪ {x}) =
1 x x
IGD(A) − |R| i∈I1 ∆di . Since ∆di is larger than zero, the
Proof: We can prove that the minus IGD+ indicator is also
IGD indicator is non-increasing. Therefore, the minus IGD non-decreasing submodular in the same manner as the proof
indicator is non-decreasing. for the submodularity of IGD indicator. The only difference is
Similarly, by adding a solution y to a solution set A ∪ {x}, to replace the Euclidean distance with the IGD+ distance.

1 X
IGD(A ∪ {x} ∪ {y}) =IGD(A) − ∆dxi B. Algorithm Proposal
|R|
i∈I1 \(I1 ∩I2 )
In each iteration of the hypervolume greedy inclusion algo-
1 X
− ∆dyi rithm, we only need to identify the solution with the largest
|R| hypervolume contribution. However, we usually calculate the
i∈I2 \(I1 ∩I2 )
1 X hypervolume contributions of all solutions. Since the calcula-
− max{∆dxi , ∆dyi }. tion of the contribution of each solution is time-consuming,
|R|
i∈(I1 ∩I2 ) such an algorithm is not efficient.
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 8

As discussed in the last subsection, the hypervolume indi- Algorithm 4 Lazy Greedy Inclusion Hypervolume Subset
cator is non-decreasing submodular. This property can help Selection (LGI-HSS)
us avoid unnecessary calculations. The submodular property Input: V (A set of non-dominated solutions), k (Solution
of the hypervolume indicator means that the hypervolume subset size)
contribution of a solution to the selected solution subset S Output: S (The selected subset from V )
never increases as the number of solutions in S increases 1: if |V | < k then
in a greedy inclusion manner. Hence, instead of recomputing 2: S=V
the hypervolume contribution of every candidate solution in 3: else
each iteration, we can utilize the following lazy evaluation 4: S = ∅, C = ∅
mechanism. 5: for each si in V do
We use a list C to store the candidate (i.e., unselected) 6: Insert (si , HV ({si })) into C
solutions and their tentative HVC (hypervolume contribution) 7: end for
values. The tentative HVC value of each solution is initialized 8: while |S| < k do
with its hypervolume (i.e., its hypervolume contribution when 9: while C 6= ∅ do
no solution is selected). The tentative HVC value of each 10: cmax = Solution with the largest HVC in C
solution is the upper bound of its true hypervolume contri- 11: Update the HVC of cmax to S
bution. For finding the solution with the largest hypervolume 12: if cmax has the largest HVC in C then
contribution from the list, we pick the most promising so- 13: S = S ∪ {cmax }
lution with the largest tentative HVC value and recalculate 14: C = C \ {cmax }
its hypervolume contribution to the current solution subset S. 15: break
If the recalculated hypervolume contribution of this solution 16: end if
is still the largest in the list, we do not have to calculate 17: end while
the hypervolume contributions of the other solutions. This is 18: end while
because the hypervolume contribution of each solution never 19: end if
increases through the execution of greedy inclusion. In this
case (i.e., if the recalculated hypervolume contribution of
the most promising solution is still the largest in the list), solution in the solution set S. The details of the lazy greedy
we move this solution from the list to the selected solution inclusion IGD subset selection algorithm (LGI-IGDSS) is
subset S. If the recalculated hypervolume contribution of this shown in Algorithm 5. It can be changed to the algorithm for
solution is not the largest in the list, its tentative HVC value is the IGD+ indicator (LGI-IGD+SS) by using the IGD+ distance
updated with the recalculated value. Then the most promising instead of the Euclidean distance to calculate the distances
solution with the largest tentative HVC value in the list is between each reference point and each solution.
examined (i.e., its hypervolume contribution is recalculated). Note that we need to find the solution with the largest
This procedure is iterated until the recalculated hypervolume hypervolume contribution in Algorithm 4 and the largest IGD
contribution is the largest in the list. (IGD+) improvement in Algorithm 5. The priority queue
In many cases, the recalculation of the hypervolume contri- implemented by the maximum heap is used to accelerate the
bution of each solution results in the same value as or slightly procedure.
smaller value than its tentative HVC value in the list since The proposed algorithms only need one parameter k, which
the inclusion of a single solution to the solution subset S is the solution subset size. This parameter is needed in all
changes the hypervolume contributions of only its neighbors subset selection algorithms. That is, the proposed algorithms
in the objective space. Thus, the solution with the largest do not need any additional parameter.
hypervolume contribution is often found without examining The idea of the lazy evaluation was proposed by Minoux
all solutions in the list. By applying this lazy evaluation [48] to accelerate the greedy algorithm for maximizing sub-
mechanism, we can avoid a lot of unnecessary calculations in modular functions. Then, it was applied to some specific areas
hypervolume-based greedy inclusion algorithms. The details such as influence maximization problems [49] and network
of the proposed lazy greedy inclusion hypervolume subset monitoring [50]. Minoux [48] proved that if the function is
selection (LGI-HSS) algorithm are shown in Algorithm 4. submodular and the greedy solution is unique, the solution
For the IGD and IGD+ indicators, the same idea can be produced by the lazy greedy algorithm and the original
used. Since the submodularity of the IGD and IGD+ indicators greedy algorithm is identical. Since the hypervolume, IGD
has been proved in section IV.A, we can obtain that the IGD and IGD+ indicators are submodular, the proposed algorithms
(IGD+) improvement of a solution to the selected solution (i.e., Algorithm 4 and Algorithm 5) find the same subsets as
set S will never increase. Thus, we do not need to calculate the corresponding original greedy inclusion algorithms if the
other solutions if the recalculated IGD (IGD+) improvement greedy solutions are unique.
of a solution is the largest among all tentative IGD (IGD+)
improvement values. Besides, to accelerate the IGD (IGD+)
improvement calculation, we use our proposal for the standard C. An Illustrative Example
IGD greedy inclusion in Section III. An array D is used to Let us explain the proposed algorithm using a simple exam-
store the distance from each reference point to the nearest ple. Fig. 6 shows the changes of the hypervolume contribution
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 9

Algorithm 5 Lazy Greedy Inclusion IGD Subset Selection its hypervolume contribution. We assume that the recalculated
(LGI-IGDSS) HVC value is 6 as shown in Fig. 6 (iv).
Input: V (A set of non-dominated solutions), k (Solution Fig. 6 (iv) shows the list after the recalculation. Since the
subset size) recalculated HVC value is still the largest in the list, solution
Output: S (The selected subset from V ) e is moved from the list to the solution subset.
1: if |V | < k then Fig. 6 (v) shows the list after the removal of e. Solution c
2: S=V with the largest HVC value is examined.
3: else In this example, for choosing the second solution from
4: S = ∅, C = ∅ the remaining four candidates (b, c, d and e), we evaluate
5: smin = Solution in V \S with the smallest the hypervolume contributions of only the two solutions (b
IGD({si }, V ) and e). In the standard greedy inclusion algorithm, all four
6: S = S ∪ {smin } candidates are examined. In this manner, the proposed algo-
7: D0 = Distance from each reference point to smin rithm decreases the computation time of the standard greedy
8: for each si in V \ S do inclusion algorithm.
9: Insert (si , IGD({smin }, R) − IGD({smin , si }, R))
into C a (10) b(9) e(8) c(6) d (5)
(i)
10: end for
11: while |S| < k do (ii) b(9) e(8) c(6) d (5)
12: while C 6= ∅ do
13: cmax = Solution with the largest IGD improvement (iii) e(8) c(6) d (5) b(4)
in C
(iv) e(6) c(6) d (5) b(4)
14: Distance from each reference point to cmax
15: Update the IGD improvement of cmax to (v) c(6) d (5) b(4)
mean(D0 ) − mean(min(D0 , D))
16: if cmax has the largest IGD improvement in C Fig. 6. Illustration of the proposed algorithm. The values in the parentheses
then are the stored tentative HVC values.
17: S = S ∪ {cmax }
18: C = C \ {cmax }
19: D = Distance from each reference point to cmax V. E XPERIMENTS
20: D = min(D0 , D) A. Algorithms Used in Computational Experiments
21: break The proposed lazy greedy inclusion hypervolume subset
22: end if selection algorithm LGI-HSS is compared with the following
23: end while two algorithms:
24: end while 1) Standard greedy inclusion hypervolume subset selection
25: end if (GI-HSS): This is the greedy inclusion algorithm described in
Section II.C. When calculating the hypervolume contribution,
the efficient method described in Section II.A is employed.
in the list C. The value in the parentheses is the stored HVC 2) Greedy inclusion hypervolume subset selection with hy-
value of each solution to the selected subset. For illustration pervolume contribution updating (UGI-HSS): The hypervol-
purposes, the solutions in the list are sorted by the stored ume contribution updating method proposed in FV-MOEA
HVC values. However, in the actual implementation of the [41] (Algorithm 2) is used. Since Algorithm 2 is for greedy
algorithm, the sorting is not necessarily needed (especially removal, it is changed for greedy inclusion here. It is the
when the number of candidate solutions is huge). This is fastest known greedy inclusion algorithm applicable to any
because our algorithm only needs to find the most promising dimension.
candidate solution with the largest HVC value in the list. Fig. The proposed lazy greedy inclusion IGD subset selection
6 (i) shows the initial list C including five solutions a, b, c, algorithm LGI-IGDSS is compared with the standard greedy
d and e. The current solution subset is empty. In Fig. 6 (i), inclusion algorithm GI-IGDSS with the efficient IGD im-
solution a has the largest HVC value. Since the initial HVC provement evaluation method in Section III.B. The proposed
value of each solution is the true hypervolume contribution to lazy greedy inclusion IGD+ subset selection algorithm LGI-
the current empty solution subset S, no recalculation is needed. IGD+SS is compared with the standard greedy inclusion
Solution a is moved from the list to the solution subset. algorithm GI-IGD+SS with the efficient IGD+ improvement
In Fig. 6 (ii), solution b has the largest HVC value in the list evaluation method in Section III.B.
after solution a is moved. Thus, the hypervolume contribution Our main focus is the selection of a solution subset from
of b is to be recalculated. We assume that the recalculated an unbounded external archive. Since the number of solutions
HVC value is 4 as shown in Fig. 6 (iii). to be selected is much smaller than the number of candidate
Fig. 6 (iii) shows the list after the recalculation. Since the solutions: k  n = |V | in HSSP, greedy removal is not
updated HVC value of b is not the largest, we need to choose efficient. Hence, greedy removal algorithms (e.g., those with
solution e with the largest HVC value in the list and recalculate IHSO* [36] and IWFG [40]) are not compared in this paper.
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 10

B. Test Problems and Candidate Solution Sets objective space), the advantage of LGI-HSS over the other
To examine the performance of the above-mentioned subset two algorithms becomes larger. Among the four test problems
selection algorithms, we use the following four representative in Table II, all the three algorithms are fast on the I-DTLZ2
test problems with different Pareto front shapes: problem and slow on the DTLZ2 and DTLZ1 problems.
(i) DTLZ2 [51] with a triangular concave (spherical) Pareto Even when we compare our LGI-HSS algorithm with the
front. fastest known greedy inclusion algorithm UGI-HSS, LGI-HSS
(ii) Inverted DTLZ2 (I-DTLZ2 [52]) with an inverted trian- is also faster. On DTLZ2 and I-DTLZ2, when the number of
gular concave (spherical) Pareto front. objectives is not very large (i.e., five-objective problems), the
(iii) DTLZ1 [51] with a triangular linear Pareto front. difference in the average computation time between the two
(iv) DTLZ7 [51] with a disconnected Pareto front. algorithms is not large (the LGI-HSS average computation
For each test problem, we use three problem instances with time is 41-67% of that of UGI-HSS). When the number of
five, eight and ten objectives (i.e., solution subset selection objectives is larger (i.e., eight-objective and ten-objective prob-
is performed in five-, eight- and ten-dimensional objective lems), the difference in the average computation time between
spaces). Candidate solution sets are generated by sampling the two algorithms becomes larger (i.e., LGI-HSS needs only
solutions from the Pareto front of each problem instance. 6-25% of the UGI-HSS computation time). On DTLZ7, LGI-
Four different settings of the candidate solution set size are HSS needs only 4-24% of the UGI-HSS computation time.
examined: 5000, 10000, 15000 and 20000 solutions. First we On DTLZ1, the difference between LGI-HSS and UGI-HSS is
uniformly sample 100,000 solutions from the Pareto front of small: LGI-HSS needs 36-88% of the UGI-HSS computation
each problem instance. In each run of a solution subset selec- time. The reason for the small difference is that each candidate
tion algorithm, a pre-specified number of candidate solutions solution on the linear Pareto front of DTLZ1 has a similar
(i.e., 5000, 10000, 15000 or 20000 solutions) are randomly hypervolume contribution value. As a result, recalculation
selected from the generated 100,000 solutions. Computational is frequently performed in our LGI-HSS. On the contrary,
experiments are performed 11 times for each setting of the there exist large differences among hypervolume contribution
candidate solution set size for each problem instance by each values of candidate solutions on the nonlinear Pareto fronts of
subset selection algorithm. The number of solutions to be DTLZ2, I-DTLZ2 and DTLZ7. This leads to a less frequent
selected is specified as 100. Thus, our problem is to select update of their hypervolume contribution values (i.e., high
100 solutions from 5000, 10000, 15000 and 20000 candidate efficiency of LGI-HSS).
solutions to maximize the hypervolume value, and to minimize From the three columns by HSS in Table I, we can also
the IGD and IGD+ values of the selected 100 solutions. observe that the average computation time of each algorithm
did not severely increase with the increase in the number of
objectives (i.e., with the increase in the dimensionality of the
C. Experimental Settings objective space) for DTLZ7 and I-DTLZ2. In some cases, the
In each hypervolume subset selection algorithm, the ref- average computation time decreased with the increase in the
erence point for hypervolume (contribution) calculation is number of objectives. The reasons are as follows. Firstly, the
specified as (1.1, 1.1, ..., 1.1) for all test problems independent WFG algorithm is used in the three algorithms to calculate the
of the number of objectives. We use the WFG algorithm1 hypervolume contribution of each solution. The computation
[35] for hypervolume calculation in each subset selection time of hypervolume contribution does not increase severely as
algorithm with the hypervolume indicator. All subset selection the number of objectives increases. Besides, the total number
algorithms are coded by MatlabR2018a. The computation time of solution evaluations needed for LGI-HSS will decrease on
of each run is measured on an Intel Core i7-8700K CPU with some problems as the number of objectives increases. This
16GB of RAM, running in Windows 10. is because the difference in the hypervolume contribution
values of the candidate solutions increases with the number
D. Comparison of Hypervolume Subset Selection Algorithms of objectives, which leads to the decrease in the number
of updates of the hypervolume contribution value of each
The average run time of each hypervolume subset selection
solution.
algorithm on each test problem is summarized in Table II (in
the columns labelled as GI-HSS, UGI-HSS and LGI-HSS).
Since the subset obtained by three algorithms on the same E. Comparison of IGD/IGD+ Subset Selection Algorithms
data set are the same (i.e., the proposed algorithm does not
As can be observed from the last four columns of Table
change the performance), we only show the run time of each
II, the proposed LGI-IGDSS and LGI-IGD+SS are much
algorithm.
faster than the standard greedy algorithms (i.e., GI-IGDSS and
Compared with the standard GI-HSS algorithm, we can see
GI-IGD+SS). Compared with the GI-IGDSS algorithm, our
that our LGI-HSS algorithm can reduce the computation time
LGI-IGDSS algorithm needs only 6-10% computation time.
by 91% to 99%. That is, the computation time of our LGI-HSS
Compared with the GI-IGD+SS algorithm, our LGI-IGD+SS
is only 1-9% of that of GI-HSS. By the increase in the number
algorithm needs only 6-9% computation time.
of objectives (i.e., by the increase in the dimensionality of the
Different from the hypervolume subset selection algorithms,
1 The code of the WFG algorithm is available from the computation time of the IGD and IGD+ subset selection
http://www.wfg.csse.uwa.edu.au/hypervolume/#code. algorithms does not have a large difference among different
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 11

TABLE II
AVERAGE C OMPUTATION T IME ( IN S ECONDS ) ON D IFFERENT P ROBLEMS OVER 11 RUNS . T HE B EST R ESULTS ARE H IGHLIGHTED BY B OLD .

Problem Candidate Solutions GI-HSS UGI-HSS LGI-HSS GI-IGDSS LGI-IGDSS GI-IGD+SS LGI-IGD+SS
5,000 46.9 10.7 4.3 54.9 4.5 246.2 20.9
10,000 100.2 21.5 8.9 137.5 10.6 980.5 81.8
Five-Objective DTLZ2
15,000 164.7 31.2 14.5 248.2 18.6 2197.5 179.8
20,000 243.2 40.7 21.1 398.5 29.1 3866.0 311.6
5,000 413.8 75.4 18.8 53.4 4.6 253.4 19.5
10,000 726.0 132.7 32.5 149.2 12.0 1015.4 74.4
Eight-Objective DTLZ2
15,000 1015.2 188.3 47.3 284.7 21.8 2294.9 165.7
20,000 1251.2 245.6 61.3 1440.1 102.8 4088.7 292.4
5,000 3521.7 540.6 138.1 55.8 5.1 263.9 20.1
10,000 5569.6 830.8 217.4 159.0 14.2 1054.7 76.8
Ten-Objective DTLZ2
15,000 7646.2 1277.6 291.8 995.2 83.0 2356.0 171.4
20,000 9116.0 1552.1 357.5 1665.6 124.5 4175.9 297.2
5,000 35.7 4.1 1.8 52.7 4.3 247.3 18.2
10,000 82.7 8.2 4.0 137.9 10.7 979.5 71.1
Five-Objective I-DTLZ2
15,000 140.4 12.0 6.9 245.9 18.3 2190.2 155.0
20,000 219.8 15.5 10.4 410.4 30.5 3890.5 276.7
5,000 46.0 5.4 0.8 59.0 5.0 255.7 17.1
10,000 108.5 11.2 1.7 158.9 12.7 1017.6 65.7
Eight-Objective I-DTLZ2
15,000 190.2 18.7 2.9 302.3 22.8 2302.6 145.5
20,000 283.8 24.3 4.2 1649.3 115.3 4070.0 254.5
5,000 43.9 8.4 0.5 55.7 5.2 263.4 17.9
10,000 108.2 14.9 1.2 157.5 14.1 1047.2 67.3
Ten-Objective I-DTLZ2
15,000 195.5 22.0 2.2 991.2 82.7 2353.0 148.9
20,000 304.1 33.9 3.3 1669.7 135.9 4162.9 272.8
5,000 48.6 17.9 6.5 49.3 3.9 249.8 18.2
10,000 108.3 35.6 14.7 131.0 9.6 985.1 70.8
Five -Objective DTLZ1
15,000 179.4 53.7 24.4 247.5 17.6 2202.8 154.3
20,000 262.8 70.9 36.2 392.4 27.5 3886.7 270.3
5,000 362.8 61.6 35.7 53.9 4.2 255.9 16.2
10,000 777.6 133.9 79.4 151.5 10.9 1016.0 61.6
Eight-Objective DTLZ1
15,000 1239.6 201.7 126.3 277.9 19.4 2282.9 134.4
20,000 1690.3 285.8 175.5 1435.8 92.1 4070.5 238.6
5,000 2436.2 278.5 228.2 55.6 4.3 262.5 16.0
10,000 5455.5 623.8 505.4 158.0 12.2 1045.2 65.9
Ten-Objective DTLZ1
15,000 8504.7 1038.3 809.7 995.7 69.3 2342.7 141.7
20,000 11552.2 1343.0 1176.0 1679.3 118.9 4144.4 260.1
5,000 40.1 13.0 2.4 48.5 4.3 247.7 20.1
10,000 91.9 25.8 5.5 131.5 11.1 978.8 76.5
Five-Objective DTLZ7
15,000 155.2 40.4 9.0 245.9 20.2 2198.2 172.0
20,000 228.5 56.7 13.3 392.7 31.8 4067.4 320.4
5,000 52.8 26.5 1.5 52.6 4.9 269.5 19.3
10,000 122.5 48.9 3.3 147.1 13.1 1108.5 69.5
Eight-Objective DTLZ7
15,000 208.7 73.6 5.6 289.1 24.5 2281.5 152.0
20,000 314.7 97.4 8.4 1439.8 114.9 4046.7 269.6
5,000 57.9 27.6 1.1 55.6 5.4 262.7 22.8
10,000 136.3 55.2 2.5 159.0 14.8 1041.5 90.0
Ten-Objective DTLZ7
15,000 237.2 82.3 4.3 985.3 84.2 2343.5 202.9
20,000 361.7 110.9 6.5 1715.5 143.5 4144.0 362.3

problems. One interesting observation is that the computation F. Number of Solution Evaluations
time of LGI-IGD+SS does not increase with the number of To further examine the effectiveness of the proposed LGI-
objectives whereas that of LGI-IGDSS clearly increases with HSS, we monitor the number of solution evaluations (i.e.,
the number of objectives. This is because the difference in the contribution calculations) to select each solution used by GI-
IGD+ contribution values of the candidate solutions increases HSS and LGI-HSS. In GI-HSS, all the remaining solutions
with the number of objectives, which leads to the decrease need to be re-evaluated during each iteration. For example,
in the number of updates of the IGD+ contribution value of when the candidate solution set size is 10,000, 10,000−(i − 1)
each solution. However, the difference in the IGD contribution solutions are examined to select the i-th solution in GI-
values of the candidate solutions does not clearly increase with HSS. However, LGI-HSS can choose the same i-th solution
the number of objectives. This observation suggests that the by evaluating only a small number of solutions. In Fig. 7,
IGD+ indicator is more similar to the hypervolume indicator we show the number of examined solutions to choose each
than the IGD indicator, which was suggested by the optimal solution (i.e., the i-th solution for i = 1, 2, ..., 100) in a single
distributions of solutions for each indicator [53]. run of LGI-HSS on the eight-objective DTLZ2 and I-DTLZ2
problems where the candidate solution size is 10,000. The
single run with the median computation time among 11 runs
on each problem is selected in Fig. 7. For comparison, the
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 12

results by GI-HSS are also shown, which is the line specified In the worst case, the number of solution evaluations in
by 10,000−(i − 1). For selecting the first solution, both the proposed algorithms is the same as the standard greedy
algorithms examine all the given 10,000 solutions. After that, algorithms (i.e., the worst time complexity of the proposed
the proposed LGI-HSS algorithm can significantly decrease algorithms is the same as that of the standard greedy al-
the number of solution evaluations. On the eight-objective gorithms). However, as shown in Figs. 7-9, the number of
DTLZ2 problem, the number of solution evaluations decreases actually evaluated solutions is much smaller than the worst-
with fluctuation as the number of selected solutions increases. case upper bound (which is shown by the dotted green line in
On the eight-objective I-DTLZ2 problem, the average number each figure).
of solution evaluations needed to select each solution is much In order to examine the effect of the number of objectives
smaller than that on the eight-objective DTLZ2 problem. This on the efficiency of the proposed algorithm, we show exper-
difference is the reason for the difference in the results in Table imental results by the proposed three algorithms on the five-
II for these two problems. objective and ten-objective I-DTLZ2 problems in Figs. 10-12.
In the same manner as in Fig. 7, we show the results by As in Figs. 7-9, a single run with the median computation time
GI-IGDSS and LGI-IGDSS in Fig. 8, and the results by GI- is selected for each algorithm on each test problem. In Fig. 10
IGD+SS and LGI-IGD+SS in Fig. 9. We can observe from by LGI-HSS, much fewer solutions are examined for the ten-
these two figures that the number of solution evaluations objective problem than the five-objective problem. As a result,
needed to select the first 50 solutions is much larger than the average computation time on the ten-objective I-DTLZ2
that for the remaining 50 solutions. We can also see that the problem by LGI-HSS in Table I was shorter than that on the
difference in the results on the two problems in these two five-objective I-DTLZ2 problem. In Fig. 11 by LGI-IGDSS,
figures is much smaller than that in Fig. 7. This is the reason the difference in the number of solution evaluations between
why similar results were obtained for all problems by LGI- the two problems is not so large if compared with Fig. 10. As a
IGDSS and LGI-IGD+SS. result, the average computation time by LGI-IGDSS increased
with the increase in the number of objectives in Table II. The
results by LGI-IGD+SS in Fig. 12 are between Fig. 10 and
Fig. 11.

G. Size of Candidate Solution Sets

In the previous subsection, 100 solutions were selected
from 5000-20000 solutions of test problems with five to ten
objectives. Our algorithms are proposed for choosing a pre-
specified number of solutions from an unbounded external
Fig. 7. The number of solution evaluations to select each solution used by archive. In order to show the size of the subset selection
GI-HSS and LGI-HSS. problem in this scenario, we monitor the number of non-
dominated solutions among all the examined solutions by
an EMO algorithm on the four test problems with eight
objectives. As an EMO algorithm, we use NSGA-II [54] and
NSGA-III [25]. The default parameter settings in [54] are
used in the algorithms (e.g., the population size is specified as
156). The number of non-dominated solutions is counted at the
200th, 400th, 600th, 800th and 1000th generations. Average
results over 11 runs are shown in Fig. 13. From these figures,
we can see that hundreds of thousands of non-dominated
solutions can be obtained by EMO algorithms for many-
Fig. 8. The number of solution evaluations to select each solution used by
objective problems. This observation supports the necessity
GI-IGDSS and LGI-IGDSS. of highly-efficient subset selection algorithms.

H. Performance on real-world solution sets

We examine the performance of the three HSS algorithms on
four real-world problems (i.e., car side impact design problem
[55], conceptual marine design problem [56], water resource
planning problem [57] and car cab design problem [25]). We
apply NSGA-II [54] and NSGA-III [25] with an unbounded
external archive to each problem. The population size is
specified as 220 for the four-objective problems, 182 for the
six-objective problem, and 210 for the nice-objective problem.
Fig. 9. The number of solution evaluations to select each solution used by After the 200th generation of each algorithm, non-dominated
GI-IGD+SS and LGI-IGD+SS. solutions in the archive are used as candidate solution sets.
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 13

(a) NSGA-II.
Fig. 10. The number of solution evaluations to select each solution used by
LGI-HSS on five-objective and ten-objective I-DTLZ2 problems.

(b) NSGA-III.

Fig. 13. The average number of non-dominated solutions among all the
examined solutions at each generation.

TABLE III
Fig. 11. The number of solution evaluations to select each solution used by AVERAGE C OMPUTATION T IME ( IN S ECONDS ) ON R EAL - WORLD
LGI-IGDSS on five-objective and ten-objective I-DTLZ2 problems. S OLUTION S ETS OVER 11 RUNS . T HE B EST R ESULTS ARE H IGHLIGHTED
BY B OLD .

Solution Set Size # Obj. GI- UGI- LGI-

In this manner, we generate eight candidate solution sets. HSS HSS HSS
Then, each hypervolume-based subset selection algorithm is NSGA-II on the car side 10913 4 84.0 10.2 7.0
applied to each candidate solution set to select 100 solutions. impact design problem
This experiment is iterated 11 time. The average size of the NSGA-III on the car side 11453 4 92.7 13.0 8.3
impact design problem
candidate solution sets and the average computation time of NSGA-II on the conceptual 8549 4 62.4 6.1 4.8
each subset selection algorithm are shown in Table III. marine design problem
NSGA-III on the concep- 10328 4 81.6 8.0 6.5
We can see from Table III that the proposed algorithms tual marine design problem
always achieve the best results (i.e., the shortest average NSGA-II on the water re- 12284 6 117.3 22.1 9.8
computation time) for all the eight settings. Especially when source planning problem
NSGA-III on the water re- 19985 6 238.1 38.3 23.2
the number of objectives is large (e.g., the nine-objective source planning problem
car cab design problem), the proposed algorithm can signif- NSGA-II on the car cab de- 22196 9 412.5 81.5 15.3
icantly decrease the computation time for subset selection. sign problem
NSGA-III on the car cab 23958 9 516.3 97.6 21.4
This is consistent with the experimental results on artificially design problem
generated solution sets. Similar significant improvement by
the proposed algorithms in the computation time is also
observed for subset selection based on the IGD and IGD+
VI. C ONCLUDING R EMARKS
indicators whereas they are not included due to the paper
length limitation. In this paper, we proposed efficient greedy inclusion al-
gorithms to select a small number of solutions from a large
candidate solution set for hypervolume maximization, and IGD
and IGD+ minimization. The proposed algorithms are based
on the submodular property of the three indicators. The core
idea of these algorithms is to use the submodular property
to avoid unnecessary contribution calculations. The proposed
lazy greedy algorithm finds the same solution subset as the
standard greedy inclusion algorithm for each indicator since
our algorithm does not change the basic framework of greedy
inclusion. Our experimental results on the four test problems
(DTLZ1, DTLZ2, DTLZ7 and Inverted DTLZ2) with five,
eight and ten objectives showed that the proposed three greedy
Fig. 12. The number of solution evaluations to select each solution used by algorithms are much faster than the standard greedy inclusion
LGI-IGD+SS on five-objective and ten-objective I-DTLZ2 problems. algorithms. Besides, the proposed LGI-HSS is much faster
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 14

than the state-of-the-art fast greedy inclusion algorithm. [14] K. Bringmann, T. Friedrich, and P. Klitzke, “Generic postprocessing via
Our experimental results clearly showed that the proposed subset selection for hypervolume and epsilon-indicator,” in International
Conference on Parallel Problem Solving from Nature. Springer, 2014,
idea can drastically decrease the computation time (e.g., to 6- pp. 518–527.
9% of the computation time by the standard greedy inclusion [15] T. Kuhn, C. M. Fonseca, L. Paquete, S. Ruzika, M. M. Duarte, and
algorithm for IGD+). The proposed idea is applicable to J. R. Figueira, “Hypervolume subset selection in two dimensions:
Formulations and algorithms,” Evolutionary Computation, vol. 24, no. 3,
any performance indicator with the submodular property such pp. 411–425, 2016.
as hypervolume, IGD and IGD+. However, when we use [16] A. P. Guerreiro and C. M. Fonseca, “Computing and updating hyper-
hypervolume approximation instead of exact calculation, the volume contributions in up to four dimensions,” IEEE Transactions on
Evolutionary Computation, vol. 22, no. 3, pp. 449–463, 2017.
calculated hypervolume indicator is not strictly submodular. [17] H. Ishibuchi, L. M. Pang, and K. Shang, “Solution subset selection
One interesting future research topic is to apply the proposed for final decision making in evolutionary multi-objective optimization,”
idea to approximate hypervolume subset selection algorithms 2020.
[18] K. Bringmann, T. Friedrich, and P. Klitzke, “Two-dimensional subset
in order to examine the quality of obtained subsets by lazy selection for hypervolume and epsilon-indicator,” in Proceedings of the
approximate algorithms. Another interesting research direction 2014 Annual Conference on Genetic and Evolutionary Computation,
is to accelerate the proposed algorithms by further explore 2014, pp. 589–596.
[19] K. Bringmann and T. Friedrich, “Approximating the volume of unions
the special properties of these indicators. Besides, in distance- and intersections of high-dimensional geometric objects,” Computational
based greedy subset selection [11], the distance of each Geometry, vol. 43, no. 6-7, pp. 601–610, 2010.
candidate solution to the selected set does not increase by [20] M. Li and X. Yao, “An empirical investigation of the optimality
and monotonicity properties of multiobjective archiving methods,” in
the addition of a new solution to the selected solution set. Evolutionary Multi-Criterion Optimization, K. Deb, E. Goodman, C. A.
Thus, using similar lazy idea to reduce the number of distance Coello Coello, K. Klamroth, K. Miettinen, S. Mostaghim, and P. Reed,
calculations is also a promising future work. Eds. Cham: Springer International Publishing, 2019, pp. 15–26.
[21] A. Liefooghe, F. Daolio, S. Verel, B. Derbel, H. Aguirre, and
The source code of the proposed lazy greedy K. Tanaka, “Landscape-aware performance prediction for evolutionary
subset selection algorithms is available from multi-objective optimization,” IEEE Transactions on Evolutionary Com-
https://github.com/weiyuchen1999/LGISS. putation, pp. 1–1, 2019.
[22] L. M. Pang, H. Ishibuchi, and K. Shang, “Algorithm configurations of
moea/d with an unbounded external archive,” 2020.
R EFERENCES [23] Y. Nan, K. Shang, H. Ishibuchi, and L. He, “Reverse strategy for non-
dominated archiving,” IEEE Access, vol. 8, pp. 119 458–119 469, 2020.
[1] M. Ehrgott, Multicriteria Optimization. Springer Science & Business [24] R. Tanabe and H. Ishibuchi, “An analysis of control parameters of
Media, 2005, vol. 491. moea/d under two different optimization scenarios,” Applied Soft Com-
[2] E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, and V. G. da puting, vol. 70, pp. 22 – 40, 2018.
Fonseca, “Performance assessment of multiobjective optimizers: an [25] K. Deb and H. Jain, “An evolutionary many-objective optimization
analysis and review,” IEEE Transactions on Evolutionary Computation, algorithm using reference-point-based nondominated sorting approach,
vol. 7, no. 2, pp. 117–132, 2003. part I: Solving problems with box constraints,” IEEE Transactions on
[3] H. Ishibuchi, Y. Setoguchi, H. Masuda, and Y. Nojima, “How to compare Evolutionary Computation, vol. 18, no. 4, pp. 577–601, 2014.
many-objective algorithms under different settings of population and [26] W. Chen, H. Ishibuchi, and K. Shang, “Lazy greedy hypervolume subset
archive sizes,” in 2016 IEEE Congress on Evolutionary Computation, selection from large candidate solution sets,” in 2020 IEEE Congress on
2016, pp. 1149–1156. Evolutionary Computation, 2020, pp. 1149–1156.
[4] B. K. Natarajan, “Sparse approximate solutions to linear systems,” SIAM [27] J. D. Knowles, D. W. Corne, and M. Fleischer, “Bounded archiving
Journal on Computing, vol. 24, no. 2, pp. 227–234, 1995. using the lebesgue measure,” in 2003 IEEE Congress on Evolutionary
[5] G. Davis, “Adaptive greedy approximations,” Constructive Approxima- Computation., vol. 4. IEEE, 2003, pp. 2490–2497.
tion, vol. 13, no. 1, pp. 57–98, 1997. [28] E. Zitzler and L. Thiele, “Multiobjective optimization using evolutionary
[6] G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher, “An analysis of ap- algorithms-a comparative case study,” in International Conference on
proximations for maximizing submodular set functions-I,” Mathematical Parallel Problem Solving from Nature. Springer, 1998, pp. 292–301.
Programming, vol. 14, no. 1, pp. 265–294, 1978. [29] K. Shang, H. Ishibuchi, L. He, and L. M. Pang, “A survey on the
[7] L. Bradstreet, L. While, and L. Barone, “Incrementally maximising hypervolume indicator in evolutionary multi-objective optimization,”
hypervolume for selection in multi-objective evolutionary algorithms,” IEEE Transactions on Evolutionary Computation, pp. 1–1, 2020.
in 2007 IEEE Congress on Evolutionary Computation, 2007, pp. 3203– [30] J. J. Durillo, A. J. Nebro, and E. Alba, “The jmetal framework for multi-
3210. objective optimization: Design and architecture,” in 2010 IEEE Congress
[8] C. Qian, Y. Yu, and Z.-H. Zhou, “Subset selection by pareto opti- on Evolutionary Computation. IEEE, 2010, pp. 1–8.
mization,” in Advances in Neural Information Processing Systems 28, [31] J. J. Durillo and A. J. Nebro, “jmetal: A java framework for multi-
C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, objective optimization,” Advances in Engineering Software, vol. 42,
Eds. Curran Associates, Inc., 2015, pp. 1774–1782. no. 10, pp. 760–771, 2011.
[9] H. Ishibuchi, H. Masuda, and Y. Nojima, “Selecting a small number [32] M. H. Overmars and C. Yap, “New upper bounds in klee’s measure
of non-dominated solutions to be presented to the decision maker,” in problem,” in 29th Annual Symposium on Foundations of Computer
2014 IEEE International Conference on Systems, Man, and Cybernetics Science, Oct 1988, pp. 550–556.
(SMC), 2014, pp. 3816–3821. [33] M. H. Overmars and C.-K. Yap, “New upper bounds in klee’s measure
[10] R. Tanabe, H. Ishibuchi, and A. Oyama, “Benchmarking multi- and problem,” SIAM Journal on Computing, vol. 20, no. 6, pp. 1034–1045,
many-objective evolutionary algorithms under two optimization scenar- 1991.
ios,” IEEE Access, vol. 5, pp. 19 597–19 619, 2017. [34] N. Beume, “S-metric calculation by considering dominated hypervolume
[11] H. K. Singh, K. S. Bhattacharjee, and T. Ray, “Distance-based subset as klee’s measure problem,” Evolutionary Computation, vol. 17, no. 4,
selection for benchmarking in evolutionary multi/many-objective opti- pp. 477–492, 2009.
mization,” IEEE Transactions on Evolutionary Computation, vol. 23, [35] L. While, L. Bradstreet, and L. Barone, “A fast way of calculating
no. 5, pp. 904–912, Oct 2019. exact hypervolumes,” IEEE Transactions on Evolutionary Computation,
[12] W. Chen, H. Ishibuchi, and K. Shang, “Modified distance-based subset vol. 16, no. 1, pp. 86–95, 2011.
selection for evolutionary multi-objective optimization algorithms,” in [36] L. Bradstreet, L. While, and L. Barone, “A fast incremental hypervolume
2020 IEEE Congress on Evolutionary Computation., 2020. algorithm,” IEEE Transactions on Evolutionary Computation, vol. 12,
[13] J. Bader and E. Zitzler, “HypE: An algorithm for fast hypervolume-based no. 6, pp. 714–723, 2008.
many-objective optimization,” Evolutionary Computation, vol. 19, no. 1, [37] G. Rote, K. Buchin, K. Bringmann, S. Cabello, and M. Emmerich, “Se-
pp. 45–76, 2011. lecting k points that maximize the convex hull volume,” in Proceedings
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. XX, NO. XX, AUGUST 2020 15

of the 19th Japan Conference on Discrete and Computational Geometry,

Graphs, and Games, 2016, pp. 58–60.
[38] T. Ulrich and L. Thiele, “Bounding the effectiveness of hypervolume-
based (µ+ λ)-archiving algorithms,” in International Conference on
Learning and Intelligent Optimization. Springer, 2012, pp. 235–249.
[39] K. Bringmann and T. Friedrich, “An efficient algorithm for computing
hypervolume contributions,” Evolutionary Computation, vol. 18, no. 3,
pp. 383–402, 2010.
[40] W. Cox and L. While, “Improving the IWFG algorithm for calculating
incremental hypervolume,” in 2016 IEEE Congress on Evolutionary
Computation. IEEE, 2016, pp. 3969–3976.
[41] S. Jiang, J. Zhang, Y.-S. Ong, A. N. Zhang, and P. S. Tan, “A
simple and fast hypervolume indicator-based multiobjective evolutionary
algorithm,” IEEE Transactions on Cybernetics, vol. 45, no. 10, pp. 2202–
2213, 2014.
[42] C. A. Coello Coello and M. Reyes Sierra, “A study of the parallelization
of a coevolutionary multi-objective evolutionary algorithm,” in Mexican
International Conference on Artificial Intelligence, 2004.
[43] H. Ishibuchi, H. Masuda, Y. Tanigaki, and Y. Nojima, “Modified
distance calculation in generational distance and inverted generational
distance,” in Evolutionary Multi-Criterion Optimization, A. Gaspar-
Cunha, C. Henggeler Antunes, and C. C. Coello, Eds. Cham: Springer
International Publishing, 2015, pp. 110–125.
[44] H. Ishibuchi, R. Imada, Y. Setoguchi, and Y. Nojima, “Reference point
specification in inverted generational distance for triangular linear pareto
front,” IEEE Transactions on Evolutionary Computation, vol. 22, no. 6,
pp. 961–975, 2018.
[45] E. M. Lopez and C. A. C. Coello, “IGD+-EMOA: A multi-objective
evolutionary algorithm based on IGD+,” in 2016 IEEE Congress on
Evolutionary Computation, 2016, pp. 999–1006.
[46] Y. Sun, G. G. Yen, and Z. Yi, “IGD indicator-based evolutionary algo-
rithm for many-objective optimization problems,” IEEE Transactions on
Evolutionary Computation, vol. 23, no. 2, pp. 173–187, 2019.
[47] H. Ishibuchi, R. Imada, Y. Setoguchi, and Y. Nojima, “How to specify
a reference point in hypervolume calculation for fair performance
comparison,” Evolutionary Computation, vol. 26, no. 3, pp. 411–440,
2018.
[48] M. Minoux, “Accelerated greedy algorithms for maximizing submodular
set functions,” in Optimization Techniques. Springer, 1978, pp. 234–
243.
[49] J. Leskovec, A. Krause, C. Guestrin, C. Faloutsos, J. VanBriesen,
and N. Glance, “Cost-effective outbreak detection in networks,” in
Proceedings of the 13th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, 2007, pp. 420–429.
[50] M. Gomez-Rodriguez, J. Leskovec, and A. Krause, “Inferring networks
of diffusion and influence,” ACM Transactions on Knowledge Discovery
from Data, vol. 5, no. 4, Feb. 2012.
[51] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler, “Scalable multi-
objective optimization test problems,” in 2002 IEEE Congress on
Evolutionary Computation., vol. 1, May 2002, pp. 825–830 vol.1.
[52] H. Jain and K. Deb, “An evolutionary many-objective optimization
algorithm using reference-point based nondominated sorting approach,
part II: Handling constraints and extending to an adaptive approach,”
IEEE Transactions on Evolutionary Computation, vol. 18, no. 4, pp.
602–622, Aug 2014.
[53] H. Ishibuchi, R. Imada, N. Masuyama, and Y. Nojima, “Comparison
of hypervolume, IGD and IGD+ from the viewpoint of optimal dis-
tributions of solutions,” in Evolutionary Multi-Criterion Optimization,
K. Deb, E. Goodman, C. A. Coello Coello, K. Klamroth, K. Miettinen,
S. Mostaghim, and P. Reed, Eds. Cham: Springer International
Publishing, 2019, pp. 332–345.
[54] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist
multiobjective genetic algorithm: NSGA-II,” IEEE Transactions on
Evolutionary Computation, vol. 6, no. 2, pp. 182–197, 2002.
[55] H. Jain and K. Deb, “An evolutionary many-objective optimization
algorithm using reference-point based nondominated sorting approach,
part ii: Handling constraints and extending to an adaptive approach,”
IEEE Transactions on Evolutionary Computation, vol. 18, no. 4, pp.
602–622, 2014.
[56] M. G. Parsons and R. L. Scott, “Formulation of multicriterion design
optimization problems for solution with scalar numerical optimization
methods,” Journal of Ship Research, vol. 48, no. 1, pp. 61–76, 2004.
[57] T. Ray, K. Tai, and K. C. Seow, “Multiobjective design optimization by
an evolutionary algorithm,” Engineering Optimization, vol. 33, no. 4,
pp. 399–424, 2001.