Accepted Manuscript

Approximate non-dominated sorting for evolutionary many-objective

optimization

Xingyi Zhang, Ye Tian, Yaochu Jin

PII: S0020-0255(16)30427-3
DOI: 10.1016/j.ins.2016.06.007
Reference: INS 12280

To appear in: Information Sciences

Received date: 17 November 2015

Revised date: 9 April 2016
Accepted date: 8 June 2016

Please cite this article as: Xingyi Zhang, Ye Tian, Yaochu Jin, Approximate non-dominated sorting for
evolutionary many-objective optimization, Information Sciences (2016), doi: 10.1016/j.ins.2016.06.007

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service
to our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please
note that during the production process errors may be discovered which could affect the content, and
all legal disclaimers that apply to the journal pertain.
 ACCEPTED MANUSCRIPT

Approximate non-dominated sorting for

evolutionary many-objective optimization
Xingyi Zhanga , Ye Tiana , Yaochu Jinb,c,∗

T
a
Institute of Bio-inspired Intelligence and Mining Knowledge, School of Computer Science and

IP
Technology, Anhui University, Hefei 230039, China
b
Department of Computing, University of Surrey, Guildford, Surrey, GU2 7XH, United
Kingdom

CR
c
College of Information Sciences and Technology, Donghua University, Shanghai 201620, China

Abstract
US
Non-dominated sorting has widely been adopted and shown to be very
AN
effective in dominance based evolutionary multi-objective optimization
where the number of objectives is two or three. In dealing with many-
objective optimization problems, where the number of objectives is more
than three, however, the effectiveness of dominance based evolutionary
M

algorithms will seriously degrade and the computational complexity of

non-dominated sorting will considerably increase. In this paper, we pro-
pose an approximate non-dominated sorting algorithm for many-objective
ED

optimization, where the dominance relationship between two solutions

is determined by a maximum of three objective comparisons on top of a
sorted population according to one of the objectives. The time complex-
ity of the proposed approximate sorting is independent of the number of
PT

objectives, which significantly enhances the time efficiency, in particular

when the number of objectives is large. To examine the influence of inac-
curate sorting on the search performance, the proposed approximate sort-
CE

ing method is embedded in three popular dominance based evolutionary

many-objective optimization algorithms. Our experimental results on 16
widely used many-objective optimization benchmark problems suggest
AC

Corresponding author.
∗

Email addresses: [email protected] (Xingyi Zhang),

field910921@[email protected] (Ye Tian), [email protected] (Yaochu
Jin)

Preprint submitted to Information Sciences June 13, 2016

 ACCEPTED MANUSCRIPT

that the approximate non-dominated sorting method is not only compu-

tationally much more efficient than the state-of-the-art of accurate non-
dominated sorting methods, but also leads to better search performance
on most test problems used in this study.

T
Keywords: Evolutionary multi-objective optimization, Computational
complexity, Non-dominated sorting, Approximation, Many-objective

IP
optimization

CR
1. Introduction
A multi-objective optimization problem (MOP) refers to a problem in-
volving multiple conflicting objectives. Mathematically, a continuous MOP
can be formulated as follows:

minimize
US
F (X) = (f1 (X), . . . , fM (X))
(1)
AN
subject to X∈Ω
where Ω ⊆ Rn is the decision space, F : Ω → RM consists of M objectives
and RM is called the objective space. Due to the conflicting nature of the
M

objectives, there is usually no single optimal solution but a set of trade-off

solutions, often known as Pareto-optimal solutions to the MOP.
Developing efficient and effective optimization algorithms for solving
ED

MOPs have long been regarded as a challenging issue in artificial intelli-

gence [6, 17, 21, 33, 36, 43, 50], despite that a large number of methods have
been developed, such as the traditional mathematical programming meth-
ods, local search techniques and evolutionary algorithms (EAs). Actually,
PT

swarm intelligence methods have long been recognized as promising ap-

proaches to solve computational hard problems, e.g., particle swarm op-
timization [30, 31] and negatively correlated search [44]. Recently, much
CE

attention has been paid to an even more challenging subset of MOPs that
have more than three conflicting objectives. Such MOPs are referred to as
many-objective optimization problems (MaOPs) in the evolutionary com-
AC

putation community, which are commonly seen in real-world applications

[5, 19, 22].
Over the past few years, many evolutionary multi-objective optimiza-
tion algorithms (MOEAs) have been proposed to tackle MaOPs, e.g., the
indicator based evolutionary algorithm (IBEA) [62], the multi-objective
evolutionary algorithm based on decomposition (MOEA/D) [57], the S

2
 ACCEPTED MANUSCRIPT

metric selection based evolutionary multi-objective optimization algorithm

(SMS-EMOA) [2], and the bi-goal evolution for many-objective optimiza-
tion problems (BiGE) [29].
Despite their huge success in solving MOPs, most popular MOEAs will
encounter various difficulties in solving MaOPs [27, 34, 35]. For domi-

T
nance based MOEAs, such as NSGA-II and SPEA2 to solve MaOPs, two
major difficulties arise. First, as the number of objectives increases, it be-

IP
comes harder to distinguish the quality of solutions based solely on dom-
inance comparisons, resulting in a dramatically decreased selection pres-

CR
sure for the population to converge towards the Pareto front [5, 23, 25].
Second, the runtime consumed by non-dominated sorting increases con-
siderably, seriously decreasing the computational efficiency of MOEAs in

US
solving MaOPs. It is worth noting that it maybe still work if the objec-
tives are handled separately by different populations, e.g., by using the
multiple populations for multiple objectives (MPMO) framework that is
different from NSGA-II or MOEA/D [56].
AN
Most on-going research on many-objective optimization focuses on ad-
dressing the first difficulty. Generally speaking, two main streams of ideas
have been developed. The first is to modify the definition of the tradi-
M

tional Pareto dominance so that a stronger dominance relation will be

adopted for better distinguishing the quality of solutions in a popula-
tion. Examples of such modified dominance include -dominance [26],
ED

L-optimality [64], fuzzy dominance [47], preference order ranking [14]

and θ-dominance [55]. The second is to introduce an additional crite-
rion to non-dominance in comparing the quality of solutions, thereby en-
abling an MOEA to select a subset of more promising ones from a large
PT

number of non-dominated solutions. Grid-based evolutionary algorithm

(GrEA) [54], preference-inspired co-evolutionary algorithm using goal vec-
tors (PICEA-g) [51], non-dominated sorting genetic algorithm III (NSGA-
CE

III) [9] and knee point driven evolutionary algorithm for many-objective
optimization (KnEA) [59] belong to this category of MOEAs. Note, how-
ever, that there also exist a large number of other many-objective optimiza-
AC

tion algorithms based on ideas different from those discussed above, see,
e.g., [7, 11, 38, 41, 48].
No much work has been dedicated to reducing the high time com-
plexity of dominance based MOEAs in dealing with MaOPs with only
a few exceptions, although several interesting algorithms have been re-
ported that aim to improve the time efficiency of non-dominated sort-

3
 ACCEPTED MANUSCRIPT

ing for MOEAs. These algorithms focus on reducing unnecessary objec-

tive comparisons in non-dominated sorting, such as fast non-dominated
sort [10], climbing sort and deductive sort [8], corner sort [49], efficient
non-domination level update mechanism [28], M-front [15] and the ef-
ficient non-dominated sorting (ENS) [58]. Unfortunately, all the above-

T
mentioned non-dominated sorting methods remain to be subject to con-
siderable increase in runtime when the number of objectives becomes higher

IP
than three.
In addition to developing efficient non-dominated sorting methods for

CR
dominance based MOEAs, it is worth mentioning that research has also
been conducted on reducing the time complexity of performance indicator
based MOEAs for solving MaOPs. For example, the hypervolume-based

US
many-objective optimization (HypE) [1] proposed to use Monte Carlo sim-
ulations to approximately calculate the hypervolume values. In this way,
the time complexity for hypervolume calculation can be greatly reduced,
which makes it possible to use HypE for solving MaOPs. A similar idea
AN
was also reported in [4], where the Monte Carlo method has been adopted
for approximating the hypervolume in the multi-objective covariance ma-
trix adaptation evolution strategy (MO-CMA-ES).
M

In this work, based on the efficient non-dominated sorting method re-

cently reported in [58], we propose an approximate efficient non-dominated
sorting approach, A-ENS for short, for dominance based MOEAs to solve
ED

MaOPs. The main idea is that, at most three objective comparisons will
be performed in A-ENS in determining the dominance relationship be-
tween two solutions, no matter how many objectives there are. If it is not
able to determine the dominance relationship between two solutions after
PT

three comparisons, then the solution that has already been assigned to a
non-dominated front will be considered to dominate the one that is yet
to be assigned to a front. This means that A-ENS will not consume more
CE

runtime for MaOPs having a large number of objectives than for MOPs
having three objectives. In order to evaluate the influence of the approx-
imate dominance ranking on the performance of MOEAs, we replace the
AC

originally used accurate non-dominated sorting method with A-ENS in

three popular dominance based MOEAs, namely, GrEA [54], NSGA-III [9]
and PICEA-g [51]. Our experimental results on 16 test problems from
two widely used many-objective optimization test suites, DTLZ and WFG,
demonstrate that A-ENS not only considerably reduces the computational
time, but also improves the performance of the above three algorithms on

4
 ACCEPTED MANUSCRIPT

a majority of the test problems considered in this work.

The rest of this paper is organized as follows. In Section 2, existing
work on non-dominated sorting is briefly reviewed. Section 3 presents the
details of the proposed A-ENS. Theoretical analysis and empirical evalu-
ations of its computational complexity are presented in Section 4. Prop-

T
erties regarding the sorting accuracy of the A-ENS are also examined to
understand the influence of approximate sorting on search behavior. Sim-

IP
ulation results of GrEA, NSGA-III and PICEA-g using the proposed A-
ENS on 16 many-objective test problems are described and discussed in

CR
Section 5. Conclusions and future work are given in Section 6.

2. Related work

US
In this section, we review a few popular non-dominated sorting meth-
ods, starting with a brief introduction to the concept of Pareto dominance
in multi-objective optimization.
AN
2.1. Pareto dominance and non-dominated sorting
Non-dominated sorting is a procedure where solutions in the popu-
lation of an MOEA are divided into several groups based on their Pareto
M

dominance relationships. Let X1 and X2 be two solutions of the M -objective

minimization problem defined in (1), X1 is said to dominate X2 , denoted
as X1 ≺ X2 , if and only if ∀i, i = 1, . . . , M , fi (X1 ) ≤ fi (X2 ), and there
ED

exists at least one j ∈ {1, . . . , M } satisfying fj (X1 ) < fj (X2 ). For two so-
lutions X1 and X2 , if there exist k, l, where 1 ≤ k, l ≤ M , k 6= l, such
that fk (X1 ) < fk (X2 ) and fl (X1 ) > fl (X2 ), or if fi (X1 ) = fi (X2 ) for all
PT

i = 1, . . . , M , then X1 is said to be non-dominated with X2 .

From the definition of Pareto dominance, we can see that all M objec-
tives need to be checked if we want to know whether one solution dom-
CE

inates another, while a minimum of two objectives and a maximum of

M objectives need to be compared to make sure that one solution is non-
dominated with another.
AC

Given a set of solutions P for M -objective optimization problems, non-

dominated sorting can be performed as follows in principle. First, all non-
dominated solutions in P are assigned to the first non-dominated front
F1 . Then, all non-dominated solutions are removed from P and the non-
dominated solutions in P \F1 are assigned to the second non-dominated
front F2 . This procedure repeats until all solutions in P are assigned to a

5
 ACCEPTED MANUSCRIPT

front. As a result, all solutions in P are divided into a number of fronts

(groups). Assuming there are K fronts in total, then solutions assigned to
front Fi are more preferred than those assigned to front Fj if 1 ≤ i < j ≤ K.
In the following, we briefly review a number of representative non-
dominated sorting algorithms reported in the literature. These algorithms

T
distinguish themselves in the way in which the solutions are sorted and
thus are different in computational complexity. Note that these non-domin-

IP
ated sorting methods are all accurate and therefore the sorting results of
these algorithms are identical.

CR
2.2. Accurate non-dominated sorting approaches
Most existing research on non-dominated sorting aims to improve its

US
time efficiency. The first idea of using non-dominated sorting for evolu-
tionary multi-objective optimization was realized in the non-dominated
sorting genetic algorithm (NSGA) in [42]. The non-dominated sorting
method reported there has a relatively high time complexity of O(M N 3 ),
AN
where M is the number of objectives and N is the number of solutions
in the population. The main reason for the high time complexity is that
each pair of solutions may be compared for more than once. An improved
M

version of non-dominated sorting, termed fast non-dominated sort, was

reported in NSGA-II [10], which might be by far the most popular MOEA
for solving MOPs. In the fast non-dominated sort, any two solutions are
ED

compared only once, thereby reducing the time complexity to O(M N 2 ).

To further enhance the computational efficiency of non-dominated sort-
ing, Jensen [24] proposed a recursive non-dominated sorting approach,
termed Jensen’s sort, based on a divide-and-conquer mechanism, which
PT

holds a time complexity of O(N logM −1 N ). As we can see, Jensen’s sort is

not well suited for optimization problems having a large number of objec-
tives, as its time complexity grows exponentially with the increase of the
CE

number of objectives. Fortin et al. [20] developed an improved version of

Jensen’s sort, which removes the limitation that all solutions in a popula-
tion cannot have the same value for a certain objective without increasing
AC

the complexity. A computationally more efficient non-dominated sorting

approach, called deductive sort, was suggested by Clymont and Keed-
well in [8], where the dominance relationship can be inferred by recording
the comparison results between solutions to reduce the needed number
of dominance comparisons. Although it is also of a time complexity of

6
 ACCEPTED MANUSCRIPT

O(M N 2 ) in the worst case, empirical evaluations demonstrated that the

deductive sort outperforms the fast non-dominated sort.
Wang and Yao [49] suggested a corner sort for non-dominated sorting,
where the corner solutions in the population are preferred in determin-
ing non-dominated solutions for each front. Corner sort holds the same

T
time complexity as deductive sort, however it was shown that corner sort
needs much fewer objective comparisons than deductive sort in empirical

IP
evaluations. Most recently, an efficient non-dominated sorting algorithm,
termed ENS, was proposed by us [58]. ENS sorts the population according

CR
to the first objective before non-dominated sorting is performed. Conse-
quently, ENS only needs to compare a solution to be assigned to a non-
dominated front with those that have already been assigned to the front.

US
It has been shown that ENS has a time complexity of O(M N √ ) in the worst
2

case. It can, however, achieve a time complexity of O(M N N ) in the best

case. Empirical results demonstrated that the computational efficiency of
ENS is very high in dealing with MOPs having two or three objectives,
AN
however, will seriously deteriorate as the number of objectives increases.
Additional non-dominated sorting methods include the arena’s princi-
ple [45], the non-dominated rank sort adopted in the omni-optimizer [13],
M

dominance decision tree based method [39], the better non-dominated

sort [40], immune recognition based algorithm [61], binary decision dia-
grams based method [32], quick sort [60], sorting based algorithm [16],
ED

divide-and-conquer based non-dominated sorting algorithm [18], efficient

non-domination level update mechanism [28] and M-front [15], to name a
few.
One central idea in improving the efficiency of non-dominated sort-
PT

ing is to reduce as much as possible redundant dominance comparisons

between solutions. Most existing non-dominated sorting approaches are
efficient in dealing with MOPs that have a small number of objectives;
CE

however, their efficiency often considerably degrades as the number of

objectives increases. The deterioration of their efficiency is due to the fact
that for two solutions with M objectives, a minimum of two objective com-
AC

parisons and a maximum of M objective comparisons are needed to deter-

mine their dominance relationship.
Note that all the above non-dominated sorting methods aim to obtain
accurate non-dominated sorting of all solutions in the population. In the
following, we present an approximate non-dominated sorting based on

7
 ACCEPTED MANUSCRIPT

ENS to further reduce the computational time.

3. Approximate non-dominated sorting

In this section, we briefly review the ENS suggested in [58] before de-

T
scribing the details of the proposed approximate non-dominated sorting
approach A-ENS.

IP
3.1. A brief summary of ENS

CR
ENS consists of the following two main steps. First, all solutions in
the population are sorted in an ascending order according to one of the
objectives, usually the first objective. Second, the solutions in the sorted

US
population are assigned to non-dominated fronts one by one, starting from
the first solution to the last one by comparing a solution to be assigned to
a front with those that have been assigned to the front. If this solution
is non-dominated with all solutions that have been assigned to the front,
AN
then the solution is also assigned to the front. Otherwise, ENS will check
whether the solution can be assigned to the next front. This procedure
repeats until all solutions in the population are assigned to a front.
Theoretical analysis and empirical evaluations illustrated that ENS is a
M

computationally very efficient non-dominated sorting method, especially

when the number of objectives is not large. The high efficiency of ENS can
be attributed to the fact that solutions to be assigned to a front only need to
ED

be compared with those that have already been assigned to a front, which
is made possible by sorting the individuals in the population according to
one of the objectives. In other words, for a population sorted according to
PT

one of the objectives in an ascending order, if we assign solutions to non-

dominated fronts from the first solution, i.e., the one having the best value
in the objective according to which the population is sorted, an unassigned
CE

solution can never dominate solutions that have already been assigned to
the fronts.
Due to the above reason, A-ENS also sorts the solutions in the popu-
AC

lation according to the first objective before non-dominated sorting is per-

formed. The detailed ideas for non-dominated sorting in A-ENS will be
presented in the following subsection.

8
 ACCEPTED MANUSCRIPT

3.2. The proposed A-ENS approach

The motivation of proposing an approximate non-dominated sorting
method is to substantially enhance the computational efficiency of dom-
inance based MOEAs for solving MaOPs. As mentioned above, A-ENS
proposed in this work performs at most three objective comparisons in

T
determining the Pareto dominance relationship between two solutions. In

IP
case the dominance relationship between two solutions cannot be deter-
mined by performing three objective comparisons, then the solution that
has already been assigned to a non-dominated front will be considered to

CR
dominate the one to be assigned to the front. In other word, the compu-
tational cost of the A-ENS for an MaOP will be no more than that for an
MOP having three objectives.

US
The details of A-ENS are presented in Algorithm 1 in pseudo code. To
describe the details of the A-ENS, let us consider an M -objective mini-
mization problem to be optimized by an MOEA whose population size is
N . Similar to ENS, A-ENS begins with sorting the N solutions according
AN
to the first objective in an ascending order. If the sorted population is de-
noted by p1 , p2 , ..., pN , then the first objective of pi is always smaller than or
equal to the first objective of pj , if i < j, where 1 ≤ i, j ≤ N . Consequently,
M

pi either dominates pj or pi and pj are non-dominated with each other.

Before performing non-dominated sorting on the sorted population P ,
A-ENS first normalizes all objective values by using the minimum and
ED

maximum of each objective. For any solution p = (f1 , f2 , ..., fM ), the nor-
malized solution p0 = (f10 , f20 , ..., fM 0
), where fi0 = (fi − fmin )/(fmax − fmin ),
fmin and fmax are the minimum and maximum values on the i-th objec-
tive in the population. After normalization, for each solution, identify the
PT

two objectives from the second to the M -th objective that have the max-
imum and minimum objective values, respectively. For example, if the
objective values of a solution to a four-objective optimization problem is
CE

(f1 , f2 , f3 , f4 ) =(0.1, 0.8, 0.2, 0.6), then the second objective f2 and the third
objective f3 will be used for dominance comparisons. These two objec-
tives of each solution, together with its mean value of the second objective
AC

to the M -th objective, will be used in A-ENS for objective comparisons to

determine the dominance relationships between solutions in the popula-
tion. The main reason for normalizing all solutions on each objective is
that, for some MOPs, it can happen that the value of one objective is al-
ways larger than that of another objective, or vice versa. Normalization

9
 ACCEPTED MANUSCRIPT

Algorithm 1: The proposed A-ENS approach

Input: P (population), M (number of objectives)
Output: F (set of fronts)
1 sort P in an ascending order according to the first objective;
2 P ← normalize(P );

T
3 for i = 1 to |P | do

IP
4 M inLoc[i] ← arg minj=2...M pji ;
5 M axLoc[i] ← arg maxj=2...M pji ;
P

CR
6 M ean[i] ← M1−1 j=2...M pji ; /*pji represents the j-th
objective value of the i-th solution in P */
7 F ←∅; /*the set of fronts*/
8
9
10
k←0;
while not empty(P ) do US /*the number of fronts*/

k ← k + 1 ; /*start to assign solutions to the next

front*/
AN
11 F [k] ← F [k] ∪ {p1 };
12 for i = 1 to |P | do
13 for j = |F [k]| to 1 do
M

M inLoc[i] M inLoc[i]
14 if pi < qj then /* q represent the
solutions having been assigned to F [k] */
15 dominated ← f alse;
ED

M axLoc[j] M axLoc[j]
16 else if pi < qj then
17 dominated ← f alse;
18 else if M ean[i] < M ean[j] then
PT

19 dominated ← f alse;
20 else
CE

21 dominated ← true;
22 if dominated == f alse then
23 F [k] ← F [k] ∪ {pi };
AC

24 P ← P − {pi };

25 return F

10
 ACCEPTED MANUSCRIPT

of the objective values makes it meaningful to compare the mean of the

objectives of different solutions.
Once the normalization of solutions is done, A-ENS starts to assign so-
lutions to the first front F1 . As the population is sorted in an ascending
order according to the first objective, the first solution p1 must belong to

T
F1 , as none of the solutions pj , j = 2, ..., N can dominate p1 . Next, A-ENS
checks whether the second solution p2 belongs to front F1 by making at

IP
most three objective comparisons with solution p1 . If p2 is non-dominated
with p1 , p2 will be assigned to F1 ; otherwise, p2 will remain in the pop-

CR
ulation P . After p2 is checked, A-ENS starts to check whether p3 can be
assigned to front F1 by comparing p3 with the solutions that have already
been assigned to front F1 , using at most three objective comparisons be-
tween each pair of solutions.
US
Now let us consider a more general situation in which we need to de-
termine the dominance relationship between two solutions p and q, where
p is a solution to be assigned to front F1 and q is a solution that has already
AN
been assigned to front F1 . Recall that the population has been sorted in
an ascending order according to the first objective, and all objective val-
ues are normalized. In A-ENS, the dominance comparison between p and
M

q consists of a maximum of three steps. First, we check which objective

(except for the first one) of solution p has the minimum value among all
its objectives. We assume this is the iL -th objective, 2 ≤ iL ≤ M . Now
ED

we compare the iL -th objective value of p with that of solution q. If the iL -

th objective value of p is smaller than that of q, we can then immediately
conclude that p and q are non-dominated, owing to the fact that the first
objective value of q is smaller than that of p. If the iL -th objective value of
PT

p is larger than that of q, the dominance relationship between them cannot

be yet determined. Therefore, we need to go to the second step starting
with checking which objective (except for the first one) of q has the max-
CE

imum value. Here we assume it is the iU -th objective. Then we compare

the iU -th objective value of p with that of q. If the iU -th objective value
of p is smaller than that of q, then p is non-dominated with q; otherwise,
AC

the third objective comparison needs to be carried out, where the mean
of the normalized objective values of p is compared with the mean of the
normalized objective values of q. If the mean objective value of q is larger
than that of p, then we consider p and q to be non-dominated; otherwise,
solution p is considered to be dominated by solution q.
Take the two solutions shown in Figure 1 as an example, where p =

11
 ACCEPTED MANUSCRIPT

The maximum value in q

Mean of q
q 0.1 0.7 0.2 0.5 0.8 0.6 0.56

compare compare compare

T
Mean of p
p 0.2 0.6 0.3 0.4 0.9 0.5 0.54

IP
Sorted
The minimum value in p

CR
Figure 1: Determining the Pareto dominance relationship between solutions p and q in
A-ENS by performing at most three objective comparisons. Note that the first objective is
not considered in the calculation of the maximum, minimum and mean of the objective
values of a solution.

US
(0.2, 0.6, 0.3, 0.4, 0.9, 0.5) and q = (0.1, 0.7, 0.2, 0.5, 0.8, 0.6) are the values of
the six objectives, respectively. First, we compare the third objective of
AN
p and q, as this is the minimum of all objectives of p, except for the first
objective. As the third objective value of q is larger than that of p, we
go to the next step to compare the fifth objective of p and q, as the fifth
objective value of q is the maximum among all objectives of q. Since the
M

fifth objective value of p is larger than that of q, we need then go to the last
step to compare the mean objective value of p with the mean of q. As the
mean of p is smaller than that of q, we consider p and q are non-dominated
ED

with each other.

Two observations can be made from the above dominance compari-
son procedure. First, A-ENS selectively compares two objectives. One is
PT

the objective that has a minimum value among all objectives (except for
the first objective) of the solution to be assigned to a front. The second
is the objective that has the maximum value among all objectives (except
CE

for the first one) of the solution that has been assigned to a front. Second,
non-dominance comparison is accurate if the non-dominance relationship
between two solutions can be determined by the first two comparisons.
However, the dominance relationship may be incorrect if it is determined
AC

in the last step, where the means of the objective values of the two solu-
tions are compared. Therefore, the non-dominated sorting using the three
comparisons is an approximate sorting method.
If the first non-dominated front F1 contains multiple solutions that
have been assigned to it, solution p needs to be compared with all these

12
 ACCEPTED MANUSCRIPT

solutions before it is found to be dominated by one of the solutions in

front F1 . When p is not dominated by any solution that has been assigned
to front F1 , it will then be assigned to F1 ; otherwise, it cannot be assigned
to front F1 and remains in P . The assignment of solutions to front F1 is
completed until all solutions in P have been checked.

T
Next, A-ENS starts to assign the remaining solutions in the population
to the second front F2 . The first solution in the remaining solutions will

IP
definitely be assigned to F2 , since the population has been sorted accord-
ing to the first objective and the order of the solutions remains unchanged

CR
although the solutions assigned to F1 have been removed. Then, domi-
nance comparison using the three objective comparisons described above
will be performed between solutions assigned to F2 and solutions to be as-

US
signed, until all solutions belonging to F2 are identified. This procedure re-
peats until all solutions in the population are assigned to a non-dominated
front.
It should be pointed out that A-ENS often needs only to find the solu-
AN
tions belonging to the first front F1 for an MOEA to solve MaOPs, since
there exist a large number of non-dominated solutions in the combined
population already in the early stage.
M

4. Analysis of complexity and sorting accuracy

Before we investigate the influence of the proposed A-ENS on the search
ED

performance of an MOEA, we first quantitatively analyze the computa-

tional complexity and empirically compare it with a few existing accurate
non-dominated sorting methods. We then examine the sorting accuracy of
PT

A-ENS on optimization problems with three or more objectives. Note that

for bi-objective optimization problems, A-ENS reduces to ENS.
CE

4.1. Analysis of computational complexity

Proposition 1. For a population consisting of N solutions having any number
of objectives equal to or larger than three, we can conclude that A-ENS performs
AC

3
at most 32 (N 2 − N ) objective comparisons and at least 2(dN 2 e − N ) objective
comparisons. Therefore,
√ the time complexity of A-ENS is √ O(N 2 ) in the worst
case, and O(N N ) in the best case, assuming that M < N , where M is the
number of objectives.

Proof. Note that A-ENS can be divided into the following four main steps:

13
 ACCEPTED MANUSCRIPT

1) Sort the population in an ascending order according to the first ob-

jective;
2) Normalize each objective of the solutions in the population;
3) Find the maximum and minimum objective values, and calculate the
mean objective value of each solution;

T
4) Assign solutions in the sorted population to non-dominated fronts.

IP
To analyze the time complexity of A-ENS, we first consider step 4), which
consumes the majority of the runtime of A-ENS. In assigning solutions to

CR
a non-dominated front, A-ENS compares any two solutions at most once
since a solution to be assigned to a front only needs to be compared with
the solutions that have been assigned to the front. In the worst case, all
solutions in the population belong to the same front, therefore any two
US
solutions in the population should be compared once, which means that
altogether (N 2 − N )/2 solution comparisons need to be performed. As
three objective comparisons are needed for comparing a pair of solutions,
AN
A-ENS performs at most 32 (N 2 − N ) objective comparisons.
√
In the best case, A-ENS assigns individuals in√the population to d N e
fronts and each front approximately contains d N e solutions. We also
M

assume that one solution comparison is enough if a solution cannot be as-

signed to a front by A-ENS. That is to say, A-ENS only needs to compare
this solution with the last solution that has already been assigned to that
ED

front after identifying that it is dominated by the solution. Thus, A-ENS

does not need to compare this solution with other √solutions that have been
assigned to the front. More specifically,
√ for the d N e solutions in front F1 ,
A-ENS needs to perform 2 (N − d N e) solution comparisons. In the best
1
PT

case, the non-dominance relationship between two solutions √ can be de-

termined by the first objective comparison, then 2 (N − d N e) objective
1
√
comparisons are needed for the N e solutions in front F1 . The remain-
CE

√ d
ing N − d N e solutions in the population also need to be compared with
the last√solution that has already been assigned to front F1 , which incurs
N − d N e solution comparisons. Recall that in A-ENS, three objective
AC

comparisons are needed in case the solution to be assigned to a front is

dominated by a solution
√ that has already been assigned to the front. As
a result, 3(N − d N e) objective
√ comparisons are needed. Thus, A-ENS
needs to perform 2 (N − d N e) objective comparisons to identify all solu-
7

tions belonging to the non-dominated front F1 .

14
 ACCEPTED MANUSCRIPT

×105
2 0.18
Fast Non-dominated Sort Fast Non-dominated Sort
1.8 Corner Sort 0.16 Corner Sort
Deductive Sort Deductive Sort
1.6 ENS-SS 0.14 ENS-SS
Number of Comparisons

A-ENS A-ENS
1.4
0.12

Runtime (s)
1.2
0.1
1

T
0.08
0.8
0.06
0.6

IP
0.4 0.04

0.2 0.02

0 0
3 5 7 9 11 13 15 3 5 7 9 11 13 15

CR
Number of Objectives Number of Objectives

(a) (b)

Figure 2: Number of objective comparisons and runtime of five non-dominated sort-

ing algorithms for random populations of size 200 with different numbers of objectives,
where the results on M -objective are obtained by averaging over 30 random populations
of size 200 with M objectives. (a) Number of objective comparisons; (b) Runtime. US
AN
√
Similarly, A-ENS needs 72 N − 13 d N e objective comparisons to identify
2 √
all solutions belonging to front F2 and 12 (N − d N e) objectives compar-
isons are needed to find the solutions in the last front. Therefore, A-ENS
M

3
needs to perform a minimum of 2(dN 2 e − N ) objective comparisons.
To summarize, step√ 4) of A-ENS has a time complexity of O(N ) in the
2

worst case and O(N N ) in the best case. The computational cost for steps
ED

1)-3) is relatively straightforward to calculate. For a population with N

solutions and M objectives, step 1) needs a time complexity of O(N logN ),
and steps √ 2) and 3) incur a time complexity of O(M N ) each. Assuming
that M < N , which is reasonable even for many-objective optimization,
PT

we can conclude that the A-ENS approach√ has an overall time complexity
of O(N ) in the worst case and O(N N ) in the best case. Therefore, the
2
CE

statement of Proposition 1 holds.

In the following, we empirically compare the runtime of A-ENS with
four popular non-dominated sorting methods, including fast non-domin-
AC

ated sort [10], deductive sort [8], corner sort [49] and ENS-SS [58]. Fig-
ure 2 presents the number of objective comparisons and runtime of the
five compared non-dominated sorting methods, respectively. The results
are obtained from 30 uniformly randomly generated populations of size
200. As can be seen from the figure, A-ENS is computationally much more

15
 ACCEPTED MANUSCRIPT

efficient than the other four non-dominated sorting algorithms, in partic-

ular when the number of objectives becomes large.
Another important observation we can make from the empirical results
shown in Figure 2 is that the number of objective comparisons and run-
time of A-ENS remain almost unchanged when the number of objectives

T
increases from 3 to 15, which is encouraging.

IP
4.2. Analysis of sorting accuracy
In the following, we will present some theoretical analysis on the sort-

CR
ing accuracy of A-ENS, which heavily depends on the number of objec-
tives.

Proposition 2. For MOPs with three objectives, A-ENS performs accurate non-
dominated sorting.
US
Proof. To prove Proposition 2, we only need to show that A-ENS can cor-
AN
rectly determine the Pareto dominance relationship between a solution to
be assigned and the solutions having been assigned to a front when the
number of objectives is three. Let p be the solution to be assigned and
q be the solution having been assigned to a front. For determining the
M

dominance relationship between p and q, A-ENS performs at most three

objective comparisons between p and q, i.e., the objective on which p has a
minimum value among its second and third objective values and the ob-
ED

jective on which q has the maximum value among its second and third
objective values, as well as the means of p and q (except for the first ob-
jective value). If the two objectives are different, then the mean of p must
PT

be larger than that of q in case the two objectives cannot determine the
dominance relationship between p and q. This means that no errors will be
introduced in determining the dominance relationship of p and q when we
compare them using their mean fitness. Since the solutions are sorted in
CE

an ascending order according to the first objective value, A-ENS compares

all objectives of p and q. So, A-ENS can correctly determine the dominance
relationship between p and q.
AC

If the objective on which p has the minimum value is exactly the one
on which q has a maximum value, then A-ENS compares p and q only
based on this objective and their mean objective value. In this case, if the
dominance relationship between p and q cannot be determined by this ob-
jective, then q must have a smaller value than p on the remaining objective

16
 ACCEPTED MANUSCRIPT

since the value on the remaining objective of q is always smaller than its
maximum objective value, while the value on the remaining objective of
p is always larger than its minimum objective value. Therefore, no error
will be introduced in determining the dominance relationship in using the
means of p and q for comparison and A-ENS can correctly determine the

T
dominance relationship between the two solutions. With the above analy-
sis, we can conclude that Proposition 2 holds.

IP
Remark 1. The following statements hold for A-ENS in sorting populations for

CR
solving MOPs with four or more objectives:
1) A-ENS cannot guarantee that it can sort the whole population correctly;
2) Almost all solutions assigned to a front by A-ENS exactly belong to the

US
front and the number of solutions correctly assigned to the first front by
A-ENS is very large when the MaOP has a large number of objectives;
3) Solutions belonging to the same front may be mistakenly assigned to differ-
AN
ent fronts by A-ENS, however, most of them will be assigned to the next
front.

Statement 1) in Remark 1 can be verified based on the following ob-

M

servation. For any two solutions with M (M ≥ 4) objectives, at least

two objectives need to be checked for determining whether they are non-
dominated with each other, while all M objectives must be checked for
ED

determining whether a solution dominates the other. Due to the fact that
A-ENS uses at most three objectives (including the first objective based on
which the population is sorted but except for the mean fitness value), A-
ENS will introduce sorting errors and therefore cannot guarantee that all
PT

solutions are assigned to the right front.

Two kinds of errors may be introduced by A-ENS in performing non-
dominated sorting:
CE

a) A solution belongs to the i-th front (i ≥ 1), however it is incorrectly

assigned to front j that is after front i, i.e., j > i;
b) A solution belongs to the i-th front (i ≥ 2), however it is incorrectly
AC

assigned to front j that is before the i-th front, i.e., j < i.

Error type a) occurs when a solution belongs to front i, but A-ENS cannot
determine that it is non-dominated with a solution having been assigned
to the front using the three objective comparisons. In this case, A-ENS

17
 ACCEPTED MANUSCRIPT

assumes that this solution is dominated by the solution having been as-
signed to the front, thereby assigning it to a front after front i. Error type
b) can be seen as a consequence of error type a): when some solutions are
incorrectly assigned to a front that is after the front they should belong to,
solutions dominated by these incorrectly assigned solutions will be possi-

T
bly assigned to a front before the front they belong to.

IP
1
Solutions correctly Solutions correctly

CR
assigned to front 1 assigned to front 1
Solutions in front 1 but Solutions in front 1 but
not assigned to front 1 0.995 0.8 assigned to front 2
1
Solutions in other fronts 0.898 Solutions in front 1 but
but assigned to front 1 assigned to other fronts
Ratio of Solutions

Ratio of Solutions
0.8 0.748 0.6

0.6

0.4

0.2

0
0.201

0
0.343

0.001 0 0 0
US 0.4

0.2

0
AN
4-objective 5-objective 8-objective 10-objective 15-objective 4-objective 5-objective 8-objective 10-objective 15-objective

(a) (b)

Figure 3: Ratio of solutions assigned to the first front by A-ENS and ratio of solutions
M

belonging to the first front that are assigned to different fronts by A-ENS for random
populations of size 200 for different numbers of objectives. The results are obtained by
averaging over 30 random populations of the same size and same objectives. (a) Ratio
of solutions assigned to the first front; (b) Ratio of solutions in the first front assigned to
ED

different fronts.

Figure 3 illustrates the ratio of solutions assigned to the first front by

A-ENS and ratio of solutions belonging to the first front that are assigned
PT

to different fronts by A-ENS for random populations of size 200 for dif-
ferent numbers of objectives. The results are obtained by averaging over
30 random populations of the same size and the same number of objec-
CE

tives. As shown in Figure 3 (a), only a very small number of solutions are
incorrectly assigned to the first front for random populations with four ob-
jectives among all random populations. These empirical results illustrate
AC

that error type b) made by A-ENS is very rare for populations with four
or more objectives. Moreover, the number of solutions correctly assigned
to the first front by A-ENS becomes larger when the number of objectives
increases.
From Figure 3 (b), we can find that A-ENS may assign solutions in the
first front to different fronts. However, a majority of the solutions are cor-

18
 ACCEPTED MANUSCRIPT

rectly assigned to the first front. Furthermore, almost all solutions in the
first front which are assigned to the other fronts by mistake are assigned
to the second front. Roughly 95% of the solutions belonging to the first
front are either correctly assigned to the first front or incorrectly assigned
to the second front. Only a very low ratio of the solutions belonging to the

T
first front are assigned to a front after front two. The above analysis and
empirical results confirm the statements of Remark 1.

IP
In the following, we further empirically investigate which solutions in
a population are likely to be incorrectly assigned by A-ENS. To this end,

CR
we consider populations consisting of non-dominated solutions and all
these solutions belong to the true Pareto front of an MOP with only one
exception, which is far from the front although it is non-dominated with

US
other solutions in the population. Therefore, this particular solution is
worse than other non-dominated solutions in the population in terms of
convergence. We check how likely it is that the non-dominated solution
with poor convergence will be considered to be dominated by other so-
AN
lutions due to the sorting errors introduced by A-ENS. For this purpose,
we randomly generate 100 solutions located in the true Pareto front of 5-
objective DTLZ2 and move a randomly selected solution away from the
M

true front by multiplying each objective value of the solution by a coef-

ficient λ ranging from 1 to 1.5. If the solution multiplied by λ is domi-
nated by a solution in the population, then a new random population will
ED

be generated until a randomly selected solution multiplied by λ is non-

dominated with other solutions in the population. Figure 4 presents the
ratio of times that a randomly selected non-dominated solution is incor-
rectly considered to be dominated by other solutions in the population by
PT

A-ENS before and after different values of λ are multiplied, for 200 ran-
dom populations obtained from the true Pareto front of 5-objective DTLZ2.
As shown in the figure, it is easy to find that the non-dominated solution
CE

with poor convergence will more likely be considered to be dominated

by A-ENS, and the probability that a solution is incorrectly considered to
be dominated by A-ENS increases as the convergence of the solution de-
AC

creases. Therefore, we can conclude that A-ENS introduces a bias towards

selecting solutions with better convergence in sorting the population. This
observation will be further illustrated by embedding A-ENS into MOEAs
for solving MaOPs, which is presented in Section 5.
From the analysis of computational complexity of A-ENS, we can con-
clude that A-ENS is computationally efficient for non-dominated sorting

19
 ACCEPTED MANUSCRIPT

DTLZ2 with 5 objectives

1
Ratio of dominated times after multiplying λ
0.9 Ratio of dominated times before multiplying λ

Ratio of dominated times 0.8

0.7

0.6

T
0.5

0.4

IP
0.3

0.2

CR
0.1

0
1.05 1.10 1.15 1.20 1.25 1.30 1.35 1.40 1.45 1.50
λ

US
Figure 4: Ratio of times that a randomly selected non-dominated solution is incorrectly
considered to be dominated by other solutions in the population by A-ENS before and
after different values of λ are multiplied, for 200 random populations obtained from the
true Pareto front of 5-objective DTLZ2.
AN
for MOPs with three or more objectives. We also show that A-ENS intro-
duces errors in non-dominated sorting for MOPs with more than three ob-
jectives. However, our analyses indicate that the introduced sorting errors
M

are minor, which can eventually bring about a slight bias towards solu-
tions having good convergence. In the next section, we will empirically
demonstrate that A-ENS can enhance the search performance of MOEAs
ED

in solving MaOPs compared with those using an accurate sorting method

on most benchmark test problems.
PT

5. Empirical comparison of MOEAs using A-ENS

In this section, we examine the search performance of the proposed
CE

A-ENS in evolutionary multi-objective optimization by replacing the ac-

curate non-dominated sorting method with A-ENS. The empirical com-
parisons are conducted on 16 test problems taken from two test suites,
DTLZ [12] and WFG [37], whose parameter settings are set to those adopted
AC

in [59]. Due to the fact that the A-ENS can also correctly perform non-
dominated sorting for MOPs with two or three objectives, we only con-
sider test problems with 4, 6, 8 and 10 objectives. We compare the quality
of the obtained non-dominated solution sets with the help of widely used
performance indicators as well as the computational efficiency in terms of

20
 ACCEPTED MANUSCRIPT

runtime of the non-dominated sorting approaches.

Two widely used performance indicators, hypervolume (HV) [3, 53]
and inverted generational distance (IGD) [63] are used to evaluate the
quality of the obtained non-dominated sets. It is believed that both indica-
tors are able to account for both convergence (closeness to the true Pareto

T
front) of MOEAs and the distribution of the achieved non-dominated so-
lutions. To evaluate the convergence and diversity separately, two addi-

IP
tional widely used quality metrics, generational distance (GD) [46], which
assesses convergence, and spread (∆) [52], which accounts for diversity,

CR
are also compared. The larger the HV value, the better the quality of the
solution set is. By contrast, small values of IGD, GD and ∆ are prefer-
able. The reference point for calculating HV is chosen as follows. We first

US
combine all solution sets obtained by the compared algorithms in all runs
on the considered test instance, then we remove all dominated solutions
in the combined solution set. The maximum values in each objective of
all the non-dominated solutions are identified as the reference point for
AN
calculating the HV. In addition, the Monte Carlo method is adopted here
for estimating the HV, where 1,000,000 sampling points are used. On the
other hand, the calculation of the three metrics IGD, GD and ∆ requires
M

a reference set of Pareto optimal solutions, which are uniformly chosen

from the true Pareto front of test problems. Note, however, that for differ-
ent test problems with different numbers of objectives, it is impossible to
ED

use exactly the same number of reference points. In this work, we set the
number of reference points to an integer that is closest to 500.
In all experiments reported in this work, the number of generations is
adopted as the termination criterion for all considered algorithms. For
PT

DTLZ1 and WFG2, the maximum number of generations is set to 700,

and to 1000 for DTLZ3 and WFG1. For DTLZ2, DTLZ4, DTLZ5, DTLZ6,
DTLZ7 and WFG3 to WFG9, we set the maximum number of generations
CE

to 250. On each test instance, 30 independent runs are performed for each
algorithm and the median performance value is reported. All simulations
are conducted on a PC with a 3.16GHz Intel Core 2 Duo CPU E8500 and
AC

the Windows 7 SP1 64 bit operating system. We should stress that, al-
though the proposed A-ENS can not identify all non-dominated solutions
in the first front, it still aims to achieve a representative of the whole Pareto
set instead of only a subset when it is embedded into the MOEAs consid-
ered in this work.

21
 ACCEPTED MANUSCRIPT

Table 1: Parameter setting in NSGA-III, where p1 and p2 are parameters controlling the
numbers of reference points along the boundary of the Pareto front and inside it, respec-
tively.

number of
parameter (p1 , p2 ) population size

T
objectives
4 (7, 0) 120

IP
6 (4, 1) 132
8 (3, 2) 156

CR
10 (3, 2) 275

5.1. NSGA-III using A-ENS

US
We first test the performance of A-ENS when it is embedded into NSGA-
III [9], a popular dominance based MOEAs recently proposed for solving
MaOPs. NSGA-III has a similar framework to NSGA-II, but a very differ-
AN
ent selection mechanism for solutions in the same front. Instead of using
the crowding distance in NSGA-II, a set of uniformly distributed reference
points are adopted in NSGA-III so that non-dominated solutions closest
to these reference points are selected. To prevent the reference points from
M

being generated on the boundary of Pareto fronts for problems with a large
number of objectives, a two-layered reference point generation method
was suggested in NSGA-III to make sure that some reference points lo-
ED

cated inside the Pareto fronts can be created.

Table 1 illustrates the parameter setting in the following experiments
for NSGA-III. The simulated binary crossover and polynomial mutation
PT

recommended in NSGA-III [9] have been adopted to create offspring, with

the distribution index of crossover nc = 30 and the distribution index of
mutation nm = 20. A crossover probability of pc = 1.0 and mutation
CE

probability of pm = 1/D are used, where D denotes the number of de-

cision variables. For convenience, we denote the NSGA-III using A-ENS
as NSGA-III/A-ENS.
AC

1) Experimental results on the performance of NSGA-III/A-ENS.

Table 2 presents the mean and standard deviation of HVs of NSGA-III
and NSGA-III/A-ENS on DTLZ1 to DTLZ7 and WFG1 to WFG9, aver-
aging over 30 independent runs, where the better mean between the two
compared algorithms is highlighted. Moreover, the Wilcoxon’s rank sum
test at a significance level of 0.05 was adopted to test the statistical sig-

22
 ACCEPTED MANUSCRIPT

Table 2: HVs of NSGA-III and NSGA-III/A-ENS on DTLZ1 to DTLZ7 and WFG1 to

WFG9, where the best mean for each test instance is highlighted.
4-objective 6-objective 8-objective 10-objective
Problem NSGA-III/ NSGA-III/ NSGA-III/ NSGA-III/
NSGA-III A-ENS NSGA-III A-ENS NSGA-III A-ENS NSGA-III A-ENS

T
9.1E-1† 8.9E-1 9.8E-1† 9.0E-1 9.2E-1† 7.1E-1 1.0E+0† 5.2E-1
DTLZ1 (9.8E-4) (7.3E-3) (3.3E-3) (4.4E-2) (1.7E-1) (2.4E-1) (6.0E-3) (2.0E-1)

IP
5.7E-1† 5.7E-1 7.4E-1 7.6E-1† 8.5E-1 9.1E-1† 9.4E-1 9.8E-1†
DTLZ2 (1.3E-3) (1.5E-3) (6.7E-3) (3.6E-3) (2.2E-2) (2.0E-3) (2.7E-2) (5.8E-4)
5.9E-1† 5.2E-1 1.0E+0† 9.9E-1 1.0E+0† 1.0E+0 1.0E+0 1.0E+0
DTLZ3

CR
(1.7E-2) (1.0E-1) (7.8E-5) (4.1E-2) (5.5E-6) (4.0E-4) (0.0E+0) (0.0E+0)
5.6E-1 5.6E-1† 7.5E-1 8.0E-1† 9.2E-1 9.5E-1† 9.9E-1 1.0E+0†
DTLZ4 (4.2E-2) (3.9E-2) (4.2E-2) (3.9E-2) (1.0E-2) (1.3E-2) (8.3E-4) (1.7E-3)
6.7E-1† 6.7E-1 8.0E-1 8.0E-1† 8.3E-1† 8.0E-1 8.6E-1† 7.9E-1
DTLZ5 (2.9E-3) (1.8E-3) (8.1E-3) (1.3E-2) (1.5E-2) (3.2E-2) (9.1E-3) (2.4E-2)

DTLZ6

DTLZ7
9.1E-1
(5.3E-3)
1.9E-1
(4.3E-3)
9.1E-1†
(7.4E-3)
1.9E-1†
(6.8E-3)
8.8E-1
(4.4E-2)
1.2E-1
(7.1E-3)
US
9.5E-1†

1.5E-1†
9.0E-1

8.6E-2
9.6E-1†
(1.0E-2) (4.3E-2) (1.1E-2)
1.3E-1†
(1.1E-2) (9.7E-3) (7.9E-3)
8.9E-1
(3.2E-2)
8.9E-2
(1.1E-2)
9.4E-1†
(1.7E-2)
1.3E-1†
(8.7E-3)
AN
6.8E-1 9.4E-1† 6.7E-1 1.0E+0† 4.3E-1 1.0E+0† 2.8E-1 1.0E+0†
WFG1 (4.8E-2) (3.7E-2) (4.9E-2) (6.9E-4) (5.0E-2) (2.1E-3) (6.5E-2) (6.6E-4)
9.4E-1† 9.4E-1 9.5E-1† 9.4E-1 9.9E-1 9.9E-1 1.0E+0 9.9E-1
WFG2 (6.9E-2) (7.6E-2) (7.4E-2) (7.4E-2) (2.2E-3) (5.7E-3) (2.6E-3) (2.5E-3)
5.9E-1 5.9E-1† 5.3E-1 5.3E-1 5.5E-1 5.9E-1† 5.5E-1 5.7E-1†
M

WFG3 (8.2E-3) (8.0E-3) (1.9E-2) (1.7E-2) (3.3-2) (8.7E-3) (2.8E-2) (1.0E-2)

4.9E-1 5.1E-1† 5.9E-1 6.6E-1† 7.1E-1 7.7E-1† 7.9E-1 8.7E-1†
WFG4 (6.3E-3) (5.4E-3) (3.7E-2) (8.6E-3) (9.4E-3) (6.2E-3) (1.2E-2) (7.6E-3)
5.0E-1 5.1E-1† 6.3E-1 6.4E-1† 7.3E-1 7.5E-1† 8.0E-1 8.3E-1†
ED

WFG5 (6.8E-3) (5.6E-3) (1.6E-2) (1.0E-2) (1.4E-2) (7.9E-3) (1.2E-2) (8.0E-3)

5.2E-1 5.2E-1 6.5E-1 6.6E-1 7.3E-1 7.5E-1† 7.9E-1 8.3E-1†
WFG6 (1.4E-2) (1.8E-2) (2.4E-2) (1.8E-2) (1.7E-2) (1.2E-2) (1.6E-2) (1.2E-2)
5.3E-1 5.4E-1† 6.6E-1 7.1E-1† 7.7E-1 8.2E-1† 8.5E-1 9.1E-1†
PT

WFG7 (6.6E-3) (3.5E-3) (4.1E-2) (5.7E-3) (8.0E-3) (3.8E-3) (1.6E-2) (3.0E-3)

4.4E-1 4.6E-1† 4.8E-1 5.3E-1† 6.1E-1 6.5E-1† 7.0E-1 7.1E-1
WFG8 (1.2E-2) (7.2E-3) (1.9E-2) (4.0E-2) (1.8E-2) (2.5E-2) (1.1E-2) (2.6E-2)
4.4E-1 4.7E-1† 5.1E-1 5.8E-1† 6.8E-1 7.5E-1† 7.7E-1 8.2E-1†
CE

WFG9 (5.8E-2) (5.5E-2) (4.8E-2) (6.4E-2) (4.6E-2) (9.5E-3) (2.5E-2) (1.1E-2)

”†”
indicates that the two results are significantly different at a level of 0.05 by the Wilcoxon’s rank sum test.
AC

nificance of the differences between the results obtained by the two algo-
rithms. As can be seen from the table, when NSGA-III adopts the pro-
posed A-ENS instead of the accurate non-dominated sorting algorithm,
the performance of NSGA-III has a clear improvement on a majority of
the test instances from DTLZ and WFG suites. From the Wilcoxon’s rank

23
 ACCEPTED MANUSCRIPT

Table 3: Runtime (s) of five different non-dominated sorting approaches in NSGA-III for
solving DTLZ and WFG test problems, where the runtime of a non-dominated sorting
approach on M objectives is obtained by averaging over the runtime consumed by the
algorithm for the embedded NSGA-III running one time on all DTLZ and WFG problems
with M objectives.

T
Non-dominated sorting approach
MOEA obj. fast non- deductive corner

IP
dominated sort sort sort ENS-SS A-ENS

4 48.52 18.32 18.77 13.46 5.58

CR
6 62.41 25.68 26.03 19.57 6.28
NSGA-III 8 93.79 38.71 38.35 30.52 8.18
10 312.13 134.36 129.05 106.63 20.09

US
sum test point of view, it can also be found that NSGA-III/A-ENS per-
forms better than NSGA-III. Among the 64 test instances used in the study,
AN
NSGA-III/A-ENS achieved statistically significantly better performance
on 44 test instances and the results on 7 instances do not have significant
difference. Therefore, we can conclude that NSGA-III using A-ENS out-
performs NSGA-III using the accurate non-dominated sorting algorithm.
M

Table 3 presents the runtime of five different non-dominated sorting

approaches in NSGA-III for solving DTLZ and WFG test problems, where
the runtime of a non-dominated sorting approach on M objectives is ob-
ED

tained by averaging over the runtime consumed by the algorithm for the
embedded NSGA-III running one time on all DTLZ and WFG problems
with M objectives. As shown in the table, the A-ENS is much less time-
PT

consuming than the compared non-dominated sorting approaches on all

considered test instances when they are embedded into NSGA-III. The
higher the number of objectives the MaOP has, the more runtime can be
CE

saved by A-ENS. From Tables 2 and 3, we can conclude that A-ENS can
not only significantly improve the computational efficiency of NSGA-III,
but also enhance the performance of NSGA-III on most DTLZ and WFG
AC

test instances with more than three objectives.

2) Discussion on the reasons for improved performance of NSGA-III/A-ENS
To further understand the influence of A-ENS on the search perfor-
mance of NSGA-III, we conduct additional experiments on two test in-
stances, 4-objective DTLZ2 and 10-objective DTLZ2. The 4-objective DTLZ2
is a representative of test instances on which the accurate non-dominated

24
 ACCEPTED MANUSCRIPT

sorting approach performs better than A-ENS in NSGA-III, while the 10-
objective DTLZ2 is a representative of those on which A-ENS outperforms
the accurate non-dominated sorting approach.
Figure 5 presents the evolutionary trajectories of IGD, GD and ∆ for
NSGA-III using an accurate non-dominated sorting approach and A-ENS,

T
respectively, on 4-objective DTLZ2 and 10-objective DTLZ2, averaging over
30 runs. As can be seen from Figure 5, in terms of both convergence and

IP
diversity, NSGA-III using the accurate non-dominated sorting approach
performs better than NSGA-III using A-ENS on 4-objective DTLZ2, while

CR
NSGA-III using A-ENS outperforms NSGA-III using the accurate non-
dominated sorting approach on 10-objective DTLZ2. From the conver-
gence point of view, however, NSGA-III using A-ENS always performs

US
better than the NSGA-III using the accurate non-dominated sorting ap-
proach on both 4-objective DTLZ2 and 10-objective DTLZ2. On the other
hand, NSGA-III using the accurate non-dominated sorting approach achie-
ves better performance than that using A-ENS on 4-objective DTLZ2 and
AN
10-objective DTLZ2 in terms of solution diversity. These results show that
the proposed A-ENS has a bias toward non-dominated solutions with bet-
ter convergence, which may lead to a slight loss of population diversity
M

when it is adopted in dominance based MOEAs for tackling MaOPs in-

stead of an accurate non-dominated sorting approach. These results are
consistent with the empirical comparisons made in Section 4.
ED

The decrease in solution diversity when A-ENS is adopted in NSGA-

III instead of an accurate non-dominated sorting approach can mainly be
attributed to the fact that A-ENS is an approximate non-dominated sort-
ing approach, which can only identify most but not all non-dominated
PT

solutions in the population combining parent and offspring at each gener-

ation. This means that each of the predefined reference points is likely to
be associated with a farther non-dominated solution in case A-ENS is used
CE

instead of an accurate non-dominated sorting approach in NSGA-III. Fig-

ure 6 plots the ratio of solutions in populations of NSGA-III using A-ENS
and an accurate non-dominated sorting approach at different generations
AC

for solving 4-objective and 10-objective DTLZ2, averaging over 30 runs.

From the figure, we can find that a small ratio of solutions are different in
the populations of NSGA-III using A-ENS and an accurate non-dominated
sorting approach, respectively, although almost all solutions in the popu-
lation of NSGA-III using A-ENS belong to the first non-dominated front.
To further investigate the difference in convergence and diversity of

25
 ACCEPTED MANUSCRIPT

DTLZ2 with 4 objectives DTLZ2 with 10 objectives

0.24
0.7
Accurate approach Accurate approach
A-ENS A-ENS
0.22 0.65

0.2 0.6

0.18 0.55
IGD

IGD

T
0.5
0.16

IP
0.45
0.14
0.4
0.12
0.35

CR
10 50 100 150 200 250 10 50 100 150 200 250
Generations Generations

(a) (b)

×10-3 DTLZ2 with 4 objectives DTLZ2 with 10 objectives

12 0.03

11

10
Accurate approach
A-ENS

US 0.028

0.026

0.024
Accurate approach
A-ENS
AN
9 0.022
GD

GD

0.02
8
0.018

7 0.016

0.014
6
M

10 50 100 150 200 250 10 50 100 150 200 250

Generations Generations

(c) (d)
ED

DTLZ2 with 4 objectives DTLZ2 with 10 objectives

0.8 1
Accurate approach Accurate approach
A-ENS 0.9 A-ENS
0.7

0.8
0.6
PT

0.7
0.5
∆

0.6
0.4
0.5
CE

0.3 0.4

0.2 0.3

10 50 100 150 200 250 10 50 100 150 200 250

Generations Generations

(e) (f)
AC

Figure 5: Evolutionary trajectories of IGD, GD and ∆ for NSGA-III using an accurate

non-dominated sorting and A-ENS, respectively, on 4-objective and 10-objective DTLZ2,
averaging over 30 runs. (a) IGD on 4-objective DTLZ2; (b) IGD on 10-objective DTLZ2;
(c) GD on 4-objective DTLZ2; (d) GD on 10-objective DTLZ2; (e) ∆ on 4-objective DTLZ2;
(f) ∆ on 10-objective DTLZ2.

26
 ACCEPTED MANUSCRIPT

DTLZ2 with 4 objectives DTLZ2 with 10 objectives

Same solutions selected for next population Same solutions selected for next population
Different solutions selected for next population Different solutions selected for next population
Solutions wrongly selected for next population Solutions wrongly selected for next population
1 1 1 1 1 1 1 1 1 1
1 1
Ratio of Solutions

Ratio of Solutions
0.8 0.8

T
0.6 0.6

0.4 0.4

IP
0.2 0.2
0 0 0 0 0 0 0 0 0 0
0 0
50 100 150 200 250 50 100 150 200 250

CR
Generations Generations

(a) (b)

Figure 6: Ratio of solutions in populations of NSGA-III at different generations when

A-ENS and an accurate non-dominated sorting approach are embedded, respectively,
for solving 4-objective and 10-objective DTLZ2, averaging over 30 runs. (a) NSGA-III
using A-ENS and an accurate non-dominated sorting approach on 4-objective DTLZ2; (b)
NSGA-III using A-ENS and an accurate non-dominated sorting approach on 10-objective
US
AN
DTLZ2.

the NSGA-III using A-ENS and using the accurate non-dominated sort-
ing, we check the distance between the solutions and the reference points
M

in NSGA-III. For each reference point, we find the solution that is closest to
the reference line joining the reference point with the origin. Then, we cal-
culate the perpendicular distance between the solution and the reference
ED

line, denoted as d2 , and the distance between the origin and the projected
point of the solution on the reference line, denoted as d1 . It is clear that
if d1 is small, then at least one solution close to the reference point has
PT

good convergence, whereas if d2 is small, at least one solution in the pop-

ulation is close to the reference line, indicating a good diversity. Figure 7
presents the means of d1 − 1 and d2 for each reference point in the popula-
CE

tion at different generations when NSGA-III uses A-ENS and the accurate
non-dominated sorting approach, respectively, for solving 4-objective and
10-objective DTLZ2, averaging over 30 runs. As shown in the figure, it is
not difficult to find that NSGA-III using A-ENS has on average a smaller
AC

d1 and a slightly larger d2 , indicating better convergence but slightly worse

diversity when A-ENS is applied instead of the accurate non-dominated
sorting.
3) Analysis of the roles of three objective comparisons of A-ENS in NSGA-III
Let us recall that at most three objective comparisons are performed in

27
 ACCEPTED MANUSCRIPT

DTLZ2 with 4 objectives DTLZ2 with 10 objectives

0.04 Accurate approach Accurate approach
0.03
A-ENS A-ENS
0.035
0.025
0.03

0.025 0.02
d1

d1

T
0.02 0.015

0.015
0.01

IP
0.01
0.005
0.005

0 0
50 100 150 200 250 50 100 150 200 250

CR
Generations Generations

(a) (b)

DTLZ2 with 4 objectives DTLZ2 with 10 objectives

0.09

0.08

0.07

0.06

0.05
Accurate approach
A-ENS

US 0.35

0.3

0.25

0.2
Accurate approach
A-ENS
AN
d2

d2

0.04 0.15
0.03
0.1
0.02
0.05
0.01

0 0
M

50 100 150 200 250 50 100 150 200 250

Generations Generations

(c) (d)
ED

Figure 7: Means of d1 − 1 and d2 for each reference point in the population of NSGA-III
at different generations using A-ENS and an accurate non-dominated sorting approach,
respectively, in solving 4-objective and 10-objective DTLZ2, averaging over 30 runs. (a)
Mean of d1 − 1 for solving 4-objective DTLZ2; (b) Mean of d1 − 1 for solving 10-objective
PT

DTLZ2; (c) Mean of d2 for solving 10-objective DTLZ2; (d) Mean of d2 for solving 10-
objective DTLZ2.
CE

A-ENS to determine the dominance relationship between a solution p to

be assigned and a solution q having been assigned to a front. The three
objectives are the objective on which p has a minimum value among all
its objective values and the objective on which q has a maximum value
AC

among all its objective values, as well as the means of p and q (except for
the first objective). In the following, we investigate the roles of the three
objective comparisons in A-ENS in determining the dominance relation-
ship between solutions when A-ENS is embedded into NSGA-III. Without
loss of generality, we consider an example where NSGA-III is used to solve

28
 ACCEPTED MANUSCRIPT

DTLZ2 with 4 objectives DTLZ2 with 10 objectives

A-ENS A-ENS
A-ENS without mean A-ENS without mean
A-ENS without max 1 A-ENS without max
1 A-ENS without min A-ENS without min
Random three objectives Random three objectives
0.8
Ratio of Solutions

Ratio of Solutions
0.8

0.6
0.6

T
0.4 0.4

IP
0.2 0.2

0 0
50 100 150 200 250 50 100 150 200 250
Generations Generations

CR
(a) (b)

Figure 8: Ratio of solutions in the first front identified by A-ENS, its three variants, each
using two of the three objective comparisons used by A-ENS, and a sorting strategy that

US
randomly selects three objectives for comparison at different generations, when they are
embedded into NSGA-III that solves the 4-objective and 10-objective DTLZ2. Results are
averaged over independent 30 runs. (a) Ratio of solutions on 4-objective DTLZ2; (b) Ratio
of solutions on 10-objective DTLZ2.
AN

4-objective and 10-objective DTLZ2.

For comparisons, we consider three variants of A-ENS, each using two
M

of the three objective comparisons. In addition, we consider an approx-

imate sorting algorithm that randomly chooses three objectives for com-
parison at each generation in determining the dominance relationship be-
ED

tween solutions. Figure 8 plots the ratio of solutions in the first front,
averaged over 30 independent runs, that are correctly identified to be
in the first front by A-ENS, its three variants and the approximate sort-
ing method that randomly selects three objectives for comparisons at each
PT

generation, when they are embedded into NSGA-III for solving 4-objective
and 10-objective DTLZ2. From this figure, we can find that the A-ENS cor-
rectly identifies much more solutions that belong to the first front than its
CE

variants and the approximate non-dominated sorting method using three

randomly selected objectives. Hence, we can conclude that although A-
ENS is an approximate sorting method, its performance is the best com-
AC

pared other possible approximate sorting strategies considered in this ex-

ample.
In the following, we examine a few additional performance indicators
when the above-mentioned approximate sorting methods are employed
in NSGA-III for solving 4-objective and 10-objective DTLZ2. Figure 9

29
 ACCEPTED MANUSCRIPT

DTLZ2 with 4 objectives DTLZ2 with 10 objectives

0.05 A-ENS A-ENS
0.045
A-ENS without mean A-ENS without mean
0.045 A-ENS without max A-ENS without max
0.04
A-ENS without min A-ENS without min
0.04
Random three objectives 0.035 Random three objectives
0.035
0.03
0.03
GD

GD
0.025
0.025

T
0.02
0.02
0.015
0.015
0.01

IP
0.01
0.005
0.005
10 50 100 150 200 250 10 50 100 150 200 250
Generations Generations

CR
(a) (b)
DTLZ2 with 4 objectives DTLZ2 with 10 objectives
1.3
1.6 A-ENS A-ENS
A-ENS without mean 1.2 A-ENS without mean
A-ENS without max

US
A-ENS without max
1.4 A-ENS without min A-ENS without min
1.1
Random three objectives Random three objectives
1.2 1

0.9
1
∆
∆

0.8
0.8
AN
0.7

0.6 0.6

0.5
0.4
0.4
10 50 100 150 200 250 10 50 100 150 200 250
Generations Generations
M

(c) (d)
DTLZ2 with 4 objectives DTLZ2 with 10 objectives
0.8
A-ENS A-ENS
1.6
A-ENS without mean A-ENS without mean
0.7
ED

A-ENS without max A-ENS without max

A-ENS without min 1.4 A-ENS without min
Random three objectives Random three objectives
0.6
1.2
0.5
IGD

IGD

1
0.4
0.8
PT

0.3
0.6
0.2
0.4
0.1
10 50 100 150 200 250 10 50 100 150 200 250
Generations Generations
CE

(e) (f)

Figure 9: Evolutionary trajectories of GD, ∆ and IGD for NSGA-III using A-ENS and
other four approximate non-dominated sorting strategies in solving 4-objective and 10-
AC

objective DTLZ2, averaged over 30 runs. (a) GD on 4-objective DTLZ2; (b) GD on 10-
objective DTLZ2; (c) ∆ on 4-objective DTLZ2; (d) ∆ on 10-objective DTLZ2; (e) IGD on
4-objective DTLZ2; (e) IGD on 10-objective DTLZ2.

30
 ACCEPTED MANUSCRIPT

presents the evolutionary trajectories of GD, ∆ and IGD for NSGA-III us-
ing A-ENS and other four approximate non-dominated sorting strategies,
for solving 4-objective and 10-objective DTLZ2. The presented results are
averaged over 30 independent runs. From this figure, we can find that
the best convergence result is obtained by the A-ENS variant that does not

T
consider the objective on which it has a maximum objective values among
all its objectives values, while A-ENS achieves the best solution diversity.

IP
This might be attributed to the fact that A-ENS can correctly identify more
solutions in the first front than the other approximate sorting algorithms.

CR
As can be seen from Figure 9, A-ENS is able to achieve a better bal-
ance between convergence and diversity compared to other approximate
sorting methods that adopt two or three objective comparisons, when they

US
are used by NSGA-III for solving MaOPs. It is necessary to stress that the
proposed A-ENS can also sort a population according to another objective
or a randomly chosen objective at each generation, however our empirical
results indicate that A-ENS is fairly insensitive to the chosen objective to
AN
sort the population when it is embedded into MOEAs.

5.2. Other dominance based MOEAs using A-ENS

M

Here we study the optimization performance of two additional recently

reported dominance based MOEAs for solving MaOPs, namely, GrEA [54]
and PICEA-g [51] using A-ENS for non-dominated sorting. This study
ED

aims to demonstrate that the enhanced optimization performance observed

in NSGA-III using A-ENS is generic to other dominance based MOEAs
for solving MaOPs. We denote the two MOEAs using A-ENS as GrEA/A-
ENS and PICEA-g/A-ENS, respectively. All parameters of GrEA and PICEA-
PT

g in the following experiments are set to those recommended in [51, 54].

For each test problem, the population size of GrEA and PICEA-g is set to
the same as that of NSGA-III, which is presented in Table 1. The number
CE

of divisions div recommended in [59] is adopted in GrEA on DTLZ and

WFG test suites.
Tables 4 and 5 present the mean and standard deviation of HVs of
AC

GrEA and GrEA/A-ENS, PICEA-g and PICEA-g/A-ENS on DTLZ1 to

DTLZ7 and WFG1 to WFG9. All results are averaged over 30 indepen-
dent runs, where the better mean between the two compared algorithms
is highlighted. The Wilcoxon’s rank sum test at a significance level of 0.05
was also adopted to test the statistical significance of the differences be-
tween the results obtained by the two algorithms. From the tables, we

31
 ACCEPTED MANUSCRIPT

Table 4: HVs of GrEA and GrEA/A-ENS on DTLZ1 to DTLZ7 and WFG1 to WFG9, where
the best mean for each test instance is highlighted.
4-objective 6-objective 8-objective 10-objective
Problem GrEA/ GrEA/ GrEA/ GrEA/
GrEA A-ENS GrEA A-ENS GrEA A-ENS GrEA A-ENS

T
6.4E-1 7.3E-1† 5.4E-1 9.9E-1† 1.0E+0 9.9E-1 1.0E+0† 1.0E+0
DTLZ1 (1.8E-1) (1.1E-1) (2.8E-1) (3.1E-2) (9.6E-6) (3.7E-2) (0.0E+0) (7.0E-5)

IP
6.0E-1 6.0E-1† 7.4E-1 7.5E-1† 9.1E-1 9.3E-1† 9.4E-1 9.4E-1†
DTLZ2 (1.4E-3) (1.6E-3) (2.4E-3) (1.5E-3) (6.1E-3) (1.8E-3) (1.6E-3) (1.4E-3)
1.2E-1 2.2E-1† 1.0E+0 9.7E-1 1.0E+0 1.0E+0 1.0E+0 1.0E+0
DTLZ3 (1.5E-1) (1.9E-1) (2.4E-4) (8.3E-2) (0.0E+0) (0.0E+0) (0.0E+0) (0.0E+0)

CR
5.8E-1 5.9E-1† 8.2E-1 8.2E-1† 8.8E-1 8.8E-1† 9.3E-1 9.3E-1†
DTLZ4 (6.0E-2) (4.7E-2) (2.4E-3) (1.0E-2) (3.8E-3) (3.2E-3) (6.1E-3) (5.8E-3)
2.7E-1 2.7E-1 7.2E-1 7.2E-1 8.0E-1 8.1E-1† 8.2E-1 8.2E-1
DTLZ5 (7.5E-4) (7.8E-4) (5.2E-3) (5.5E-3) (9.3E-3) (9.1E-3) (1.7E-2) (3.6E-2)
6.8E-1† 6.7E-1 9.6E-1† 9.5E-1
US
9.6E-1 9.4E-1 9.6E-1
DTLZ6 (9.8E-4) (3.5E-3) (5.4E-3) (2.3E-2) (1.1E-2) (4.3E-2) (1.7E-2) (1.9E-2)
1.8E-1† 1.7E-1 1.7E-1† 1.6E-1 3.1E-1 3.5E-1† 1.9E-1
DTLZ7 (2.1E-3) (2.1E-3) (9.4E-3) (4.8E-3) (1.1E-2) (8.2E-3) (2.5E-2) (1.2E-2)
9.7E-1†

2.9E-1†
AN
9.1E-1 9.3E-1† 9.3E-1 9.7E-1† 9.6E-1 9.8E-1† 9.5E-1 9.8E-1†
WFG1 (1.0E-1) (6.9E-2) (1.5E-1) (6.6E-2) (7.9E-2) (3.4E-2) (1.2E-1) (6.1E-2)
9.0E-1 9.3E-1† 9.5E-1 9.6E-1† 9.6E-1 1.0E+0† 9.7E-1 1.0E+0†
WFG2 (6.9E-2) (5.5E-2) (3.3E-2) (7.5E-2) (8.2E-3) (1.1E-3) (1.2E-2) (7.2E-4)
M

5.6E-1† 5.5E-1 2.8E-1 3.0E-1† 2.6E-1 2.9E-1† 2.7E-1 2.7E-1

WFG3 (6.7E-3) (5.5E-3) (7.0E-3) (2.8E-3) (8.7E-3) (2.4E-3) (4.6E-3) (4.7E-3)
5.3E-1 5.3E-1† 6.4E-1 6.6E-1† 6.9E-1 7.3E-1† 8.2E-1 8.6E-1†
WFG4 (7.5E-3) (5.8E-3) (1.2E-2) (1.1E-2) (8.2E-3) (5.5E-3) (1.4E-2) (2.3E-2)
ED

5.2E-1 5.2E-1† 6.4E-1 6.5E-1† 6.8E-1 7.1E-1† 8.0E-1 8.2E-1†

WFG5 (4.6E-3) (4.1E-3) (1.1E-2) (9.2E-3) (1.2E-2) (8.0E-3) (2.6E-2) (3.5E-2)
5.0E-1 5.1E-1 6.3E-1 6.5E-1† 6.4E-1 6.8E-1† 7.8E-1 8.1E-1†
WFG6 (1.6E-2) (1.5E-2) (1.9E-2) (1.8E-2) (2.0E-2) (1.7E-2) (2.9E-2) (2.1E-2)
PT

5.6E-1 5.6E-1† 6.9E-1 7.2E-1† 7.3E-1 7.7E-1† 8.7E-1 9.1E-1†

WFG7 (2.8E-3) (2.4E-3) (6.8E-3) (4.9E-3) (8.2E-3) (5.6E-3) (1.9E-2) (1.1E-2)
4.7E-1 4.7E-1† 5.6E-1 5.7E-1 5.1E-1 5.9E-1† 7.4E-1 7.4E-1
WFG8 (8.1E-3) (6.2E-3) (4.7E-2) (3.1E-2) (2.9E-2) (3.9E-2) (1.3E-2) (3.2E-2)
CE

5.1E-1 5.2E-1† 6.0E-1 6.1E-1† 6.8E-1 7.1E-1† 8.1E-1 8.3E-1†

WFG9 (5.3E-3) (5.7E-3) (3.9E-2) (5.1E-2) (1.3E-2) (6.7E-3) (2.0E-2) (3.8E-2)
”†”
indicates that the two results are significantly different at a level of 0.05 by the Wilcoxon’s rank sum test.
AC

can find that both GrEA and PICEA-g using A-ENS perform better than
the original MOEA using the accurate non-dominated sorting approach
on most test instances from DTLZ and WFG suites. These results confirm
that A-ENS results in better optimization performance than the accurate
non-dominated sorting approach in GrEA and PICEA-g.

32
 ACCEPTED MANUSCRIPT

Table 5: HVs of PICEA-g and PICEA-g/A-ENS on DTLZ1 to DTLZ7 and WFG1 to WFG9,
where the best mean for each test instance is highlighted.
4-objective 6-objective 8-objective 10-objective
Problem PICEA-g/ PICEA-g/ PICEA-g/ PICEA-g/
PICEA-g A-ENS PICEA-g A-ENS PICEA-g A-ENS PICEA-g A-ENS

T
7.9E-2 1.7E-1† 1.9E-1 4.0E-1† 1.0E-1 1.8E-1† 2.4E-1 2.5E-1
DTLZ1 (8.6E-2) (1.3E-1) (1.7E-1) (2.5E-1) (7.2E-2) (1.4E-1) (1.5E-1) (1.3E-1)

IP
5.4E-1 5.4E-1 7.3E-1 7.3E-1† 8.4E-1 8.4E-1 9.1E-1 9.3E-1†
DTLZ2 (6.0E-3) (6.9E-3) (9.5E-3) (9.4E-3) (3.4E-2) (9.9E-3) (4.3E-2) (4.3E-3)
1.3E-2 1.8E-2 5.0E-1 8.4E-1† 9.3E-1 9.7E-1† 9.9E-1 1.0E+0†
DTLZ3 (5.6E-3) (2.9E-2) (3.9E-1) (1.8E-1) (1.9E-1) (1.2E-1) (1.3E-2) (2.4E-3)

CR
5.5E-1 5.6E-1 7.1E-1 7.3E-1 8.3E-1 8.3E-1 9.2E-1 9.2E-1
DTLZ4 (6.2E-2) (3.9E-2) (5.5E-2) (3.3E-2) (1.7E-2) (1.7E-2) (5.2E-3) (5.8E-3)
4.8E-1† 4.6E-1 5.3E-1† 5.1E-1 4.9E-1† 4.8E-1 4.5E-1† 4.1E-1
DTLZ5 (2.1E-2) (3.9E-2) (9.9E-3) (1.2E-2) (2.6E-3) (3.3E-3) (6.1E-3) (1.0E-2)
7.6E-1 7.7E-1 8.5E-1† 8.3E-1
US
8.6E-1† 8.3E-1 9.5E-1†
DTLZ6 (4.2E-2) (4.7E-3) (4.1E-2) (1.6E-2) (6.2E-3) (1.7E-2) (2.4E-3) (5.3E-3)
1.4E-1† 1.1E-1 6.5E-2 6.2E-2 5.0E-2 5.1E-2 7.4E-2†
DTLZ7 (3.0E-2) (2.3E-2) (6.2E-3) (6.8E-3) (4.2E-3) (5.3E-3) (4.9E-3) (3.7E-3)
9.4E-1

7.1E-2
AN
9.7E-1 9.7E-1 1.0E+0 9.9E-1 9.9E-1† 9.8E-1 1.0E+0† 9.8E-1
WFG1 (4.6E-2) (5.0E-2) (3.1E-3) (1.7E-2) (2.9E-2) (6.4E-2) (1.5E-3) (6.7E-2)
9.5E-1 9.5E-1 9.4E-1† 9.3E-1 9.9E-1† 9.8E-1 1.0E+0† 9.8E-1
WFG2 (6.4E-2) (7.3E-2) (1.0E-1) (9.9E-2) (2.4E-3) (9.5E-3) (3.9E-3) (3.2E-2)
M

4.2E-1† 3.8E-1 2.3E-1† 1.0E-1 1.7E-1† 9.1E-2 2.2E-1† 1.0E-1

WFG3 1.6E-2) (3.8E-2) (4.9E-2) (1.7E-2) (6.3E-2) (1.4E-2) (3.3E-2) (1.4E-2)
5.3E-1 5.3E-1 4.9E-1 6.0E-1† 7.0E-1 7.5E-1† 8.0E-1 8.2E-1
WFG4 (5.1E-3) (5.2E-3) (7.1E-2) (7.4E-2) (6.6E-2) (6.2E-2) (5.1E-2) (4.3E-2)
ED

5.0E-1 5.0E-1 6.6E-1 6.6E-1† 7.6E-1 7.7E-1† 8.4E-1 8.3E-1

WFG5 (5.3E-3) (3.2E-3) (5.4E-3) (4.2E-3) (1.6E-2) (4.4E-3) (1.9E-2) (2.5E-2)
4.8E-1 4.8E-1 6.5E-1 6.5E-1 7.4E-1 7.4E-1 8.126E-1 8.2E-1
WFG6 (1.4E-2) (1.3E-2) (1.7E-2) (1.3E-2) (2.6E-2) (2.2E-2) (2.9E-2) (2.5E-2)
PT

5.3E-1 5.3E-1 6.7E-1 7.0E-1† 8.0E-1 8.2E-1† 8.9E-1 9.1E-1†

WFG7 (5.5E-3) (3.5E-3) (5.1E-2) (2.6E-2) (4.6E-2) (5.3E-3) (4.1E-2) (2.4E-2)
4.3E-1 4.4E-1 5.4E-1† 5.3E-1 6.5E-1† 6.4E-1 7.5E-1† 7.0E-1
WFG8 (8.7E-3) (4.8E-3) (5.1E-3) (1.5E-2) (1.3E-2) (2.5E-2) (1.8E-2) (2.3E-2)
CE

5.0E-1 5.0E-1 5.8E-1 6.2E-1 7.6E-1 7.6E-1 6.1E-01 8.2E-1†

WFG9 (3.0E-2) (2.6E-2) (8.1E-1) (4.7E-2) (2.2E-2) (8.0E-3) (1.4E-2) (1.9E-2)
”†”
indicates that the two results are significantly different at a level of 0.05 by the Wilcoxon’s rank sum test.
AC

Table 6 presents the runtime of five different non-dominated sorting

approaches in GrEA and PICEA-g for solving DTLZ and WFG test prob-
lems, where the runtime of a non-dominated sorting approach on M ob-
jectives is obtained by averaging over the runtime consumed by the algo-
rithm for the MOEA running one time on all DTLZ and WFG problems

33
 ACCEPTED MANUSCRIPT

Table 6: Runtime of five different non-dominated sorting approaches in GrEA and

PICEA-g for solving DTLZ and WFG test problems, where the runtime of a non-
dominated sorting approach on M objectives is obtained by averaging over the runtime
consumed by the algorithm for the MOEA running one time on all DTLZ and WFG prob-
lems with M objectives.

T
Non-dominated sorting approach
MOEA obj. fast non- deductive corner

IP
dominated sort sort sort ENS-SS A-ENS

4 49.42 18.82 19.32 13.87 5.90

CR
6 62.43 25.77 26.29 19.61 6.32
GrEA 8 92.95 40.14 40.14 30.84 8.36
10 307.68 135.89 132.68 106.54 20.58

PICEA-g
4
6
8
48.98
63.96
100.71
US 18.06
26.01
41.30
18.46
26.33
40.72
13.13
19.46
31.88
5.55
5.77
6.39
AN
10 336.97 143.58 137.86 113.62 15.17

with M objectives. From the table, we can find that the efficiency of A-
M

ENS is much higher than the compared accurate non-dominated sorting

approaches when they are adopted by GrEA and PICEA-g on all consid-
ered test instances.
ED

From the above empirical studies, we confirm that A-ENS is not only
computationally more efficient, but also able to enhance the search perfor-
mance of dominance based MOEAs such as GrEA, NSGA-III and PICEA-
g. Therefore, we conclude that A-ENS is a promising non-dominated
PT

sorting method compared to existing accurate non-dominated sorting ap-

proaches for dominance based MOEAs to solve MaOPs.
CE

6. Conclusions
In this paper, an approximate non-dominated sorting approach, termed
AC

A-ENS, has been proposed for dominance based MOEAs to tackle MaOPs.
As a maximum of three objective comparisons are performed to determine
the dominance relationship between two solutions, A-ENS is able to effec-
tively reduce the computational cost for non-dominated sorting, especially
when the number of objectives is large. Theoretical analysis indicates√that
the time complexity of A-ENS is O(N 2 ) in the worst case and O(N N )

34
 ACCEPTED MANUSCRIPT

in the best case, where N is the population size, for any MaOPs having
more than three objectives. Comparative studies have been conducted on
three state-of-the-art dominance based MOEAs using A-ENS and accurate
sorting methods, respectively, for solving two suites of widely used many-
objective test problems. Simulation results confirm that the proposed A-

T
ENS can considerably enhance computational efficiency of the MOEAs in
solving MaOPs. In addition, these results suggest A-ENS can enhance the

IP
search performance of dominance based MOEAs, especially when they
are employed to solve MaOPs.

CR
The success of A-ENS proposed in this work indicates that approxi-
mate non-dominated sorting may be of great interest for designing effi-
cient and effective evolutionary many-objective optimization. Preliminary

US
quantitative and qualitative analysis have been provided in this work to
understand the performance enhancement resulting in by A-ENS, never-
theless, it should be noted that a full understanding of the influence of
approximate sorting on the search performance remains open, partly due
AN
to the stochastic nature of the search mechanisms in all metaheuristics
such as evolutionary algorithms. Future work will be devoted to devel-
oping new efficient and effective MOEAs for solving MaOPs using ap-
M

proximate non-dominated sorting. For example, it might be of interest

to combine accurate and approximate non-dominated sorting strategies
at different search stages. Furthermore, it may be worthwhile to develop
ED

new MOEAs for MaOPs that can fully exploit the benefit of approximate
non-dominated sorting methods.

Acknowledgments
PT

Valuable suggestions in improving the experimental design from Dr.

Handing Wang are gratefully acknowledged. This work was supported in
CE

part by National Natural Science Foundation of China (61272152, 61502004

and 61502001), and the Joint Research Fund for Overseas Chinese, Hong
Kong and Macao Scholars of the National Natural Science Foundation of
China (61428302).
AC

References
[1] J. Bader and E. Zitzler. HypE: an algorithm for fast hypervolume-
based many-objective optimization. Evolutionary Computation,
19(1):45–76, 2011.

35
 ACCEPTED MANUSCRIPT

[2] N. Beume, B. Naujoks, and M. Emmerich. SMS-EMOA: multiobjec-

tive selection based on dominated hypervolume. European Journal of
Operational Research, 181(3):1653–1669, 2007.

[3] K. Bringmann and T. Friedrich. Approximation quality of the hyper-

T
volume indicator. Artificial Intelligence, 195:265–290, 2013.

IP
[4] K. Bringmann, T. Friedrich, C. Igel, and T. Voss. Speeding up many-
objective optimization by Monte Carlo approximations. Artificial In-
telligence, 204:22–29, 2013.

CR
[5] D. Brockhoff, T. Friedrich, N. Hebbinghaus, C. Klein, F. Neumann,
and E. Zitzler. On the effects of adding objectives to plateau functions.

US
IEEE Transactions on Evolutionary Computation, 13(3):591–603, 2009.

[6] R. Cheng, Y. Jin, K. Narukawa, and B. Sendhoff. A multiobjective evo-

lutionary algorithm using gaussian process-based inverse modeling.
AN
IEEE Transactions on Evolutionary Computation, 19(6):838–856, 2015.

[7] R. Cheng, Y. Jin, M. Olhofer, and B. Sendhoff. A reference vec-

tor guided evolutionary algorithm for many-objective optimization.
M

IEEE Transactions on Evolutionary Computation, in press, 2016.

[8] K. M. Clymont and E. Keedwell. Deductive sort and climbing sort:

ED

new methods for non-dominated sorting. Evolutionary Computation,

20(1):1–26, 2012.

[9] K. Deb and H. Jain. An evolutionary many-objective optimization

PT

algorithm using reference-point based non-dominated sorting ap-

proach, part I: solving problems with box constraints. IEEE Trans-
actions on Evolutionary Computation, 18(4):577–601, 2014.
CE

[10] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and elitist

multi-objective genetic algorithm: NSGA-II. IEEE Transactions on Evo-
lutionary Computation, 6(2):182–197, 2002.
AC

[11] K. Deb, A. Sinha, P. J. Korhonen, and J. Wallenius. An interactive

evolutionary multi-objective optimization method based on progres-
sively approximated value functions. IEEE Transactions on Evolution-
ary Computation, 14(5):723–739, 2010.

36
 ACCEPTED MANUSCRIPT

[12] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler. Scalable test problems

for evolutionary multiobjective optimization. Springer, 2005.

[13] K. Deb and S. Tiwari. Omni-optimizer: a procedure for single and

multi-objective optimization. In Proceedings of the Third international

T
conference on Evolutionary Multi-Criterion Optimization, pages 47–61,
2005.

IP
[14] F. di Pierro, S.-T. Khu, and D. A. Savić. An investigation on preference
order ranking scheme for multiobjective evolutionary optimization.

CR
IEEE Transactions on Evolutionary Computation, 11(1):17–45, 2007.

[15] M. Drozdı́k, Y. Akimoto, H. Aguirre, and K. Tanaka. Computational

US
cost reduction of non-dominated sorting using M-front. IEEE Trans-
actions on Evolutionary Computation, in press, 2014.

[16] J. Du, Z. Cai, and Y. Chen. A sorting based algorithm for finding non-
AN
dominated set in multi-objective optimization. In Proceedings of the
Third International Conference on Natural Computation, pages 436–440,
2007.
M

[17] M. Ehrgott. Multiobjective optimization. AI Magazine, 29(4):47–57,

2009.
ED

[18] H. Fang, Q. Wang, Y. Tu, and M. F. Horstemeyer. An efficient non-

dominated sorting method for evolutionary algorithms. Evolutionary
Computation, 16(3):355–384, 2008.
PT

[19] P. J. Fleming, R. C. Purshouse, and R. J. Lygoe. Many-objective op-

timization: an engineering design perspective. In Proceedings of the
Third International Conference on Evolutionary multi-criterion optimiza-
CE

tion, pages 14–32, 2005.

[20] F.-A. Fortin, S. Grenier, and M. Parizeau. Generlizing the improved

run-time complexity algorithm for non-dominated sorting. In Pro-
AC

ceedings of the fifteenth annual conference on Genetic and evolutionary com-

putation, pages 615–622, 2013.

[21] A. E. Gerevini, A. Saetti, and I. Serina. An approach to efficient plan-

ning with numerical fluents and multi-criteria plan quality. Artificial
Intelligence, 172(8):899–944, 2008.

37
 ACCEPTED MANUSCRIPT

[22] D. Hadka and P. Reed. Diagnostic assessment of search controls and

failure modes in many-objective evolutionary optimization. Evolu-
tionary Computation, 20(3):423–452, 2012.

[23] K. Ikeda, H. Kita, and S. Kobayashi. Failure of Pareto-based MOEAs:

T
does non-dominated really mean near to optimal? In Proceedings of
the 2001 IEEE Congress on Evolutionary Computation, pages 957–962,

IP
2001.

[24] M. T. Jensen. Reducing the run-time complexity of multiobjective

CR
EAs: the NSGA-II and other algorithms. IEEE Transactions on Evolu-
tionary Computation, 7(5):503–515, 2003.

US
[25] V. Khare, X. Yao, and K. Deb. Performance scaling of multiobjective
evolutionary algorithms. In Proceedings of the Second International Con-
ference on Evolutionary Multi-Criterion Optimization, pages 376–390,
2003.
AN
[26] M. Laumanns, L. Thiele, K. Deb, and E. Zitzler. Combining conver-
gence and diversity in evolutionary multiobjective optimization. Evo-
lutionary Computation, 10(3):263–282, 2002.
M

[27] B. Li, J. Li, K. Tang, and X. Yao. Many-objective evolutionary algo-

rithms: A survey. ACM Computing Surveys, 48(1):1–35, 2015.
ED

[28] K. Li, K. Deb, Q. Zhang, and S. Kwong. Efficient non-domination

level update approach for steady-state evolutionary multiobjective
optimization. Technical report, Department of Electtrical and Com-
PT

puter Engineering, Michigan State University, East Lansing, USA,

Tech. Rep. COIN Report Number 2014014, 2014.
CE

[29] M. Li, S. Yang, and X. Liu. Bi-goal evolution for many-objective opti-
mization problems. Artificial Intelligence, 228:45–65, 2015.

[30] Y. Li, Z. H. Zhan, S. Lin, J. Zhang, and X. Luo. Competitive and coop-
AC

erative particle swarm optimization with information sharing mecha-

nism for global optimization problems. Information Sciences, 293:370–
382, 2015.

[31] Q. Liu, W. Wei, H. Yuan, Z. H. Zhan, and Y. Li. Topology selection for
particle swarm optimization. Information Sciences, 363:154–173, 2016.

38
 ACCEPTED MANUSCRIPT

[32] M. Lukasiewycz, M. Glaß, C. Haubelt, and J. Teich. Symbolic archive

representation for a fast nondominance test. In Proceedings of the 4th
International Conference on Evolutionary Multi-Criterion Optimization,
pages 111–125, 2007.

T
[33] M. Negnevitsky. Artificial intelligence: a guide to intelligent systems.
Pearson Education, 2005.

IP
[34] R. C. Purshouse and P. J. Fleming. Evolutionary many-objective opti-
mization: an exploratory analysis. In Proceedings of 2003 Congress on

CR
Evolutionary Computation, pages 2066–2073, 2003.
[35] R. C. Purshouse and P. J. Fleming. On the evolutionary optimization
of many conflicting objectives. IEEE Transactions on Evolutionary Com-
putation, 11(6):770–784, 2007.
US
[36] B. Y. Qu and P. N. Suganthan. Multi-objective evolutionary algo-
AN
rithms based on the summation of normalized objectives and diver-
sified selection. Information sciences, 180(17):3170–3181, 2010.
[37] L. Barone S. Huband, P. Hingston and L. While. A review of mul-
tiobjective test problems and a scalable test problem toolkit. IEEE
M

Transactions on Evolutionary Computation, 10(5):477–506, 2006.

[38] D. K. Saxena and K. Deb. Dimensionality reduction of objectives and
ED

constraints in multi-objective optimization problems: a system de-

sign perspective. In Proceedings of the 2008 IEEE Congress on Evolu-
tionary Computation, pages 3204–3211, 2008.
PT

[39] O. Schütze. A new data structure for the nondominance problem

in multi-objective optimization. In Proceedings of the Second Interna-
tional Conference on Evolutionary Multi-Criterion Optimization, pages
CE

509–518, 2003.
[40] C. Shi, M. Chen, and Z. Shi. A fast nondominated sorting algorithm.
In Proceedings of 2005 International Conference on Neural Networks and
AC

Brain, pages 1605–1610, 2005.

[41] H. K. Singh, A. Isaacs, and T. Ray. A Pareto corner search evolution-
ary algorithm and dimensionality reduction in many-objective opti-
mization problems. IEEE Transactions on Evolutionary Computation,
15(4):539–556, 2011.

39
 ACCEPTED MANUSCRIPT

[42] N. Srinivas and K. Deb. Multiobjective optimization using non-

dominated sorting in genetic algorithms. Evolutionary Computation,
2(3):221–248, 1995.

[43] K. C. Tan, T. H. Lee, and E. F. Khor. Evolutionary algorithms for multi-

T
objective optimization: performance assessments and comparisons.
Artificial intelligence review, 17(4):251–290, 2002.

IP
[44] K. Tang, P. Yang, and X. Yao. Negatively correlated search. IEEE
Journal on Selected Areas in Communications, 34(3):542–550, 2016.

CR
[45] S. Tang, Z. Cai, and J. Zheng. A fast method of constructing the non-
dominated set: arena’s principle. In Proceedings of the Fourth Interna-

US
tional Conference on Natural Computation, pages 391–395, 2008.

[46] D. A. V. Veldhuizen and G. B. Lamont. Multiobjective evolutionary

algorithm research: A history and analysis. Technical report, De-
AN
partment of Electrical and Computer Engineering. Graduate School
of Engineering, Air Force Inst Technol, Wright Patterson, Tech. Rep.
TR-98-03, 1998.
M

[47] G. Wang and H. Jiang. Fuzzy-dominance and its application in evo-

lutionary many objective optimization. In Proceedings of the 2007 In-
ternational Conference on Computational Intelligence and Security Work-
ED

shops, pages 195–198, 2007.

[48] H. Wang, Y. Jin, and X. Yao. Diversity assessment in many-objective

optimization. IEEE Transactions on Cybernetics, in press, 2016.
PT

[49] H. Wang and X. Yao. Corner sort for Pareto-based many-objective

optimization. IEEE Transactions on Cybernetics, 44(1):92–102, 2014.
CE

[50] H. Wang, Q. Zhang, L. Jiao, and X. Yao. Regularity model for

noisy multiobjective optimization. IEEE Transactions on Cybernetics,
in press, 2015.
AC

[51] R. Wang, R. C. Purshouse, and P. J. Fleming. Preference-inspired

coevolutionary algorithms for many-objective optimization. IEEE
Transactions on Evolutionary Computation, 17(4):474–494, 2013.

40
 ACCEPTED MANUSCRIPT

[52] Y. Wang, L. Wu, and X. Yuan. Multi-objective self-adaptive differen-

tial evolution with elitist archive and crowding entropy-based diver-
sity measure. Soft Computing, 14(3):193–209, 2010.

[53] L. While, P. Hingston, L. Barone, and S. Huband. A faster algorithm

T
for calculating hypervolume. IEEE Transactions on Evolutionary Com-
putation, 10(1):29–38, 2006.

IP
[54] S. Yang, M. Li, X. Liu, and J. Zheng. A grid-based evolutionary al-
gorithm for many-objective optimization. IEEE Transactions on Evolu-

CR
tionary Computation, 17(5):721–736, 2013.

[55] Y. Yuan, H. Xu, B. Wang, and X. Yao. A new dominance relation

US
based evolutionary algorithm for many-objective optimization. IEEE
Transactions on Evolutionary Computation, in press, 2015.

[56] Z.H. Zhan, J. Li, J. Cao, J. Zhang, H. Chung, and Y.H. Shi. Multiple
AN
populations for multiple objectives: A coevolutionary technique for
solving multiobjective optimization problems. IEEE Transactions on
Cybernetics, 43(2):445–463, 2013.
M

[57] Q. Zhang and H. Li. MOEA/D: a multi-objective evolutionary al-

gorithm based on decomposition. IEEE Transactions on Evolutionary
Computation, 11(6):712–731, 2007.
ED

[58] X. Zhang, Y. Tian, R. Cheng, and Y. Jin. An efficient approach

to non-dominated sorting for evolutionary multi-objective optimiza-
tion. IEEE Transactions on Evolutionary Computation, 19(2):201–213,
PT

2015.

[59] X. Zhang, Y. Tian, and Y. Jin. A knee point driven evolutionary al-
CE

gorithm for many-objective optimization. IEEE Transactions on Evolu-

tionary Computation, 19(6):761–776, 2015.

[60] J. Zheng, C. X. Ling, Z. Shi, and Y. Xie. Some discussions about MO-
AC

GAs: individual relations, non-dominated set, and application on au-

tomatic negotiation. In Proceedings of 2004 IEEE Congress on Evolution-
ary Computation, pages 706–712, 2004.

[61] X. Zhou, J. Shen, and J. Shen. An immune recognition based algo-

rithm for finding non-dominated set in multi-objective optimization.

41
 ACCEPTED MANUSCRIPT

In Proceedings of 2008 IEEE Pacific-Asia Workshop on Computational In-

telligence and Industrial Application, pages 305–310, 2008.

[62] E. Zitzler and S. Künzli. Indicator-based selection in multiobjective

search. In Proceedings of the 8th International Conference on Parallel Prob-

T
lem Solving from Nature, pages 832–842, 2004.

IP
[63] E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, and V. G. D.
Fonseca. Performance assessment of multiobjective optimizers: an
analysis and review. IEEE Transactions on Evolutionary Computation,

CR
7(2):117–132, 2003.

[64] X. Zou, Y. Chen, M. Liu, and L. Kang. A new evolutionary algorithm

US
for solving many-objective optimization problems. IEEE Transactions
on Systems, Man, and Cybernetics-Part B, 38(5):1402–1412, 2008.
AN
M
ED
PT
CE
AC

42