Multi-Objective Evolutionary Algorithms For Classification: A Review
Multi-Objective Evolutionary Algorithms For Classification: A Review
Multi-Objective Evolutionary Algorithms For Classification: A Review
Master in Computer Engineering, Department of Computer Engineering, MIT, Pune, Pune University
Gadiya Estate, Sno: 81/6, Paud Road, Kothrud Depo, Pune 038
2
ABSTRACT
Multi-objective evolutionary algorithms are evolutionary systems which are used for optimizing various measures of the
evolving systems. Most of the real life data mining problems are optimization problems, where the aim is to evolve a candidate
model that optimizes certain performance criteria. Classification problem can be thought of as multi-objective problem as it
may require to optimize accuracy, model complexity, interestingness, misclassification rate, sensitivity, specificity etc. The
performance of these MOEAs used is depends on various characteristics like evolutionary techniques used, chromosome
representation, parameters like population size, crossover rate, mutation rate, stopping criteria, number of generations,
objectives taken for optimization, fitness function used, optimization strategy etc. This paper reports the comprehensive survey
on recent developments in the multi-objective evolutionary algorithms for classification problems.
1. INTRODUCTION
Data Mining is the process of discovering interesting and important patterns from large datasets. The main objective of
any data mining process is to build an efficient predictive or descriptive model of huge volume of data. This model
must best fit the data also able to generalize to new data. There are different types of tasks associated to data mining
process i.e. classification, clustering, association rule mining etc. Due to the complexity of classification problems,
cannot be solved using standard mathematical techniques. Evolutionary algorithms have been found to be useful in
automatic processing of large volume of raw noisy data due to their inherent parallel architecture [1], [2]. Traditionally,
evolutionary algorithms were used to solve the single-objective classification problems. But there are many real life
classification problems having multiple conflicting objectives, which need to be optimized simultaneously to obtain
optimal solutions. Conflicting objectives for classification problems can be accuracy, sensitivity, specificity,
misclassification rate, mean squared error etc. Therefore, the concept of multi-objective optimization is highly
applicable to classification problems. Classification problems are multi-objective in nature, and the goal is to
simultaneously optimize all conflicting objectives. Optimum performance in one objective results in low performance in
one or more of the other objectives, creating necessity for compromise [1]. For solving multi-objective optimization
problems, number of evolutionary algorithms (EAs) have been proposed in the literature [ref should add]. In singleobjective optimization single optimum solution is generated in final generation, while in multi-objective optimization,
set of non-dominated solutions are generated in final generation, where each objective can be improved only by
degradation of at least one of the other objectives. Multi-objective evolutionary algorithms (MOEAs) [3] have become
popular in data mining. MOEAs can be found in the literature for solving data mining task such as classification,
clustering, association rule mining. A variety of MOEAs used for different classification problems like in medical field.
Different classification techniques such as Decision tree, Rule based classifier, neural network, Bayesian network,
Support vector machine can be used for predicting class labels for unknown tuples. In this paper, we attempt to make
comprehensive survey of the recent developments in the MOEAs for solving different classification problems. Section 2
gives detailed introduction of multi-objective optimization. Evolutionary computing approaches for multi-objective
optimization provided in section 3. Issues need to be considered during implementation of multi-objective evolutionary
algorithms are given in section 4. Section 5 gives review of different classification problems solved using different
multi-objective optimization. Conclusion and Future scope drawn in section 6.
Page 292
2. MULTI-OBJECTIVE OPTIMIZATION
This section introduced some basic concepts of MOO. Then, an overview of available MOEAs is provided.
2.1 Basic Concepts of Multi-objective Optimization
In real life situations, there may be multiple objectives need to be optimized simultaneously in order to solve certain
classification problems. Classification problems are multi-objective in nature as they required simultaneously
optimization of multiple objectives like accuracy, sensitivity, mean squared error etc. The main difficulty with multiobjective optimization is that there is no accepted predefined definition of optimum, so it is difficult to compare one
solution with another one. Multi-Objective Optimization Problem (MOOP), in general may be stated as, finding the
value for a set of n decision variables which must satisfy some constraints (J inequalities and K equalities) such that the
M objective functions are optimized and can be modelled as follows [3]:
Maximize/Minimize
fm(x),
m=1,2,3,4,......M;
Subject to
gj(x) 0,
j=1,2,3,4,........J;
hk(x) = 0,
k=1,2,3,4,.......K;
xiL xi xiU,
i=1,2,3,4,........n.
A solution x is vector of n decision variables: x = (x1, x2, x3, ......., xn)T. The solutions which satisfy the constraints and
variable bounds constitute a feasible decision variable space S. gj is the set of inequality constraints and h k is set of
equality constraints. Multi-objective optimization objective functions constitute multi-dimensional space, in addition to
the usual decision variable space. The additional space is called objective space Z. For each solution x in the decision
variable space, there exists a point in the objective space, denoted by f(x) = z = (z1, z2, . . . , zM)T. The mapping takes
place between n-dimensional solution vector and M-dimensional objective vector.
We introduce some definitions to describe the concept of optimality.
Definition 1 (Dominance) [3]: A solution x(1) is said to dominate the other solution x(2), x(1) x(2), if following both
conditions are satisfied:
1. if x(1) is no worse than x(2) in all objectives and
2. x(1) is strictly better than x(2) in at least one objective.
If any of the above conditions violated, the solution x(1) does not dominate solution x(2).
Definition 2 (Non-dominated set) [3]: Among the set of solutions P, the non-dominated solutions are those P which
are not dominated by any member of the set P.
Definition 3 (Globally Pareto-optimal set) [3]: The non-dominated set of the entire feasible search space S is the
Globally Pareto-optimal set.
Definition 4 (Strong Dominance) [3]: A solution x(1) strongly dominates solution x(2), if solution x(1) is strictly better
than solution x(2) in all M objectives.
Definition 5 (Weak Dominance) [3]: Among the set of solutions P, weakly non-dominated set of solutions P are those
that are not strongly dominated by any member of set P.
2.2 Multi-objective Evolutionary Algorithms
Traditional search and optimization methods such as gradient-based methods are difficult to extend to the multiobjective concept because their basic design excludes the consideration of multiple solutions. In contrast, evolutionary
algorithms are well-suited for handling such situations. There are different approaches for solving multi-objective
optimization problems [3], [4]. MOEAs have evolved over several years, starting from conventional weighted formula
approach to the elitist Pareto approach. In the weighted formula approaches, multiple objective functions are combined
into single scalar value using weights, and that single-objective function is then optimized using conventional
evolutionary algorithms. Schaffer in 1984 implemented first multi-objective genetic algorithm to find the set of nondominated solutions. In this population based nonelitist, non-pareto approach such as Vector Evaluated Genetic
Algorithm (VEGA) [5], a selection operator is used and numbers of subpopulations are generated equal to the number
of objectives to be optimized. Each individual of subpopulation assigned fitness with respect to the respective objective
function. In Pareto-based approach, selection should be made using a non-dominated ranking scheme and that diversity
should be maintained with the use of a sharing function. The most representative non-elitist Pareto-based MOEAs are
multiple objective genetic algorithm (MOGA) [6], niched Pareto Genetic algorithm (NPGA) [7], non-dominated
sorting genetic algorithm (NSGA) [8]. In MOGA [6], a ranking scheme is used to rank each individual corresponding
to the number of individuals in the current population by which it is dominated. Fitness sharing is used in order to
maintain diversity, with a mating restriction scheme to avoid crossover between very distant individuals in the search
space. In NSGA [8], population is sorted in various fronts. Non-dominated individuals belonging to the first front are
more fit, hence they are removed from the population and the process is repeated until the entire population is
classified. A tournament selection scheme based on Pareto dominance is used in NPGA [7]. These techniques do not
use elitism, and therefore, they cannot give guarantee of preserving non-dominated solutions during search. In recent
years, a number of elitist Pareto-based multi-objective evolutionary algorithms have been proposed. The most
representative elitist MOEAs include Strength Pareto evolutionary algorithm (SPEA) [9], SPEA2 [10], Pareto archived
evolution strategy (PAES) [11], Pareto Envelope-based selection algorithm (PESA) [12], and PESA II [13], and non-
Page 293
dominated sorting genetic algorithm-II (NSGA II) [14]. In SPEA [9], it introduces elitism by maintaining external
population. This population maintains non-dominated solutions that are found from beginning of the simulation. At
each generation newly found non-dominated solutions are compared with external population and resulting nondominated solutions are preserved. SPEA2 [10] incorporate a fine-grained fitness assignment strategy, a density
estimation technique and an enhanced archive truncation method. In PAES [11], an archive of non-dominated
solutions is considered for maintaining population diversity. A newly generated offspring is compared with the archive
to verify if it dominates any member of the archive. If yes, then the offspring enters the archive and is accepted as a
new parent. In NSGA II [14], uses crowding distance for density estimation for each individual. Crowded distance of a
solution is the average side-length of the cube enclosing the point without including any other point in the population.
Solutions of the last accepted front are ranked according to the crowded distance. Each solution is assigned fitness
equal to its Non-domination level. Binary tournament selection, recombination and mutation are used to create an
offspring population. Classification is the one of the important task in data mining. It is a supervised technique which
is used to predict class labels of unknown tuples. Different techniques which are used for classification are Decision
tree classifier, Rule based classifier, Neural network, Bayesian network and Support vector machine Most of the
applications of MOEAs for classification problems have used one of these Pareto-based elitist approaches as their
underlying optimization strategy.
Page 294
back or feed-forward network. Algorithms like back-propagation only train certain restricted topologies and types of
networks. On the other hand genetic algorithm can train any types of network also mixture of two types of networks.
Page 295
Multi-objective evolutionary approach applied for system identification with recurrent neural network by J.H.Ang,
C.K.Goh, E.J.Teoh and A.A.Mamun [20]. Authors incorporated few features such as variable-length chromosome
representation in the form of structural mutation and micro genetic algorithm for local search in multi-objective
evolutionary recurrent neural network. They considered simultaneous evolution of synaptic weights and neural network
architecture. Objectives to be considered for optimization are accuracy and network complexity [20] which are
conflicting with each other. Uniform mutation was used to evolve required set of connection weights. In this approach,
elitism was implemented as fixed size archive to prevent loss of good individuals due to the stochastic nature of
optimization process. This approach is effective for system identification, where hidden neurons were evolved using
structural mutation and variable chromosomes was used to model network architecture. Renata Furtuna, Silvia
Curteanu and Florin Leon proposed an elitist non-dominated sorting genetic algorithm enhanced with a neural network
applied to the multi-objective optimization of a polysiloxane synthesis process [21]. Objectives considered by authors
for optimization are, first is to maximize the reaction conversion and second is to minimize the difference between the
obtained viscometric molecular weight and the desired molecular weight [21]. A feed-forward neural network was used
with NSGA II [14]. NSGA II [14] approach was used for multi-objective optimization of polysiloxane synthesis
process. This NSGA II approach use elitist mechanism to preserve good individuals for next generation as there may be
chance of getting loss of good individuals due to optimization process. The real coding was used for chromosome
encoding as it is more suited for real life problems. Binary tournament selection was used for selection of parents for
new individuals reproduction. Ranking method was used for selection of new population for next generation.
Crowding distance was used to maintain diversity of solutions. This approach quickly gives optimal solutions as an
acceptable compromise between objectives competes with each other [21]. Manuel Cruz-Ramrez, Csar HervsMartnez, Juan Carlos Fernndez, Javier Briceno and Manuel de la Mata proposed multi-objective approach with
evolutionary artificial neural networks for predicting patient survival after liver transplantation [22]. Objectives which
are to be considered for optimization are Accuracy and minimum sensitivity. NSGA II [14], a multi-objective
evolutionary algorithm was used to train radial basis function neural networks. The optimal neural network models
obtained from Pareto fronts were used to give input for rule based system which was used to find perfect donorrecipient match. A major disadvantage of evolutionary approach is, it is computationally expensive and thus
evolutionary approach is slow. Hybrid technique such as by augmenting evolutionary algorithm with local search was
used to speed up slow convergence.
References
[1] U. Maulik, S. Bandyopadhyay, and A. Mukhopadhyay, Multiobjective Genetic Algorithms for Clustering
Applications in Data Mining and Bioinformatics. Berlin, Germany: Springer, 2011.
[2] J. Han and M. Kamber, Data Mining: Concepts and Techniques. San Francisco, CA, USA: Morgan Kaufmann,
2006.
[3] K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms. London, U.K.: Wiley, 2001.
[4] C. A. Coello Coello, G. B. Lamont, and D. A. van Veldhuizen, Evolutionary Algorithms for Solving MultiObjective Problems (Genetic and Evolutionary Computation), 2nd ed. Berlin/Heidelberg, Germany: Springer,
2007.
[5] J. D. Schaffer, Multiple objective optimization with vector evaluated genetic algorithms, in Proc. 1st Int. Conf.
Genet. Algorithms Their Appl., 1985, pp. 93100.
[6] C. M. Fonseca and P. J. Fleming, Genetic algorithms for Multiobjective optimization: Formulation, discussion
and generalization, in Proc. 5th Int. Conf. Genet. Algorithms, 1993, pp. 416423.
[7] J. Horn and N. Nafpliotis, Multiobjective optimization using the niched Pareto genetic algorithm, Univ. Illinois
at Urbana-Champaign, Urbana, IL, USA, Tech. Rep. IlliGAl Rep. 93005, 1993.
[8] N. Srinivas and K. Deb, Multiobjective optimization using nondominated sorting in genetic algorithms, Evol.
Comput., vol. 2, no. 3, pp. 221248, 1994.
[9] E. Zitzler and L. Thiele, Multiobjective evolutionary algorithms: A comparative case study and the strength
Pareto approach, IEEE Trans. Evol. Comput., vol. 3, no. 4, pp. 257271, Nov. 1999.
Page 296
[10] E. Zitzler, M. Laumanns, and L. Thiele, SPEA2: Improving the strength Pareto evolutionary algorithm, in Proc.
EUROGEN, 2001, pp. 95100.
[11] J. D. Knowles and D. W. Corne, The Pareto archived evolution strategy: A new baseline algorithm for Pareto
multiobjective optimisation, in Proc. IEEE Cong. Evol. Comput., 1999, pp. 98105.
[12] D. W. Corne, J. D. Knowles, and M. J. Oates, The Pareto envelope based selection algorithm for multiobjective
optimization, in Proc. Conf. PPSN-VI, 2000, pp. 839848.
[13] D. W. Corne, N. R. Jerram, J. D. Knowles, and M. J. Oates, PESA-II: Region-based selection in evolutionary
multiobjective optimization, in Proc. GECCO, 2001, pp. 283290.
[14] K. Deb, A. Pratap, S. Agrawal, and T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II,
IEEE Trans. Evol. Comput., vol. 6, no. 2, pp. 182197, Apr. 2002.
[15] Sujatha Srinivasan and Sivakumar Ramakrishnan, Evolutionary multi-objective optimization for rule mining: a
review, Springer, March 2011.
[16] Matthew Butler and Ali Daniyal, Multi-objective optimization with an evolutionary artificial neural network for
financial forecasting, ACM 978-1-60558-325-9/09/07, GECCO09, July 812, 2009.
[17] Abbaas HA, Sarker R and Newton C, A Pareto-differential evolution approach to vector optimization problems,
IEEE congress on evolutionary computation, vol 2, 2001.
[18] Hussein A. Abbass, An evolutionary artificial neural networks approach for breast cancer diagnosis, Elsevier
Science, 2002.
[19] S. Dehuri, S. Patnaik, A. Ghosh and R. Mall, Application of elitist multi-objective genetic algorithm for
classification rule generation , Elsevier, 2007.
[20] J.H.Ang, C.K.Goh, E.J.Teoh and A.A.Mamun, Multi-objective evolutionary recurrent neural networks for
System Identification, IEEE, 2007.
[21] Renata Furtuna, Silvia Curteanu and Florin Leon, An elitist non-dominated sorting genetic algorithm enhanced
with a neural network applied to the multi-objective optimization of a polysiloxane synthesis process, Elsevier,
2011.
[22] Manuel Cruz-Ramrez, Csar Hervs-Martnez, Juan Carlos Fernndez, Javier Briceno Manuel de la Mata,
Predicting patient survival after liver transplantation using evolutionary multi-objective artificial neural networks
Elsevier, 2013.
AUTHORS
Seema Mane pursuing M.E. degree in Computer Engineering from Maharashtra Institute of Technology, Pune in
Pune University and received B.E. degree in Computer Engineering from Amrutvahini College of Engineering,
Sangamner in Pune University in 2012and her research interest includes Data Mining.
Mrs. Shilpa Sonawani is working as Assistant Professor at, MAEERs Maharashtra Institute of Technology, Pune.
She received her degrees, B.E. (Comp. Sci. & Engg.) from College of Engineering Badnera-Amravati in 1996, and
M.Tech. (Comp. Sci. & Engg.) in 2008 from Guru Gobind Singh Indraprastha University, Delhi. She is a life
member of ISTE, New Delhi. Her area of interest is Data mining, Multiobjective evolutionary algorithms.
Dr. Sachin R. Sakhare is currently working as Professor at Vishwakarma Institute of Information Technology,
Pune. He did his B.E.( Comp. Sci. & Engg.) from College of Engineering Badnera-Amravati in 1996, M.E. ( Comp.
Sci. & Engg.) in 2006 and Ph.D. from S.G.B. Amravati University in 2014 in the faculty of Engineering &
Technology in the area of Operating Systems. He is a life member of ISTE, New Delhi, and Member of IAENG
Hong Kong.
Mrs. Pradnya Kulkarni is currently working as Assistant Professor at, MAEERs Maharashtra Institute of
Technology, Pune. She received her degrees, B.E. (Computer Engineering) from Dr. D. Y. Patil College of
Engineering, Kolhapur in Shivaji University in 2001 and M.E. (CSE and IT) from Vishwakarma Institute of
Technology, Pune in Savitribai Phule Pune University in 2008. Her area of interest is Data mining and Image
Processing.
Page 297