See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/3931641

Tuning of the structure and parameters of neural network using an improved

genetic algorithm

Conference Paper in IEEE Transactions on Neural Networks · February 2001

DOI: 10.1109/IECON.2001.976448 · Source: IEEE Xplore

CITATIONS READS
283 1,073

4 authors, including:

Hak-Keung Lam S. H. Ling

King's College London University of Technology Sydney
552 PUBLICATIONS 19,214 CITATIONS 164 PUBLICATIONS 4,296 CITATIONS

SEE PROFILE SEE PROFILE

F.H.F. Leung
The Hong Kong Polytechnic University
218 PUBLICATIONS 5,157 CITATIONS

SEE PROFILE

All content following this page was uploaded by S. H. Ling on 06 June 2014.

The user has requested enhancement of the downloaded file.

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 14, NO. 1, JANUARY 2003 79

Tuning of the Structure and Parameters of a Neural

Network Using an Improved Genetic Algorithm
Frank H. F. Leung, Member, IEEE, H. K. Lam, S. H. Ling, and Peter K. S. Tam

Abstract—This paper presents the tuning of the structure and [7], system modeling, and control [16]. Owing to its particular
parameters of a neural network using an improved genetic algo- structure, a neural network is very good in learning [2] using
rithm (GA). It will also be shown that the improved GA performs some learning algorithms such as GA [1] and backpropagation
better than the standard GA based on some benchmark test func-
tions. A neural network with switches introduced to its links is pro-
[2]. In general, the learning steps of a neural network are as fol-
posed. By doing this, the proposed neural network can learn both lows. First, a network structure is defined with a fixed number
the input–output relationships of an application and the network of inputs, hidden nodes and outputs. Second, an algorithm is
structure using the improved GA. The number of hidden nodes is chosen to realize the learning process. However, a fixed struc-
chosen manually by increasing it from a small number until the ture may not provide the optimal performance within a given
learning performance in terms of fitness value is good enough. Ap- training period. A small network may not provide good per-
plication examples on sunspot forecasting and associative memory
are given to show the merits of the improved GA and the proposed formance owing to its limited information processing power. A
neural network. large network, on the other hand, may have some of its con-
nections redundant [18], [19]. Moreover, the implementation
Index Terms—Genetic algorithm (GA), neural networks, param-
eter learning, structure learning. cost for a large network is high. To obtain the network struc-
ture automatically, constructive and destructive algorithms can
be used [18]. The constructive algorithm starts with a small net-
I. INTRODUCTION work. Hidden layers, nodes, and connections are added to ex-
pand the network dynamically [19]–[24]. The destructive algo-
G ENETIC algorithm (GA) is a directed random search
technique [1] that is widely applied in optimization
problems [1], [2], [5]. This is especially useful for complex
rithm starts with a large network. Hidden layers, nodes, and con-
nections are then deleted to contract the network dynamically
optimization problems where the number of parameters is large [25], [26]. The design of a network structure can be formulated
and the analytical solutions are difficult to obtain. GA can help into a search problem. GAs [27], [28] were employed to ob-
to find out the optimal solution globally over a domain [1], [2], tain the solution. Pattern-classification approaches [29] can also
[5]. It has been applied in different areas such as fuzzy control be found to design the network structure. Some other methods
[9]–[11], [15], path planning [12], greenhouse climate control were proposed to learn both the network structure and connec-
[13], modeling and classification [14] etc. tion weights. An ANNA ELEONORA algorithm was proposed
A lot of research efforts have been spent to improve the per- [36]. New genetic operator and encoding procedures that allows
formance of GA. Different selection schemes and genetic op- an opportune length of the coding string were introduced. Each
erators have been proposed. Selection schemes such as rank- gene consists of two parts: the connectivity bits and the connec-
based selection, elitist strategies, steady-state election and tour- tion weight bits. The former indicates the absence or presence
nament selection have been reported [32]. There are two kinds of of a link, and the latter indicates the value of the weight of a link.
genetic operators, namely crossover and mutation. Apart from A GNARL algorithm was also proposed in [37]. The number of
random mutation and crossover, other crossover and mutation hidden nodes and connection links for each network is first ran-
mechanisms have been proposed. For crossover mechanisms, domly chosen within some defined ranges. Three steps are then
two-point crossover, multipoint crossover, arithmetic crossover, used to generate an offspring: copying the parents, determining
and heuristic crossover have been reported [1], [31]–[33]. For the mutations to be performed, and mutating the copy. The mu-
mutation mechanisms, boundary mutation, uniform mutation, tation of a copy is separated into two classes: the parametric mu-
and nonuniform mutation can be found [1], [31]–[33]. tations that alter the connection weights, and the structural mu-
Neural network was proved to be a universal approximator tations that alter the number of hidden nodes and the presence of
[16]. A three-layer feedforward neural network can approxi- network links. An evolutionary system named EPNet can also
mate any nonlinear continuous function to an arbitrary accuracy. be found for evolving neural networks [19]. Rank-based selec-
Neural networks are widely applied in areas such as prediction tion and five mutations were employed to modify the network
structure and connection weights.
In this paper, a three-layer neural network with switches in-
Manuscript received September 5, 2001; revised February 6, 2002 and July 2,
2002. This work was supported by the Centre for Multimedia Signal Processing,
troduced in some links is proposed to facilitate the tuning of
Department of Electronic and Information Engineering, The Hong Kong Poly- the network structure in a simple manner. A given fully con-
technic University, under Project A432. nected feedforward neural network may become a partially con-
The authors are with the Centre for Multimedia Signal Processing, Depart- nected network after learning. This implies that the cost of im-
ment of Electronic and Information Engineering, The Hong Kong Polytechnic
University, Kowloon, Hong Kong. plementing the proposed neural network, in terms of hardware
Digital Object Identifier 10.1109/TNN.2002.804317 and processing time, can be reduced. The network structure and
1045-9227/03$17.00 © 2003 IEEE

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
80 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 14, NO. 1, JANUARY 2003

Fig. 1. Procedure of standard GA.

parameters will be tuned simultaneously using a proposed im-

proved GA. As application examples, the proposed neural net-
work with link switches tuned by the improved GA is used to
(2)
estimate the number of sunspots [7], [8] and realize an associa-
tive memory. The results will be compared with those obtained (3)
by traditional feedforward networks [2] trained by 1) the stan-
dard GA with arithmetic crossover and nonuniform mutation where denotes the population size;
[1], [2], [5] and 2) the backpropagation with momentum and denotes the number of variables to be tuned; ,
adaptive learning rate [30]. ; , are the
This paper is organized as follows. In Section II, the improved parameters to be tuned; and are the min-
genetic algorithm is presented. In Section III, it will be shown imum and maximum values of the parameter , respectively
that the improved GA performs more efficiently than the stan- for all . It can be seen from (1)–(3) that the potential solution
dard GA [1], [2], [5] based on some benchmark test functions set contains some candidate solutions (chromosomes).
[3], [4], [6], [17]. In Section IV, the neural network with link The chromosome contains some variables (genes).
switches, and the tuning of its structure and parameters using
the improved GA will be presented. Application examples will B. Evaluation
be presented in Section V. A conclusion will be drawn in Sec-
Each chromosome in the population will be evaluated by a
tion VI.
defined fitness function. The better chromosomes will return
higher values in this process. The fitness function to evaluate
II. IMPROVED GA
a chromosome in the population can be written as
The standard GA process [1], [2], [5] is shown in Fig. 1. First,
a population of chromosomes is created. Second, the chromo-
somes are evaluated by a defined fitness function. Third, some fitness (4)
of the chromosomes are selected for performing genetic oper-
ations. Forth, genetic operations of crossover and mutation are The form of the fitness function depends on the application.
performed. The produced offspring replace their parents in the
initial population. In this reproduction process, only the selected C. Selection
parents in the third step will be replaced by their corresponding Two chromosomes in the population will be selected to un-
offspring. This GA process repeats until a user-defined criterion dergo genetic operations for reproduction by the method of spin-
is reached. In this paper, the standard GA is modified and new ning the roulette wheel [1]. It is believed that high potential par-
genetic operators are introduced to improve its performance. ents will produce better offspring (survival of the best ones).
The improved GA process is shown in Fig. 2. Its details will The chromosome having a higher fitness value should therefore
be given as follows. have a higher chance to be selected. The selection can be done
by assigning a probability to the chromosome
A. Initial Population
The initial population is a potential solution set . The first
set of population is usually generated randomly (5)

(1)

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
LEUNG et al.: TUNING OF THE STRUCTURE AND PARAMETERS OF A NEURAL NETWORK 81

Fig. 2. Procedure of the improved GA.

The cumulative probability for the chromosome is defined will produce one offspring. First, four chromosomes will be
as generated according to the following mechanisms:

(7)
(6)

(8)
The selection process starts by randomly generating a nonzero
floating-point number, . Then, the chromosome
is chosen if ( ). It can be observed from
(9)
this selection process that a chromosome having a larger
will have a higher chance to be selected. Consequently, the best
chromosomes will get more offspring, the average will stay and
the worst will die off. In the selection process, only two chro- (10)
mosomes will be selected to undergo the genetic operations.
(11)
(12)
D. Genetic Operations
where denotes the weight to be determined by users,
The genetic operations are to generate some new chromo- denotes the vector with each element obtained by
somes (offspring) from their parents after the selection process. taking the maximum among the corresponding element of
They include the crossover and the mutation operations. and . For instance, .
1) Crossover: The crossover operation is mainly for ex- Similarly, gives a vector by taking the minimum
changing information from the two parents, chromosomes value. For instance, .
and , obtained in the selection process. The two parents Among to , the one with the largest fitness value is used

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
82 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 14, NO. 1, JANUARY 2003

as the offspring of the crossover operation. The offspring is de- to change the value of a randomly selected gene to its upper
fined as or lower bound. Uniform mutation is to change the value of a
randomly selected gene to a value between its upper and lower
bounds. Nonuniform mutation is capable of fine-tuning the pa-
(13) rameters by increasing or decreasing the value of a randomly
selected gene by a weighted random number. The weight is usu-
denotes the index which gives a maximum value of , ally a monotonic decreasing function of the number of iteration.
. In our approach, we have three offspring generated in the muta-
If the crossover operation can provide a good offspring, tion process. From (14), the first mutation is in fact the uniform
a higher fitness value can be reached in less iteration. In mutation. The second mutation allows some randomly selected
general, two-point crossover, multipoint crossover, arithmetic genes to change simultaneously. The third mutation changes all
crossover or heuristic crossover can be employed to realize the genes simultaneously. The second and the third mutations allow
crossover operation [1], [31]–[33]. The offspring generated multiple genes to be changed. Hence, the searching domain is
by these methods, however, may not be better than that from larger than that formed by changing a single gene. The genes
our approach. As seen from (7)–(10), the potential offspring will have a larger space for improving when the fitness values
spreads over the domain. While (7) and (10) result in searching are small. On the contrary, when the fitness values are nearly
around the center region of the domain [a value of near to the same, changing the value of a single gene (the first muta-
one in (10) can move to be near ], (8) and (9) tion) will give a higher probability of improving the fitness value
move the potential offspring to be near the domain boundary [a as the searching domain is smaller and some genes may have
large value of in (8) and (9) can move and to be near reached their optimal values.
and respectively]. After the operation of selection, crossover, and mutation, a
2) Mutation: The offspring (13) will then undergo the mu- new population is generated. This new population will repeat
tation operation. The mutation operation is to change the genes the same process. Such an iterative process can be terminated
of the chromosomes. Consequently, the features of the chromo- when the result reaches a defined condition, e.g., the change of
somes inherited from their parents can be changed. Three new the fitness values between the current and the previous iteration
offspring will be generated by the mutation operation is less than 0.001, or a defined number of iteration has been
reached.

III. BENCHMARK TEST FUNCTIONS

Some benchmark test functions [3], [4], [6], [17] are used to
(14) examine the applicability and efficiency of the improved GA.
Six test functions , will be used, where
where , can only take the value of 0 . is an integer denoting the dimen-
or 1, , , are randomly generated sion of the vector . The six test functions are defined as follows:
numbers such that . The
first new offspring ( ) is obtained according to (14) with (15)
that only one ( being randomly generated within the range)
is allowed to be one and all the others are zeros. The second
new offspring is obtained according to (14) with that some where and the minimum point is at
randomly chosen are set to be one and others are zero. The third
new offspring is obtained according to (14) with all .
These three new offspring will then be evaluated using the fit-
ness function of (4). A real number will be generated randomly
and compared with a user-defined number . If the (16)
real number is smaller than , the one with the largest fitness
value among the three new offspring will replace the chromo- where and the minimum point is at
some with the smallest fitness in the population. If the real
number is larger than , the first offspring will replace
the chromosome with the smallest fitness value in the popu- (17)
lation if ; the second and the third offspring will
do the same. is effectively the probability of accepting a bad
where and the minimum point is at
offspring in order to reduce the chance of converging to a local
. The floor function, ,
optimum. Hence, the possibility of reaching the global optimum
is to round down the argument to an integer
is kept.
In general, various methods like boundary mutation, uniform
mutation, or nonuniform mutation [1], [32], [33] can be em- (18)
ployed to realize the mutation operation. Boundary mutation is

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
LEUNG et al.: TUNING OF THE STRUCTURE AND PARAMETERS OF A NEURAL NETWORK 83

where and the minimum point is at . times of simulations based on the proposed and standard GAs
is a function to generate uniformly a floating-point are shown in Fig. 3 and tabulated in Table I. Generally, it can
number between zero and one inclusively be seen that the performance of the proposed GA is better than
that of the standard GA.

IV. NEURAL NETWORK WITH LINK SWITCHES AND TUNING

USING THE IMPROVED GA
(19) In this section, a neural network with link switches is pre-
sented. By introducing a switch to a link, the parameters and
where we have as shown in the equation at the bottom of the structure of the neural network can be tuned using the im-
the next page, and and the maximum point is at proved GA.

A. Neural Network With Link Switches

Neural networks [5] for tuning usually have a fixed structure.
The number of connections may be too large for a given ap-
(20) plication such that the network structure is unnecessarily com-
where and the minimum point is at . It plex and the implementation cost is high. In this section, a mul-
should be noted that the minimum values of all functions in the tiple-input–multiple-output (MIMO) three-layer neural network
defined domain are zero except for . The fitness functions is proposed as shown in Fig. 4. The main different point is that
for to and are defined as a unit step function is introduced to each link. Such a unit step
function is defined as
(21)
if
(23)
if
and the fitness function for is defined as
This is equivalent to adding a switch to each link of the neural
(22) network. Referring to Fig. 4, the input–output relationship of the
proposed MIMO three-layer neural network is as follows:
and we have the equation shown at the bottom of the page.
The proposed GA goes through these six test functions.
The results are compared with those obtained by the standard
GA with arithmetic crossover and nonuniform mutation [1],
[31]–[33]. For each test function, the simulation takes 500 (24)
iterations and the population size is ten for the proposed and
the standard GAs. The probability of crossover is set at 0.8 for , , are the inputs which are functions
all functions and the probability of mutation for functions to of a variable ; denotes the number of inputs; denotes
are 0.8, 0.8, 0.7, 0.8, 0.8, and 0.35, respectively. The shape the number of the hidden nodes; , ;
parameters of the standard GA [2] for nonuniform mutation, , denotes the weight of the link between the th
which is selected by trial and error through experiments for hidden node and the th output; denotes the weight of the
good performance, are set at for , and , link between the th input and the th hidden node; denotes
for , for and . For the proposed GA, the values the parameter of the link switch from the th input to the th
of are set to be 0.5, 0.99, 0.1, 0.5, 0.01, and 0.01 for the hidden node; denotes the parameter of the link switch from
six test functions, respectively. The probability of acceptance the th hidden node to the th output; denotes the number
is set at 0.1 for all functions. These values are selected by of outputs of the proposed neural network; and denote the
trial and error through experiments for good performance. The biases for the hidden nodes and output nodes, respectively;
initial values of in the population for a test function are set and denote the parameters of the link switches of the biases
to be the same for both the proposed and the standard GAs. to the hidden and output layers, respectively; denotes
For tests 1 to 6, the initial values are , , the logarithmic sigmoid function:
, , and ,
respectively. The results of the average fitness values over 100 (25)

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
84 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 14, NO. 1, JANUARY 2003

(a) (b)

(c) (d)

(e) (f)
Fig. 3. Simulation results of the improved and standard GAs. The averaged fitness value of test functions obtained by the improved (solid line) and standard
(dotted line) GAs. (a) f (x). (b) f (x). (c) f (x). (d) f (x). (e) f (x). (f) f (x).

, , is the th output of the proposed the weights of the links govern the input–output relationship of
neural network. By introducing the switches, the weights the neural network while the switches of the links govern the
and , and the switch states can be tuned. It can be seen that structure of the neural network.

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
LEUNG et al.: TUNING OF THE STRUCTURE AND PARAMETERS OF A NEURAL NETWORK 85

TABLE I
SIMULATION RESULTS OF THE PROPOSED GA AND THE STANDARD GA
BASED ON THE BENCHMARK TEST FUNCTIONS

Fig. 5. Sunspot cycles from year 1700 to 1980.

V. APPLICATION EXAMPLES
Two application examples will be given in this section to il-
lustrate the merits of the proposed neural networks tuned by the
improved GA.

A. Forecasting of the Sunspot Number

An application example on forecasting the sunspot number
[7], [8], [27] will be given in this section. The sunspot cycles
from 1700 to 1980 are shown in Fig. 5. The cycles generated
are nonlinear, nonstationary, and non-Gaussian which are dif-
ficult to model and predict. We use the proposed three-layer
neural network (three-input-single-output) with link switches
Fig. 4. Proposed three-layer neural network.
for the sunspot number forecasting. The inputs, , of the pur-
posed neural network are defined as ,
B. Tuning of the Parameters and Structure , and , where denotes the year and
The proposed neural network can be employed to learn the is the sunspot numbers at the year . The sunspot numbers
input–output relationship of an application by using the im- of the first 180 years (i.e., ) are used to train
proved GA. The input–output relationship is described by the proposed neural network. Referring to (24), the proposed
neural network used for the sunspot forecasting is governed by
(26)

where and
are the given inputs and the desired
outputs of an unknown nonlinear function respectively, (29)
denotes the number of input–output data pairs. The fitness func-
tion is defined as The number of hidden nodes is changed from three to seven
to test the learning performance. The fitness function is defined
fitness (27) as follows:
err
fitness (30)
err
err (28)
err (31)

The objective is to maximize the fitness value of (27) The improved GA is employed to tune the parameters and
using the improved GA by setting the chromosome to be structure of the neural network of (29). The objective is to
for all , , . It can be seen maximize the fitness function of (30). The best fitness value is
from (27) and (28) that a larger fitness value implies a smaller one and the worst one is zero. The population size used for the
error value. improved GA is ten, and for all values of

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
86 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 14, NO. 1, JANUARY 2003

TABLE II TABLE III

SIMULATION RESULTS FOR THE APPLICATION EXAMPLE OF FORECASTING THE TRAINING ERROR AND FORECASTING ERROR IN MEAN ABSOLUTE
SUNSPOT NUMBER AFTER 1000 ITERATIONS OF LEARNING ERROR (MAE) FOR THE APPLICATION EXAMPLE OF FORECASTING
THE SUNSPOT NUMBER

. The lower and the upper bounds of the link weights are de-
fined as
and, , ;
[16]. The chromosomes used
for the improved GA are .
The initial values of all the link weights between the input and
hidden layers are one, and those between the hidden and output
layers are 1. The initial value of in (23) is 0.5.
For comparison purpose, a fully connected three-layer
feedforward neural network (three-input–one-output) [2] is
also trained by the standard GA with arithmetic crossover and
nonuniform mutation [1], [2], [5], and backpropagation with
momentum and adaptive learning rate [30]. Also, the proposed
neural network trained by the standard GA will be considered.
For the standard GA, the population size is ten, the probability
of crossover is 0.8 and the probability of mutation is 0.1.
The shape parameters of the standard GA with arithmetic
crossover and nonuniform mutation, which is selected by trial
and error through experiment for good performance, is set to Fig. 6. Simulation results of a 96-year prediction using the proposed neural
be one. For the backpropagation with momentum and adaptive network (n = 6) with the proposed GA (dashed line), and the actual sunspot
numbers (solid line) for the years 1885–1980.
learning rate, the learning rate is 0.2, the ratio to increase the
learning rate is 1.05, the ratio to decrease the learning rate is
connected link is 18 after learning (the number of links of a
0.7, the maximum validation failures is five, the maximum
fully connected network is 31 which includes the bias links).
performance increase is 1.04, the momentum constant is 0.9.
It is about 42% reduction of links. The training error and the
The initial values of the link weights are the same as those of
forecasting error are 11.5730 and 14.0933, respectively.
the proposed neural network. For all approaches, the learning
processes are carried out by a personal computer with a P4 1.4 B. Associative Memory
GHz CPU. The number of iterations for all approaches is 1000.
The tuned neural networks are used to forecast the sunspot Another application example on tuning an associative
number during the years 1885–1980. The number of hidden memory will be given in this section. In this example, the
nodes is changed from four to eight. The simulation results associative memory, which maps its input vector into itself, has
for the comparisons are tabulated in Tables II and III. From ten inputs and ten outputs. Thus, the desired output vector is its
Table II, it is observed that the proposed neural network trained input vector. Referring to (24), the proposed neural network is
with the improved GA provides better results in terms of given by
accuracy (fitness values) and number of links. The training
error [governed by (31)] and the forecasting error [governed
by ] are tabulated in Table III.
Referring to Table III, the best result is obtained when the
number of hidden node is six. Fig. 6 shows the simulation
results of the forecasting using the proposed neural network
trained with the improved GA (dashed lines) and the actual
sunspot numbers (solid lines) for . The number of (32)

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
LEUNG et al.: TUNING OF THE STRUCTURE AND PARAMETERS OF A NEURAL NETWORK 87

TABLE IV VI. CONCLUSION

SIMULATION RESULTS FOR THE APPLICATION EXAMPLE OF ASSOCIATIVE
MEMORY AFTER 500 ITERATIONS OF LEARNING An improved GA has been proposed in this paper. By using
the benchmark test functions, it has been shown that the im-
proved GA performs more efficiently than the standard GA. Be-
sides, by introducing a switch to each link, a neural network
that facilitates the tuning of its structure has been proposed.
Using the improved GA, the proposed neural network is able
to learn both the input–output relationship of an application and
the network structure. As a result, a given fully connected neural
network can be reduced to a partially connected network after
learning. This implies that the cost of implementation of the
neural network can be reduced. Application examples on fore-
casting the sunspot number and tuning an associative memory
using the proposed neural network trained with the improved
GA have been given. The simulation results have been com-
pared with those obtained by the proposed network trained by
50 input vectors (each input vector has the property that
the standard GA, and traditional feedforward networks trained
to test the learning performance. The fitness function is de-
by the standard GA (with arithmetic crossover and nonuniform
fined as follows:
mutation) and the backpropagation (with momentum and adap-
fitness (33) tive learning rate).
err
REFERENCES
[1] J. H. Holland, Adaptation in Natural and Artificial Systems. Ann
err (34) Arbor, MI: Univ. Michigan Press, 1975.
[2] D. T. Pham and D. Karaboga, Intelligent Optimization Techniques, Ge-
The improved GA is employed to tune the parameters and netic Algorithms, Tabu Search, Simulated Annealing and Neural Net-
works. New York: Springer-Verlag, 2000.
structure of the neural network of (32). The objective is to [3] Y. Hanaki, T. Hashiyama, and S. Okuma, “Accelerated evolutionary
maximize the fitness function of (33). The best fitness value computation using fitness estimation,” in Proc. IEEE Int. Conf. Syst.,
is one and the worst one is zero. The population size used for Man, Cybern., vol. 1, 1999, pp. 643–648.
[4] K. A. De Jong, “An analysis of the behavior of a class of genetic adaptive
the improved GA is ten; and for all values systems,” Ph.D. dissertation, Univ. Michigan, Ann Arbor, MI, 1975.
of . The lower and the upper bounds of the link weights are + =
[5] Z. Michalewicz, Genetic Algorithm Data Structures Evolution Pro-
defined as and grams, 2nd extended ed. New York: Springer-Verlag, 1994.
[6] G. X. Yao and Y. Liu, “Evolutionary programming made faster,” IEEE
, ; , Trans. Evol. Comput., vol. 3, pp. 82–102, July 1999.
[16]. The chromosomes used for the improved GA [7] M. Li, K. Mechrotra, C. Mohan, and S. Ranka, “Sunspot numbers
are . The initial values forecasting using neural network,” in Proc. 5th IEEE Int. Symp. Intell.
Contr., 1990, pp. 524–528.
of the link weights are all zero. For comparison purpose, [8] T. J. Cholewo and J. M. Zurada, “Sequential network construction for
a fully connected three-layer feedforward neural network time series prediction,” in Proc. Int. Conf. Neural Networks, vol. 4, 1997,
(ten-input–ten-output) [2] trained by the standard GA (with pp. 2034–2038.
[9] B. D. Liu, C. Y. Chen, and J. Y. Tsao, “Design of adaptive fuzzy logic
arithmetic crossover and nonuniform mutation) and another controller based on linguistic-hedge concepts and genetic algorithms,”
trained by backpropagation (with momentum and adaptive IEEE Trans. Syst., Man, Cybern. B, vol. 31, pp. 32–53, Feb. 2001.
learning rate) are considered again. Also, the proposed neural [10] Y. S. Zhou and L. Y. Lai, “Optimal design for fuzzy controllers by ge-
netic algorithms,” IEEE Trans. Ind. Applicat., vol. 36, pp. 93–97, Jan.
network trained by the standard GA will be considered. For 2000.
the standard GA, the population size is ten, the probability [11] C. F. Juang, J. Y. Lin, and C. T. Lin, “Genetic reinforcement learning
of crossover is 0.8 and the probability of mutation is 0.03. through symbiotic evolution for fuzzy controller design,” IEEE Trans.
Syst., Man. Cybern. B, vol. 30, pp. 290–302, Apr. 2000.
The shape parameters of the standard GA with arithmetic [12] H. Juidette and H. Youlal, “Fuzzy dynamic path planning using genetic
crossover and nonuniform mutation, which is selected by trial algorithms,” Electron. Lett., vol. 36, no. 4, pp. 374–376, Feb. 2000.
and error through experiments for good performance, is set to [13] R. Caponetto, L. Fortuna, G. Nunnari, L. Occhipinti, and M. G. Xibilia,
“Soft computing for greenhouse climate control,” IEEE Trans. Fuzzy
be three. For the backpropagation with momentum and adaptive Syst., vol. 8, pp. 753–760, Dec. 2000.
learning rate, the learning rate is 0.2, the ratio to increase the [14] M. Setnes and H. Roubos, “GA-fuzzy modeling and classification: Com-
learning rate is 1.05, the ratio to decrease the learning rate is plexity and performance,” IEEE Trans. Fuzzy Syst., vol. 8, pp. 509–522,
Oct. 2000.
0.7, the maximum validation failures is five, the maximum [15] K. Belarbi and F. Titel, “Genetic algorithm for the design of a class of
performance increase is 1.04, the momentum constant is 0.9. fuzzy controllers: An alternative approach,” IEEE Trans. Fuzzy Systems,
The initial values of the links weights are the same as those vol. 8, pp. 398–405, Aug. 2000.
[16] M. Brown and C. Harris, Neural Fuzzy Adaptive Modeling and Con-
of the proposed approach. The number of iterations for all trol. Englewood Cliffs, NJ: Prentice-Hall, 1994.
approaches is 500. The simulation results are tabulated in [17] S. Amin and J. L. Fernandez-Villacanas, “Dynamic local search,” in
Table IV. It can be seen from Table IV that the fitness values Proc. 2nd Int. Conf. Genetic Algorithms in Engineering Systems: Inno-
vations and Applications, 1997, pp. 129–132.
for different approaches are similar, but our approach can offer [18] X. Yao, “Evolving artificial networks,” Proc. IEEE, vol. 87, pp.
a smaller network. 1423–1447, July 1999.

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.
88 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 14, NO. 1, JANUARY 2003

[19] X. Yao and Y. Liu, “A new evolutionary system for evolving artificial Frank H. F. Leung (M’92) was born in Hong Kong in 1964. He received the
neural networks,” IEEE Trans. Neural Networks, vol. 8, pp. 694–713, B.Eng. and Ph.D. degrees in electronic engineering from the Hong Kong Poly-
May 1997. technic University in 1988 and 1992, respectively.
[20] F. J. Lin, C. H. Lin, and P. H. Shen, “Self-constructing fuzzy neural net- He joined the Hong Kong Polytechnic University in 1992 and is now an As-
work speed controller for permanent-magnet synchronous motor drive,” sociate Professor in the Department of Electronic and Information Engineering.
IEEE Trans. Fuzzy Syst., vol. 9, pp. 751–759, Oct. 2001. He has published more than 100 research papers on computational intelligence,
[21] Y. Hirose, K. Yamashita, and S. Hijiya, “Back-propagation algorithm control and power electronics. At present, he is actively involved in the research
which varies the number of hidden units,” Neural Networks, vol. 4, no. on Intelligent Multimedia Home and electronic Book.
1, pp. 61–66, 1991. Dr. Leung is a Reviewer for many international journals and had helped the
[22] A. Roy, L. S. Kim, and S. Mukhopadhyay, “A polynomial time algorithm organization of many international conferences. He is a Chartered Engineer and
for the construction and training of a class of multiplayer perceptions,” a Member of the Institution of Electrical Engineers (IEE).
Neural Networks, vol. 6, no. 4, pp. 535–545, 1993.
[23] N. K. Treadold and T. D. Gedeon, “Exploring constructive cascade net-
works,” IEEE Trans. Neural Networks, vol. 10, pp. 1335–1350, Nov.
1999.
[24] C. C. Teng and B. W. Wah, “Automated learning for reducing the config-
uration of a feedforward neural network,” IEEE Trans. Neural Networks,
vol. 7, pp. 1072–1085, Sept. 1996.
[25] Y. Q. Chen, D. W. Thomás, and M. S. Nixon, “Generating-shrinking H. K. Lam received the B.Eng. (Hons.) and Ph.D. degrees form the Department
algorithm for learning arbitrary classification,” Neural Networks, vol. 7, of Electronic and Information Engineering, The Hong Kong Polytechnic Uni-
no. 9, pp. 1477–1489, 1994. versity, Hong Kong, in 1995 and 2000, respectively.
[26] M. C. Moze and P. Smolensky, “Using relevance to reduce network size He is currently a Research Fellow in the Department of Electronic and In-
automatically,” Connect. Sci., vol. 1, no. 1, pp. 3–16, 1989. formation Engineering at The Hong Kong Polytechnic University. His current
[27] H. K. Lam, S. H. Ling, F. H. F. Leung, and P. K. S. Tam, “Tuning of the research interests include intelligent control and systems, computational intel-
structure and parameters of neural network using an improved genetic ligence, and robust control.
algorithm,” in Proc. 27th Annu. Conf. IEEE Ind. Electron. Soc., Denver,
CO, Nov. 2001, pp. 25–30.
[28] G. P. Miller, P. M. Todd, and S. U. Hegde, “Designing neural networks
using genetic algorithms,” in Proc. 3rd Int. Conf. Genetic Algorithms
Applications, 1989, pp. 379–384.
[29] N. Weymaere and J. Martens, “On the initialization and optimization
of multiplayer perceptrons,” IEEE Trans. Neural Networks, vol. 5, pp.
738–751, Sept. 1994.
[30] S. S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd S. H. Ling received the B.Eng. (Hons.) degree from the Department of Electrical
ed. Upper Saddle River, NJ: Prentice-Hall, 1999. Engineering, The Hong Kong Polytechnic University, Hong Kong, in 1999. He
[31] X. Wang and M. Elbuluk, “Neural network control of induction ma- is currently a Research Student in the Department of Electronic and Informa-
chines using genetic algorithm training,” in Conf. Record 31st IAS An- tion Engineering, The Hong Kong Polytechnic University. His research interests
nual Meeting, vol. 3, 1996, pp. 1733–1740. include evolutionary computation, fuzzy logic, neural networks, and intelligent
[32] L. Davis, Handbook of Genetic Algorithms. New York: Van Nostrand homes.
Reinhold, 1991.
[33] M. Srinivas and L. M. Patnaik, “Genetic algorithms: A survey,” IEEE
Computer, vol. 27, pp. 17–27, June 1994.
[34] J. D. Schaffer, D. Whitley, and L. J. Eshelman, “Combinations of ge-
netic algorithms and neural networks: A survey of the state of the art,”
in Proc. Int. Workshop Combinations Genetic Algorithms Neural Net-
works, 1992, pp. 1–37.
[35] S. Bornholdt and D. Graudenz, “General asymmetric neural networks Peter K. S. Tam received the B.E., M.E., and Ph.D. degrees from the University
and structure design by genetic algorithms: A learning rule for tem- of Newcastle, Newcastle, Australia, in 1971, 1973, and 1976, respectively, all
poral patterns,” in Proc. Int. Conf. Syst., Man, Cybern., vol. 2, 1993, in electrical engineering.
pp. 595–600. From 1967 to 1980, he held a number of industrial and academic positions
[36] V. Maniezzo, “Genetic evolution of the topology and weight distribution in Australia. In 1980, he joined The Hong Kong Polytechnic University, Hong
of neural networks,” IEEE Trans. Neural Networks, vol. 5, pp. 39–53, Kong, as a Senior Lecturer. He is currently an Associate Professor in the De-
Jan. 1994. partment of Electronic and Information Engineering. He has participated in the
[37] P. J. Angeline, G. M. Saunders, and J. B. Pollack, “An evolutionary al- organization of a number of symposiums and conferences. His research inter-
gorithm that constructs recurrent neural networks,” IEEE Trans. Neural ests include signal processing, automatic control, fuzzy systems, and neural net-
Networks, vol. 5, pp. 54–65, Jan. 1994. works.

Authorized
View publication stats licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 7, 2009 at 02:05 from IEEE Xplore. Restrictions apply.