Introduction to

Mathematical
Optimization
From Linear Programming
to
Metaheuristics

Xin-She Yang

Cambridge International Science Publishing

 Introduction to
Mathematical Optimization
– From Linear Programming to Metaheuristics
 Introduction to
Mathematical Optimization
– From Linear Programming to Metaheuristics

Xin-She Yang

University of Cambridge, United Kingdom

Cambridge International Science Publishing

Published by
Cambridge International Science Publishing
7 Meadow Walk, Great Abington, Cambridge CB21 6AZ, UK
http://www.cisp-publishing.com

First Published 2008

c Cambridge International Science Publishing

2008 c Xin-She Yang 2008

Conditions of Sale

All rights reserved. No part of this publication may be repro-

duced or transmitted in any form or by any means, electronic
or mechanical, including photocopy, recording, or any
informa-tion storage and retrieval system, without permission
in writing from the copyright holder.

British Library Cataloguing in Publication Data

A catalogue record for this book is available from the British Library

ISBN 978-1-904602-82-8

Cover design by Terry Callanan

Printed and bound in the UK by Lightning Source UK Ltd

Contents

Preface v

I Fundamentals 1
1 Mathematical Optimization 3

1.1 Optimization . . . . . . . . . . . . . . . . . . . . 3
1.2 Optimality Criteria . . . . . . . . . . . . . . . . . 6
1.3 Computational Complexity . . . . . . . . . . . . 7
1.4 NP-Complete Problems . . . . . . . . . . . . . . 9
2 Norms and Hessian Matrices 11

2.1 Vector and Matrix Norms . . . . . . . . . . . . . 11

2.2 Eigenvalues and Eigenvectors . . . . . . . . . . . 14
2.3 Spectral Radius of Matrices . . . . . . . . . . . . 18
2.4 Hessian Matrix . . . . . . . . . . . . . . . . . . . 20
2.5 Convexity . . . . . . . . . . . . . . . . . . . . . . 22
3 Root-Finding Algorithms 25

3.1 Simple Iterations . . . . . . . . . . . . . . . . . . 25

3.2 Bisection Method . . . . . . . . . . . . . . . . . . 27
3.3 Newton’s Method . . . . . . . . . . . . . . . . . . 29
3.4 Iteration Methods . . . . . . . . . . . . . . . . . 30
4 System of Linear Equations 35

4.1 Linear systems . . . . . . . . . . . . . . . . . . . 35

4.2 Gauss Elimination . . . . . . . . . . . . . . . . . 38

i
 ii CONTENTS
4.3 Gauss-Jordan Elimination . . . . . . . . . . . . . 41

4.4 LU Factorization . . . . . . . . . . . . . . . . . . 43
4.5 Iteration Methods . . . . . . . . . . . . . . . . . 45
4.5.1 Jacobi Iteration Method . . . . . . . . . . 45
4.5.2 Gauss-Seidel Iteration . . . . . . . . . . . 50
4.5.3 Relaxation Method . . . . . . . . . . . . . 51
4.6 Nonlinear Equation . . . . . . . . . . . . . . . . . 51
4.6.1 Simple Iterations . . . . . . . . . . . . . . 52
4.6.2 Newton-Raphson Method . . . . . . . . . 52

II Mathematical Optimization 53
5 Unconstrained Optimization 55

5.1 Univariate Functions . . . . . . . . . . . . . . . . 55

5.2 Multivariate Functions . . . . . . . . . . . . . . . 56
5.3 Gradient-Based Methods . . . . . . . . . . . . . . 59
5.3.1 Newton’s Method . . . . . . . . . . . . . . 59
5.3.2 Steepest Descent Method . . . . . . . . . 60
5.4 Hooke-Jeeves Pattern Search . . . . . . . . . . . 64
6 Linear Mathematical Programming 67

6.1 Linear Programming . . . . . . . . . . . . . . . . 67

6.2 Simplex Method . . . . . . . . . . . . . . . . . . 70
6.2.1 Basic Procedure . . . . . . . . . . . . . . 70
6.2.2 Augmented Form . . . . . . . . . . . . . . 72
6.2.3 A Case Study . . . . . . . . . . . . . . . . 73
7 Nonlinear Optimization 79

7.1 Penalty Method . . . . . . . . . . . . . . . . . . . 79

7.2 Lagrange Multipliers . . . . . . . . . . . . . . . . 81
7.3 Kuhn-Tucker Conditions . . . . . . . . . . . . . . 83
7.4 No Free Lunch Theorems . . . . . . . . . . . . . 84
III Metaheuristic Methods 87

8 Tabu Search 89
CONTENTS iii
8.1 Tabu Search . . . . . . . . . . . . . . . . . . . . . 89

8.2 Travelling Salesman Problem . . . . . . . . . . . 93

8.3 Tabu Search for TSP . . . . . . . . . . . . . . . . 95
9 Ant Colony Optimization 99

9.1 Behaviour of Ants . . . . . . . . . . . . . . . . . 99

9.2 Ant Colony Optimization . . . . . . . . . . . . . 101
9.3 Double Bridge Problem . . . . . . . . . . . . . . 103
9.4 Multi-Peak Functions . . . . . . . . . . . . . . . 104
10 Particle Swarm Optimization 107

10.1 Swarm Intelligence . . . . . . . . . . . . . . . . . 107

10.2 PSO algorithms . . . . . . . . . . . . . . . . . . . 108
10.3 Accelerated PSO . . . . . . . . . . . . . . . . . . 109
10.4 Multimodal Functions . . . . . . . . . . . . . . . 111
10.5 Implementation . . . . . . . . . . . . . . . . . . . 113
10.6 Constraints . . . . . . . . . . . . . . . . . . . . . 117
11 Simulated Annealing 119

11.1 Fundamental Concepts . . . . . . . . . . . . . . . 119

11.2 Choice of Parameters . . . . . . . . . . . . . . . . 120
11.3 SA Algorithm . . . . . . . . . . . . . . . . . . . . 122
11.4 Implementation . . . . . . . . . . . . . . . . . . . 123
12 Multiobjective Optimization 129

12.1 Pareto Optimality . . . . . . . . . . . . . . . . . 129

12.2 Weighted Sum Method . . . . . . . . . . . . . . . 133
12.3 Utility Method . . . . . . . . . . . . . . . . . . . 136
12.4 Metaheuristic Search . . . . . . . . . . . . . . . . 139
12.5 Other Algorithms . . . . . . . . . . . . . . . . . . 143
Bibliography 144

Index 149
Preface

Optimization is everywhere from routine business transactions

to important decisions of any sort, from engineering design to
industrial manufacturing, and from choosing a career path to
planning our holidays. In all these activities, there are always
some things (objectives) we are trying to optimize and these
ob-jectives could be cost, profit, performance, quality,
enjoyment, customer-rating and others. The formal approach to
these op-timization problems forms the major part of the
mathematical optimization or mathematical programming.
The topics of mathematical optimization are broad and the
related literature is vast. It is often a daunting task for begin-
ners to find a right book and to learn the right (and useful)
algorithms widely used in mathematical programming. Even for
lecturers and educators, it is not trivial to decide what al-
gorithms to teach and to provide a balanced coverage of a wide
range of topics because there are so many algorithms to
choose from. From my own learning experience, I understand
that some algorithms took substantial amount of time and effort
in programming and it was frustrating to realise that it did not
work well for optimization problems at hand in the end. Af-ter
some frustrations, then I realized other algorithms worked much
efficiently for a given problem. The initial cause was of-ten that
the advantages and disadvantages were not explicitly explained
in the literature or I was too eager to do some op-timization
simulations without realizing certain pitfalls of the related
algorithms. Such learning experience is valuable to me in
writing this book so that we can endeavour to provide a

v
balanced view of various algorithms and to provide a right
cov-erage of useful and yet efficient algorithms selected
from a wide range of optimization techniques.
Therefore, this book strives to provide a balanced coverage of
efficient algorithms commonly used in solving mathemat-ical
optimization problems. It covers both the convectional algorithms
and modern heuristic and metaheuristic methods. Topics include
gradient-based algorithms (such as the Newton-Raphson method
and steepest descent method), Hooke-Jeeves pattern search,
Lagrange multipliers, linear programming, par-ticle swarm
optimization (PSO), simulated annealing (SA), and Tabu search.
We also briefly introduce the multiobjective opti-mization including
important concepts such as Pareto optimal-ity and utility method,
and provide three Matlab and Octave programs so as to
demonstrate how PSO and SA work. In ad-dition, we will use an
example to demonstrate how to modify these programs to solve
multiobjective optimization problems using the recursive method.

I would like to thank many of my mentors, friends and

colleagues: Drs A. C. Fowler and S. Tsou at Oxford Univer-
sity, Drs J. M. Lees, T. Love, C. Morley, and G. T. Parks at
Cambridge University. Special thanks to Dr G. T. Parks who
introduced me to the wonderful technique of Tabu Search.
I also thank my publisher, Dr Victor Riecansky, at Cam-
bridge International Science Publishing, for his help and pro-
fessionalism. Last but not least, I thank my wife, Helen, and
son, Young, for their help.

Xin-She Yang

Cambridge, 2008

vi
 Part I

Fundamentals
Chapter 1

Mathematical
Optimization

Optimization is everywhere, from business to engineering

de-sign, from planning your holiday to your daily routine.
Busi-ness organizations have to maximize their profit and
minimize the cost. Engineering design has to maximize the
performance of the designed product while of course
minimizing the cost at the same time. Even when we plan
holidays we want to maximize the enjoyment and minimize
the cost. Therefore, the studies of optimization are of both
scientific interest and prac-tical implications and
subsequently the methodology will have many applications.

1.1 Optimization
Whatever the real world problem is, it is usually possible to
formulate the optimization problem in a generic form. All op-
timization problems with explicit objectives can in general be
expressed as nonlinearly constrained optimization problems
in the following generic form
maximize/minimize f (x), x = (x1, x2, ..., xn)T ∈ <n,
x <n
subject to φj (x) = 0, (j = 1, 2, ..., M ),

3
 4 Chapter 1. Mathematical Optimization
ψk (x) ≥ 0, (k = 1, ..., N ), (1.1)

where f (x), φi(x) and ψj (x) are scalar functions of the real
T
col-umn vector x. Here the components xi of x = (x1, ..., xn)
are called design variables or more often decision variables,
and they can be either continuous, or discrete or mixed of
these two. The vector x is often called a decision vector
n
which varies in a n-dimensional space < . The function f (x)
is called the objective function or cost function. In addition,
φi(x) are con-straints in terms of M equalities, and ψj (x) are
constraints written as N inequalities. So there are M + N
constraints in total. The optimization problem formulated
here is a nonlinear constrained problem.
The space spanned by the decision variables is called the
n
search space < , while the space formed by the objective func-
tion values is called the solution space. The optimization prob-
n
lem essentially maps the < domain or space of decision vari-
ables into a solution space < (or the real axis in general).
The objective function f (x) can be either linear or non-
linear. If the constraints φi and ψj are all linear, it becomes a
linearly constrained problem. Furthermore, φi, ψj and the
objective function f (x) are all linear, then it becomes a linear
programming problem. If the objective is at most quadratic
with linear constraints, then it is called quadratic program-
ming. If all the values of the decision variables can be
integers, then this type of linear programming is called
integer program-ming or integer linear programming.
Linear programming is very important in applications and
has been well-studied, while there is still no generic method for
solving nonlinear programming in general, though some im-
portant progress has been made in the last few decades. It is
worth pointing out that the term programming here means
planning, it has nothing to do with computer programming and
the wording coincidence is purely incidental.
On the other hand, if no constraints are specified so that
xi can take any values in the real axis (or any integers), the
optimization problem is referred to as the unconstrained opti-mization
problem.
1.1 Optimization 5

The simplest optimization without any constraints is prob-

ably the search of the maxima or minima of a function. For
example, finding the maximum of an univariate function f (x)
−x2 −∞ < x < ∞ , (1.2)
f (x) = xe ,

is a simple unconstrained problem. While the following prob-

lem is a simple constrained minimization problem
2 2 2 (1.3)
f (x1, x2) = x1 + x1x2 + x2 ,(x1, x2) ∈ < ,

subject to
x1 ≥ 1,x2 − 2 = 0. (1.4)

2 −x2
Example 1.1: To find the minimum of f (x) = x e , we have
0
the stationary condition f (x) = 0 or
0 −x2 2 −x2 3 −x2
f (x) = 2x × e + x × (−2x)e = 2(x − x )e = 0. As
−x2
e > 0, we have
2
x(1 − x ) = 0,
or
x = 0, x = ±1.
The second derivative
00 −x2 2 4
f (x) = 2e (1 − 5x + 2x ),
which is an even function with respect to x. So at x = ±1, f
00 2 4 −(±1)2 −1
(±1) = 2[1 − 5(±1) + 2(±1) ]e = −4e < 0. Thus,
−1
the maximum of fmax = e occur at x∗ = ±1. At x = 0, we have f
00
(0) = 2 > 0, thus the minimum of f (x) occur at x∗ = 0
with fmin(0) = 0.

It is worth pointing out that the objectives are explicitly

known in all the optimization problems to be discussed in this
book. However, in reality, it is often difficult to quantify what we
want to achieve, but we still try to optimize certain things
6 Chapter 1. Mathematical Optimization

such as the degree of enjoyment or the quality of service on

holiday. In other cases, it might be impossible to write the
objective function in an explicit mathematical form.
Whatever the objectives, we have to evaluate the
objectives many times. In most cases, the evaluations of the
objective functions consume a lot of computational power
(which costs money) and design time. Any efficient algorithm
that can re-duce the number of objective evaluations will
save both time and money. The algorithms presented in this
book will still be applicable to the cases where the objectives
are not known explicitly, though certain modifications are
required to suit a particular application. The basic principle of
these search al-gorithms remain the same.

1.2 Optimality Criteria

In mathematical programming, there are many important
con-cepts that will be introduced in this book. Now we first
intro-duce three related concepts: feasible solutions, the
strong local maximum and the weak local maximum.
A point x which satisfies all the constraints is called a feasi-
ble point and thus is a feasible solution to the problem. The set
of all feasible points is called the feasible region. A point x ∗ is
called a strong local maximum of the nonlinearly constrained
optimization problem if f (x) is defined in a δ-neigbourhood N
(x∗, δ) and satisfies f (x∗) > f (u) for ∀u ∈ N (x∗, δ) where
δ > 0 and u 6=x∗. If x∗ is not a strong local maximum, the
inclusion of equality in the condition f (x ∗) ≥ f (u) for ∀u ∈ N (x∗, δ)
defines the point x∗ as a weak local maximum (see Fig. 1.1). The
local minima can be defined in the similar
manner when > and ≥ are replaced by < and ≤, respectively.
Figure 1.1 shows various local maxima and minima. Point A
is a strong local maximum, while point B is a weak local
maximum because there are many (well, infinite) different val-
ues of x which will lead to the same value of f (x ∗). Point D is a
global maximum. However, point C is a strong local minimum,
but it has a discontinuity in f 0(x∗). So the stationary condition
 1.3 Computational Complexity 7
f (x) global maximum

6 weak local @
@RD
strong maximum
local maximum B

A KA
C
I@
A
@
A
weak local A
minimum local minimum
with discontinuity
local minimum -
0 x
Figure 1.1: Strong and weak maxima and minima.

0
for this point f (x∗) = 0 is not valid. We will not deal with this
type of minima or maxima in detail. In our present discussion,
0
we will assume that both f (x) and f (x) are always continuous
or f (x) is everywhere twice-continuously differentiable.

2
Example 1.2: The minimum of f (x) = x at x = 0 is a strong
2 2
local minimum. The minimum of g(x, y) = (x − y) + (x − y)
at x = y = 0 is a weak local minimum because g(x, y) = 0
along the line x = y so that g(x, y = x) = 0 = g(0, 0).

1.3 Computational Complexity

The efficiency of an algorithm is often measured by the algo-
rithmic complexity or computational complexity. In literature,
this complexity is also called Kolmogorov complexity. For a
given problem size n, the complexity is denoted using Big-O
2
notations such as O(n ) or O(n log n).
Loosely speaking, for two functions f (x) and g(x), if

lim f (x) → K, (1.5)

x→x0 g(x)
8 Chapter 1. Mathematical Optimization

where K is a finite, non-zero limit, we write

f = O(g). (1.6)

The big O notation means that f is asymptotically equivalent

to the order of g(x). If the limit is unity or K = 1, we say f (x) is
order of g(x). In this special case, we write

f ∼ g, (1.7)

which is equivalent to f /g → 1 and g/f → 1 as x → x 0. Ob-

viously, x0 can be any value, including 0 and ∞. The notation
∼ does not necessarily mean ≈ in general, though they
might give the same results, especially in the case when x →
0 [for example, sin x ∼ x and sin x ≈ x if x → 0].
When we say f is order of 100 (or f ∼ 100), this does not
mean f ≈ 100, but it can mean that f is between about 50 to
150. The small o notation is often used if the limit tends to 0. That is
lim f → 0, (1.8)
x→x0 g
or
f = o(g). (1.9)
2
If g > 0, f = o(g) is equivalent to fg. For example, for
x 2 x
∀x ∈ R, we have e ≈ 1 + x + O(x ) ≈ 1 + x + 2
+ o(x).

Example 1.3: A classical example is Stirling’s asymptotic

series for factorials
√ nn 1 1 139
n! ∼ 2πn ( e ) (1 + 12n + 288n2 − 51480n3 − ...). (1.10)
This is a good example of asymptotic series. For standard
k
power expansions, the error R k (h ) → 0, but for an asymptotic
series, the error of the truncated series R k decreases√ and gets
n
smaller compared with the leading term [here 2πn(n/e) ].
However, Rn does not necessarily tend to zero. In fact,

1 √ n
R2 = 12 n · 2πn(n/e) ,
1.4 NP-Complete Problems 9

is still very large as R2 → ∞ if n 1. For example, for n = 100,

we have n! = 9.3326 × 10157, while the leading approximation is
√
n 157
2πn(n/e) = 9.3248 ×10 . The difference between these
154
two values is 7.7740 × 10 , which is still very large, though
three orders smaller than the leading approximation.
Let us come back to the computational complexity of an
algorithm. For the sorting algorithm for a given number of n
data entries, sorting these numbers into either ascending or
descending order will take the computational time as a
function of the problem size n. O(n) means a linear
2
complexity, while O(n ) has a quadratic complexity. That is, if
n is doubled, then the time will double for linear complexity,
but it will quadruple for quadratic complexity.
For example, the bubble sorting algorithm starts at the
be-ginning of the data set by comparing the first two
elements. If the first is smaller than the second, then swap
them. This comparison and swap process continues for each
possible pair of adjacent elements. There are n × n pairs as
we need two loops over the whole data set, then the
2
algorithm complexity is O(n ). On the other hand, the
quicksort algorithm uses a divide-and-conquer approach via
partition. By first choosing a pivot element, we put all the
elements into two sublists with all the smaller elements
before the pivot and all the greater elements after it. Then,
the sublists are recursively sorted in a similar manner. This
algorithm will result in a complexity of O(n log n). The
quicksort is much more efficient than the bubble algorithm.
2
For n = 1000, the bubble algorithm will need about O(n ) ≈
6
O(10 ) calculations, while the quicksort only requires O(n log
3
n) ≈ O(3 ×10 ) calculations (at least two orders less).

1.4 NP-Complete Problems

In mathematical programming, an easy or tractable problem is
a problem whose solution can be obtained by computer algo-
rithms with a solution time (or number of steps) as a polyno-
10 Chapter 1. Mathematical Optimization

mial function of problem size n. Algorithms with polynomial-

time are considered efficient. A problem is called the P-
problem or polynomial-time problem if the number of steps
needed to find the solution is bounded by a polynomial in n
and it has at least one algorithm to solve it.
On the other hand, a hard or intractable problem requires
solution time that is an exponential function of n, and thus
exponential-time algorithms are considered inefficient. A prob-
lem is called nondeterministic polynomial (NP) if its solution can
only be guessed and evaluated in polynomial time, and there is
no known rule to make such guess (hence, nondetermin-istic).
Consequently, guessed solutions cannot guarantee to be
optimal or even near optimal. In fact, no known algorithms ex-
ist to solve NP-hard problems, and only approximate solutions
or heuristic solutions are possible. Thus, heuristic and meta-
heuristic methods are very promising in obtaining approximate
solutions or nearly optimal/suboptimal solutions.
A problem is called NP-complete if it is an NP-hard
problem and all other problems in NP are reducible to it via
certain re-duction algorithms. The reduction algorithm has a
polynomial time. An example of NP-hard problem is the
Travelling Sales-man Problem, and its objective is to find the
shortest route or minimum travelling cost to visit all given n
cities exactly once and then return to the starting city.
The solvability of NP-complete problems (whether by
poly-nomial time or not) is still an unsolved problem which
the Clay Mathematical Institute is offering a million dollars
reward for a formal proof. Most real-world problems are NP-
hard, and thus any advance in dealing with NP problems will
have potential impact on many applications.
Chapter 2

Norms and Hessian

Matrices

Before we proceed to study various optimization methods,

let us first review some of the fundamental concepts such as
norms and Hessian matrices that will be used frequently
through this book.

2.1 Vector and Matrix Norms

For a vector v, its p-norm is denoted by kvkp and defined as

n (2.1)
kvk p = ( |v i| p) 1/p,

X
i=1

where p is a positive integer. From this definition, it is straight-

forward to show that the p-norm satisfies the following condi-
tions: kvk ≥ 0 for all v, and kvk = 0 if and only if v = 0. This is the
non-negativeness condition. In addition, for any real number α,
we have the scaling condition: kαvk = αkvk.
Three most common norms are one-, two- and infinity-
norms when p = 1, 2, and ∞, respectively. For p = 1, the one-
norm is just the simple sum of each component |v i|, while the
two-norm kvk2 for p = 2 is the standard Euclidean norm
11
12 Chapter 2. Norms and Hessian Matrices
because kvk2 is the length of the vector v

√ q 2 2 2 ,
kvk2 = v·v = v1 + v2 + ... + vn (2.2)

where u · v is the inner product of two vectors u and v.

For the special case p = ∞, we denote vmax the maximum
absolute value of all the components v i, or vmax ≡ max |vi| =
max(|v1|, |v2|, ..., |vn|).
n n 1/p
k k∞ = p→∞
v

| i| vmax
i=1 ma
|x vi | p )1/p

p→∞ i=1 v
p

lim ( v = lim p
X X

1 1 n 1
vi
|v
p
= plim (v max
)p (

X
p
| ) p =v
max
lim (
p

X
|v vi
|p ) p . (2.3)

→∞ →∞
max i=1 max

Since |vi/vmax| ≤ 1 and for all terms |v i/vmax| < 1, we have |

vi/vmax |p → 0 when p → ∞. Thus, the only non-zero term in
the sum is one when |vi/vmax| = 1, which means that
plim n (2.4) |v i /v ma x|p = 1.

X
→∞
i=1

Therefore, we finally have

kvk∞ = vmax = max |vi|. (2.5)

For the uniqueness of norms, it is necessary for the norms to

satisfy the triangle inequality

ku + vk ≤ kuk + kvk. (2.6)

It is straightforward to check that for p = 1, 2, and ∞ from

their definitions, they indeed satisfy the triangle inequality.
The equality occurs when u = v. It leaves as an exercise to
check this inequality is true for any p > 0.
T and v =
Example 2.1:T For the vector u = 5 2 3 −2
−2 0 1 2 , then the p-norms of u are

kuk1 = |5| + |2| + |3| + | − 2| = 12,

2.1 Vector and Matrix Norms 13
q √
2 2 2 2
kuk2 = 5 + 2 + 3 + (−2) = 42,
and
kuk∞ = max(5, 2, 3, −2) = 5.

Similarly, kvk1 = 5, kvk2 = 3 and kvk∞ = 2. We know that

5 + −2 3
2+0 2
u+v= 3+1 = 4 ,
2+2 0

−
√
and its corresponding norms are ku + vk1 = 9, ku + vk2 = 29
and ku + vk∞ = 4. It is straightforward to check that
ku + vk1 = 9 < 12 + 5 = kuk1 + kvk1,
√ √
ku + vk2 = 29 < 42 + 3 = kuk2 + kvk2,
and
ku + vk∞ = 4 < 5 + 4 = kuk∞ + kvk∞.

Matrices are the extension of vectors, so we can define

the corresponding norms. For an m × n matrix A = [a ij ], a
simple way to extend the norms to use the fact that Au is a
vector for any vector kuk = 1. So the p-norm is defined as
kAk p = (
m n |a i j |p )1 /p . (2.7)

XX
i=1 j=1

Alternatively, we can consider that all the elements or en-

tries aij form a vector. A popular norm, called Frobenius form
(also called the Hilbert-Schmidt norm), is defined as

kAk F =
a2
m n 1/2 (2.8)
i=1 j=1 ij .

XX

In fact, Frobenius norm is 2-norm.

14 Chapter 2. Norms and Hessian Matrices

Other popular norms are based on the absolute column

sum or row sum. For example,
≤≤ m X

k A k = max ( |a ij | ), (2.9)
1 1 j n
i=1
which is the maximum of the absolute column sum, while
≤≤ n
X

kA k = max ( |aij |), (2.10)

∞ 1 i m
j=1

is the maximum of the absolute row sum. The max norm is

defined as
kAk = max{|a |}. (2.11)
max ij
From the definitions of these norms, we know that they
sat-isfy the non-negativeness condition kAk ≥ 0, the scaling
condi-tion kαAk = |α|kAk, and the triangle inequality kA + Bk
≤ kAk + kBk.
2 3
Example 2.2: For the matrix A = 4 !
−5 , we know that
q √
2 2 2 2
kAkF = kAk2 = 2 + 3 + 4 + (−5) = 54,

|2| + |3|
kAk∞ = max = 9,

|4| + | − 5|
and
kAkmax = 5.

2.2 Eigenvalues and Eigenvectors

The eigenvalues λ of a n × n square matrix A are determined
by
Au = λu, (2.12)
2.2 Eigenvalues and Eigenvectors 15
or

(A − λI )u = 0. (2.13)
where I is a unitary matrix with the same size as A. Any non-
trivial solution requires that
det |A − λI | = 0, (2.14)
a a a
or 21 22 λ ... 2n
a ... a
a11 − λ 12 1n

− = 0, (2.15)
..
.. .

a a
. a
n1 n2 ... nn λ

which
again can be written as a poly nomial

n n−1 (2.16)
λ + αn−1λ + ... + α0 = (λ − λ1)...(λ − λn) = 0,

where λi are the eigenvalues which could be complex

numbers. For each eigenvalue λ, there is a corresponding
eigenvector u whose direction can be uniquely determined.
However, the length of the eigenvector is not unique
because any non-zero multiple of u will also satisfy equation
(2.12), and thus can be considered as an eigenvector. For
this reason, it is usually nec-essary to apply an additional
condition by setting the length as unity, and subsequently the
eigenvector becomes a unit eigen-vector.
In general, a real n × n matrix A has n eigenvalues λ i(i =
1, 2, ..., n), however, these eigenvalues are not necessarily
T
dis-tinct. If the real matrix is symmetric, that is to say A = A,
then the matrix has n distinct eigenvectors, and all the eigen-
values are real numbers. Furthermore, the inverse of a
positive definite matrix is also positive definite. For a linear
system Au = f where f is a known column vector, if A is
positive definite, then the system can be solved more
efficiently by the matrix decomposition method.

Example 2.3: The eigenvalues of the square matrix

!
4 9

A = 2 −3 ,
16 Chapter 2. Norms and Hessian Matrices
can be obtained by solving

2 3 λ
4−λ −−
9 = 0.

We have

(4 − λ)(−3 − λ) − 18 = (λ − 6)(λ + 5) = 0.
Thus, the eigenvalues are λ = 6 and λ = −5. Let v = (v1 v2)T
be the eigenvector, we have for λ = 6
!
| − | 29 v2 !
A λI = −2 9 v1 = 0,
−
which means that

−2v1 + 9v2 = 0, 2v1 − 9v2 = 0.

These two equations are virtually the same (not linearly

indepen-dent), so the solution is
9
v = v.
1 2 2

Any vector parallel to v is also an eigenvector. In order to get a

unique eigenvector, we have to impose an extra requirement,
that is the length of the vector is unity. We now have
2 2
v1 + v2 = 1,
or
9 v2 2 2
( 2 ) + v2 = 1,
√ √
which gives v2 = ±2/ 85, and v1 = ±9/ 85. As these two
vectors are in opposite direction, we can choose any of the
two directions. So the eigenvector for the eigenvalue λ = 6 is
√
9/ 85
v =√ .

2/85
2.2 Eigenvalues and Eigenvectors 17

Similarly, the corresponding eigenvector for the eigenvalue λ = −5

√ √
T
is v = (− 2/2 2/2) .
A square symmetric matrix A is said to be positive definite
if all its eigenvalues are strictly positive (λ i > 0 where i = 1, 2,
T
..., n). By multiplying (2.12) by u , we have
T T T (2.17)
u Au = u λu = λu u,
which leads to u Au
T

λ= T . (2.18)
u u
This means that
T if λ > 0. (2.19)
u Au > 0,

In fact, for any vector v, the following relationship holds

T (2.20)
v Av > 0.

For v can be a unit vector, thus all the diagonal elements of A

should be strictly positive as well. If all the eigenvalues are
non-negative or λi ≥ 0, then the matrix is called positive
semi-definite. In general, an indefinite matrix can have both
positive and negative eigenvalues.

Example 2.4: In general, a 2 × 2 symmetric matrix A

!
α β

A= β γ ,

is positive definite if
αu21 + 2βu1u2 + γu22 > 0, for all u = (u1, u2)T
6= 0. The inverse ofA is
αγ − β2 −β α
!

1 γ
A−1 = −β ,

which is also positive definite.

18 Chapter 2. Norms and Hessian Matrices
From the previous example, we know that the eigenvalues of

A= 2 1!,
1 2

are λ = 3, −1. So the matrix is indefinite. For another matrix

!
4 6

B= 6 20 ,

we can find its eigenvalues using the similar method as

discussed earlier, and the eigenvalues are λ = 2, 22. So
matrix B is positive definite. The inverse of B
44 6 4!
= 1 20
B−1 −6 ,
−
−1
is also positive definite because B has two eigenvalues: λ
= 1/2, 1/22.

2.3 Spectral Radius of Matrices

Another important concept related to the eigenvalues of
matrix is the spectral radius of a square matrix. If λ i(i = 1,
2, ..., n) are the eigenvalues (either real or complex) of a
matrix A, then the spectral radius ρ(A) is defined as
)≡
ρ( 1 i n {|λ i|} , (2.21)
A max
≤≤

which is the maximum absolute value of all the eigenvalues.

Geometrically speaking, if we plot all the eigenvalues of the
matrix A on the complex plane, and draw a circle on a complex
plane so that it encloses all the eigenvalues inside, then the
minimum radius of such a circle is the spectral radius.
For any 0 < p ∈ <, the eigenvectors have non-zero
p
norms kuk 6= 0 andku k 6= 0. UsingAu = λu and taking the
norms, we have
p p p p
|λ| ku k = k(λu) k = k(Au) k ≤ kA kku k.
p p (2.22)
2.3 Spectral Radius of Matrices 19

p
By dividing both sides of the above equation by ku k 6= 0, we
reach the following inequality

p p 1/p (2.23)
|λ| ≤ kA k ,

which is valid for any eigenvalue. Therefore, it should also be

valid for the maximum absolute value or ρ(A). We finally have

ρ(A) ≤ kA k
p 1/p
, (2.24)

which becomes an equality when p → ∞.

The spectral radius is very useful in determining whether
an iteration algorithm is stable or not. Most iteration schemes
can be written as
(2.25)
u(n+1) = Au(n) + b,

where b is a known column vector and A is a square matrix

with known coefficients. The iterations start from an initial
(0) (0)
guess u (often, set u = 0), and proceed to the
(n+1)
approximate solution u . For the iteration procedure to be
stable, it requires that ρ(A) ≤ 1. If ρ(A) > 1, then the algorithm
will not be stable and any initial errors will be amplified in
each iteration.
In the case of A is a lower (or upper) matrix
a ... 0
a21 22
a 0 ... 0
11
A= . , (2.26)
.. .
. .

a a
an1 n2 ... nn

then its eigenvalues are the diagonal entries: a 11, a22, ...,
ann. In addition, the determinant of the triangular matrix A is
sim-ply the product of its diagonal entries. That is
n
Y

det(A) = |A| =aii = a11a22...ann. (2.27)

i=1
20 Chapter 2. Norms and Hessian Matrices
Obviously, a diagonal matrix 1
0
d
2 ... 0
d 0 ... 0
.
D = diag(d1, d2, ..., dn) = .. ,(2.28)
0 0 ... dn

is just a special case of a triangular matrix. Thus, the proper-

ties for its inverse, eigenvalues and determinant are the
same as the above.
These properties are convenient in determining the stability of
an iteration scheme such as the Jacobi-type and Gauss-Seidel
iteration methods where A may contain triangular matrices.

Example 2.5: Determine if the following iteration is stable or

not u2 = 1 2 2 u2 + 2 .

u1 5 2 0 u1 2
u2 n+1 4 −
1/2
2u3 −
5

We know the eigenvalues of

A= 1 −2 2 ,
5 2 0
4 1/2 2
−
are
λ1 = 5.5548, λ2 = −2.277 + 1.0556i, λ3 = −2.277 − 1.0556i.
Then the spectral radius is
ρ(A) = max (|λi|) ≈ 5.55 > 1,
i {1,2,3}

therefore, the iteration process will not be convergent.

2.4 Hessian Matrix

The gradient vector of a multivariate function f (x) is defined as
G1(x) ≡ rf (x) ≡ ∂x1 , ∂x2 ,..., ∂xn T ,
(2.29)
∂f ∂f ∂f
2.4 Hessian Matrix 21

T
where x = (x1, x2, ..., xn) is a vector. As the gradient rf (x) of
a linear function f (x) is always a constant vector k, then any
linear function can be written as
T (2.30)
f (x) = k x + b,
where b is a constant vector.
The second derivatives of a generic function f (x) form an
n × n matrix, called Hessian matrix, given by
∂f ... ∂f
2
∂x ∂x1∂xn
2 .1 .
G2 (x) ≡r f (x) ≡
. .
, (2.31)
. .

2
∂f ∂ f
...
∂x1∂xn ∂xn 2

which is symmetric due to the fact that

2 = ∂2f . (2.32)
∂ f
∂xi∂xj ∂xj ∂xi

1 T T
f (x) = 2 x Ax + k x + b.
When the Hessian matrix G2(x) = A is a constant matrix (the values
of its entries are independent of x), the function f (x) is called a
quadratic function, and can subsequently be written as

(2.33)

The factor 1/2 in the expression is to avoid the appearance

everywhere of a factor 2 in the derivatives, and this choice is
purely out of convenience.
2 2
Example 2.6: The gradient of f (x, y, z) = x + y + yz sin(x) is
T
G1 = 2x + yz cos(x) 2y + z sin(x) y sin(x) .
The Hessian matrix is given by
2
∂2f
2
∂ f G2= ∂y∂x ∂y
2
∂x ∂x∂y
2 2
∂ f∂ f
2 2
∂ f ∂ f ∂z∂x∂z∂y
 2
∂ f
∂x∂z 2−yz sin(x)
2

∂ f = z cos(x)
∂y∂z
2

∂ f y cos(x)
∂z 2
22 Chapter 2. Norms and Hessian Matrices

(a) (b)

Figure 2.1: Convexity: (a) non-convex, and (b) convex.

f (x)
6
A

HH
HH Q
HH
HHB
Hr
P
-
0Lβ - Lα -
x
Figure 2.2: Convexity of a function f (x). Chord AB lies above
the curve segment joining A and B. For any point P , we have
Lα = αL, Lβ = βL and L = |xB − xA|.

2.5 Convexity

Nonlinear programming problems are often classified

accord-ing to the convexity of the defining functions.
Geometrically speaking, an object is convex if for any two
points within the object, every point on the straight line
segment joining them is also within the object. Examples are
a solid ball, a cube or a pyramid. Obviously, a hollow object
is not convex. Three examples are given in Fig. 2.1.
Mathematically speaking, a set S ∈ <n in a real vector
2.5 Convexity 23
space is called a convex set if

tx + (1 − t)y ∈ S, ∀(x, y) ∈ S, t ∈ [0, 1]. (2.34)

A function f (x) defined on a convex set Ω is called convex if

and only if it satisfies

f (αx + βy) ≤ αf (x) + βf (y),∀x, y ∈ Ω, (2.35)

and
α ≥ 0, β ≥ 0, α + β = 1. (2.36)
Geometrically speaking, the chord AB lies above the curve seg-
ment AP B joining A and B (see Fig. 2.2). For example, for any
point P between A and B, we have xP = αxA + βxB with
α =L = x −x ≥ 0,
α B P
L xB − xA
β = Lβ = xP − xA ≥ 0, (2.37)
L xB − xA
which indeed gives α + β = 1. In addition, we know that
αx + βx = x (x − x ) + x (x − x ) = x . (2.38)
A B A B P B P A P

xB − xA xB − xA
The value of the function f (xP ) at P should be less than or
equal to the weighted combination αf (xA) + βf (xB ) (or the
value at point Q). That is

f (xP ) = f (αxA + βxB ) ≤ αf (xA) + βf (xB ). (2.39)

2
Example 2.7: For example, the convexity of f (x) = x − 1
requires
2 2 2 (2.40)
(αx + βy) − 1 ≤ α(x − 1) + β(y − 1), ∀x, y ∈ <,
where α, β ≥ 0 and α + β = 1. This is equivalent to require

2 2 2 (2.41)
αx + βy − (αx + βy) ≥ 0,
24 Chapter 2. Norms and Hessian Matrices

where we have used α + β = 1. We now have

2 2 2 2 2 2
αx + βy − α x − 2αβxy − β y
2 2 (2.42)
= α(1 − α)(x − y) = αβ(x − y) ≥ 0,
2
which is always true because α, β ≥ 0 and (x−y) ≥ 0.
2
Therefore, f (x) = x − 1 is convex for ∀x ∈ <.
A function f (x) on Ω is concave if and only if g(x) = −f (x)
is convex. An interesting property of a convex function f is
that the vanishing of the gradient df /dx|x∗ = 0 guarantees
that the point x∗ is a global minimum of f . If a function is not
convex or concave, then it is much more difficult to find
global minima or maxima.
Chapter 3

Root-Finding
Algorithms

The essence of finding the solution of an optimization

problem is equivalent to finding the critical points and
extreme points. In order to find the critical points, we have to
solve the sta-tionary conditions when the first derivatives are
zero, though it is a different matter for the extreme points at
boundaries. Therefore, root-finding algorithms are important.
Close-form solutions are rare, and in most cases, only
approximate solu-tions are possible. In this chapter, we will
introduce the fun-damentals of the numerical techniques
concerning root-finding algorithms.

3.1 Simple Iterations

The essence of root-finding algorithms is to use iteration proce-

dure to obtain the approximate (well sometimes quite accurate)
solutions, starting from some initial guess solution. For exam-
ple, even ancient Babylonians knew how to find the square root
of 2 using the iterative method. From the numerical technique
we learnt at school, we know that we can numerically compute
√
the square root of any real number k ( so that x = k) using

25
26 Chapter 3. Root-Finding Algorithms
the equation 1 k

x= (x + ), (3.1)
2 x
starting from a random guess, say, x = 1. The reason is that
√
the above equation can be rearranged to get x = k. In order
to carry out the iteration, we use the notation x n for the value
of x at n-th iteration. Thus, equation (3.1) provides a way of
calculating the estimate of x at n + 1 (denoted as xn+1). We
have 1 k
x = (xn + ). (3.2)
n+1 2 xn
If we start from an initial value, say, x0 = 1 at n = 0, we can
do the iterations to meet the accuracy we want.
√
Example 3.1: To find 5, we have k = 5 with an initial guess x0
= 1, and the first five iterations are as follows:
1 5
x1 = 2 (x0 x ) = 3,
+ (3.3)
0
1 5 ) ≈ 2.333333333,
x2 = 2 (x1 + x (3.4)
1
x3 ≈ 2.238095238, x4 ≈ 2.236068895, (3.5)

x5 ≈ 2.236067977. (3.6)
We can see that x5 after 5 iterations is very close to its true
√

value 5 = 2.23606797749979..., which shows that the

iteration method is quite efficient.
The reason that this iterative process works is that the se-
√
ries x1, x2, ..., xn converges into the true value √k due to the fact
1 2
that xn+1/xn = 2 (1 + k/x n) → 1 as xn → k. However, a good
choice of the initial value x0 will speed up the conver-gence. Wrong
choice of x0 could make the iteration fail, for example, we cannot
use x0√= 0 as the initial guess, and we
cannot use x0 < 0 either as k > 0 (in this case, the iterations √
will approach another root − k).
So a sensible choice should be an educated guess. At the
initial step, if x20 < k, x0 is the lower bound and k/x0 is upper
3.2 Bisection Method 27

f (x)
6 A

-
xa x∗ xn xb
0 x
B

Figure 3.1: Bisection method for finding the root x ∗ of f (x∗)

= 0 between two bounds xa and xb in the domain x ∈ [a, b].

2
bound. If x 0 > k, then x0 is the upper bound and k/x0 is the
lower bound. For other iterations, the new bounds will be x n
and k/xn. In fact, the value xn+1 is always between these two
bounds xn and k/xn, and the new estimate xn+1 is thus the
mean or average of the two bounds. This guarantees that
√
the series converges into the true value of k. This method is
similar to the bisection method below.

3.2 Bisection Method

√
The above-mentioned iteration method to find x = k is in fact
equivalent to find the solution or the root of the function f (x)
2
= x − k = 0. For any function f (x) in the interval [a, b], the
root-finding bisection method works in the following way as
shown in Fig. 3.1.
The iteration procedure starts with two initial guessed bounds
xa (lower bound), and xb (upper bound) so that the true root x = x∗
lies between these two bounds. This requires that f (xa) and f (xb)
have different signs. In our case shown in Fig. 3.1, f (xa) > 0 and f
(xb) < 0, but f (xa)f (xb) < 0. The obvious choice is xa = a and xb = b.
The next estimate is just the
28 Chapter 3. Root-Finding Algorithms
midpoint of A and B, and we have

1
xn = 2 (xa + xb). (3.7)
We then have to test the sign of f (x n). If f (xn) < 0 (having the
same sign as f (x b)), we then update the new upper bound as
xb = xn. If f (xn) > 0 (having the same sign as f (x a)), we update
the new lower bound as x a = xn. In a special case when f (x n) =
0, we have found the true root. The iterations continue in the
same manner until a given accuracy is achieved or the
prescribed number of iterations is reached.
√
Example 3.2: If we want to find π, we have
2
f (x) = x − π = 0.
√
We can use xa = 1 and xb = 2 since π < 4 (thus π < 2). The
first bisection point is
x = 1 (x + x ) = 1 (1 + 2) = 1.5.
1 a b
2 2
Since f (xa) < 0, f (xb) > 0 and f (x1) = −0.8916 < 0, we
update the new lower bound xa = x1 = 1.5. The second
bisection point is
1
x2 = 2 (1.5 + 2) = 1.75,
and f (x2) = −0.0791 < 0, so we update lower bound again x a
= 1.75. The third bisection point is
1
x3 = 2 (1.75 + 2) = 1.875.
Since f (x3) = 0.374 > 0, we now update the new upper
bound xb = 1.875. The fourth bisection point is
1
x4 = 2 (1.75 + 1.875) = 1.8125.
√
It is within 2.5% from the true value of π ≈ 1.7724538509.
In general, the convergence of the bisection method is
very slow, and Newton’s method is a much better choice in most
cases.
3.3 Newton’s Method 29

3.3 Newton’s Method

Newton’s method is a widely-used classic method for finding
the zeros of a nonlinear univariate function of f (x) on the inter-
val [a, b]. It is also referred to as the Newton-Raphson method.
At any given point xn shown in Fig. 3.2, we can approximate the
function by a Taylor series for x = xn+1 = xn about xn,
0
f (xn+1) = f (xn + x) ≈ f (xn) + f (xn)Δx, (3.8)

which leads to
x − x = x ≈ f (x ) − f (x ) ,
n+1 n
n+1 n 0
(3.9)
f (xn)
or f (x ) f (x )
x n+1 ≈ x n + n+1 − n .
(3.10)
0
f (xn)
Since we try to find an approximation to f (x) = 0 with f (x n+1),
we can use the approximation f (xn+1) ≈ 0 in the above expres-
sion. Thus we have the standard Newton iterative formula
f (xn)
xn+1 = xn − . (3.11)
f 0(xn)
The iteration procedure starts from an initial guess x0 and
continues until certain criterion is met. A good initial guess
will use less number of steps, however, if there is no obvious
initial good starting point, you can start at any point on the
interval [a, b]. But if the initial value is too far from the true
zero, the iteration process may fail. So it is a good idea to
limit the number of iterations.

Example 3.3: To find the root of

−x
f (x) = x − e = 0,

we use the Newton’s method starting from x0 = 1. We know

0 −x
that f (x) = 1 + e ,
30 Chapter 3. Root-Finding Algorithms

f (x)
6 A

x
∗ -
xn
x
0 n+1 x

Figure 3.2: Newton’s method to approximate the root x ∗ by

xn+1 from the pervious value xn.

and thus the iteration formula becomes

−x
xn − e n

x =x − .
n+1 n 1 + e−xn
Using x0 = 1, we have
−1
1−e
x = 1− ≈ 0.5378828427,
1 1 + e−1
and
x2 ≈ 0.5669869914, x3 ≈ 0.5671432859.
We can see that x3 (only three iterations) is very close to the
true root is x∗ ≈ 0.5671432904.
We have seen that Newton’s method is very efficient and
is thus so widely used. This method can be modified for
solving unconstrained optimization problems because it is
0
equivalent to find the root of the first derivative f (x) = 0
once the objective function f (x) is given.

3.4 Iteration Methods

Sometimes we have to find roots of functions of multiple vari-
ables, and Newton’s method can be extended to carry out such
3.4 Iteration Methods 31
task. For nonlinear multivariate functions

F (x) = [F1(x), F2(x), ..., FN (x)] ,

T (3.12)

T T
where x = (x, y, ..., z) = (x1, x2, ..., xp) , an iteration method
is usually needed to find the roots

F (x) = 0. (3.13)

The Newton-Raphson iteration procedure is widely used. We

n
first approximate F (x) by a linear residual function R(x; x ) in
n
the neighbourhood of an existing approximation x to x, and
we have

n n n n (3.14)
R(x, x ) = F (x ) + J(x )(x − x ),
and
J(x) = rF , (3.15)
where J is the Jacobian of F . That is
J = ∂F .
ij i
(3.16)

∂xj
n
Here we have used the notation x for the vector x at the n-th
n
iteration, which should not be confused with the power u of a
vector u. This might be confusing, but such notations are widely
used in the literature of numerical analysis. An alterna-tive (and
n (n)
better) notation is to denote x as x , which shows the vector
value at n-th iteration using a bracket. However, we will use
both notations if no confusion could arise.
n+1
To find the next approximation x from the current es-
n n+1 n
timate x , we have to try to satisfy R(x , u ) = 0, which is
equivalent to solve a linear system with J being the
coefficient matrix
n+1 n −1 n (3.17)
x = x − J F(x ),
under a given termination criterion

n+1 n
kx −x k≤.
32 Chapter 3. Root-Finding Algorithms

0
Iterations require an initial starting vector x , which is often
0
set to x = 0.
Example 3.4: To find the roots of the system

−y 2
x−e = 0, x − y = 0,
we first write it as ! !
x1
2
x2 x2 y!
F (x) = −x , x = x .
x1 − e 2 x= 1
−

The Newton-Raphson iteration formula becomes

n+1 n −1 n
x = x − J F (x ),
where the Jacobian J is
∂F1 ∂F1 ! !
∂F2 ∂F2
,
J=
= 2x1 1
∂x1 ∂x2 1 e−x2

∂x1 ∂x2
−

whose inverse is
−x
−1 − 2x1e 2 −2x1 1 !
A=J
−1 = 1 −1
−e−x2
= −x −
1 + 2x1e 2 2x1 1 !.
1 1 e x2
−
Therefore, the iteration equation becomes

xn+1 = xn − un
where ! !
−x 1 2
1 + 2x2e 2 2x1 x1 x2
n −1 n x
u =J F (x ) = 1 1 e−x2 1
− e−x2
− ! −
1 + 2x1e−x2 x12 + x2 2x1e−x2
2 −x
= 1 x1 + (x1 − 1 − x2)e 2 .
−
3.4 Iteration Methods 33

If we start with the initial guess x 0 = (0, 0)T , we have the first
estimate x1 as !
0 − 0! 0!
1
x 0 −1 =
= 1 ,
and the second iteration gives !
0.3333 .
2 ! !
x = 0 − −0.33333 =
1 0.33333 0.66667
If we continue this way, the third iteration gives

!
3
x =x −
2
−0.09082847 = 0.42415551! .
0.01520796 0.6514462
Finally, the fourth iteration gives !.
0.42630051

4 3 − !
x =x − − 0.002145006 =
0.001472389 0.65291859

The true roots occur at (0.6529186405, 0.4263027510), and

we can see that even after only four iterations, the estimates
are very close to the true values.
With these fundamentals of mathematics and numerical
techniques, we are now ready to solve optimization
problems. In Part II, we will introduce the conventional
methods that are widely used in mathematical programming.
Chapter 4

System of Linear
Equations

4.1 Linear systems

A linear system of m equations for n unknowns

a11u1 + a12u2 + ... + a1nun = b1,

a21u1 + a22u2 + ... + a2nun = b2,

. .
. .

. .
am1u1 + am2u2 + ... + amnun = bn, (4.1)
can be written in the compact form as b 2
a a
a21 22 ... 2n u2
a a ... a1n
11 12 u1 b1
.
. .. .. ..

am1
a
m2
.. ... a
mn un
. = bn
. , (4.2)

or simply
Au = b. (4.3)

If m < n, the system is under-determined as the conditions are

not adequate to guarantee a unique solution. On the other

35
36 Chapter 4. System of Linear Equations

hand, the system is over-determined if m > n because there

are too many conditions and a solution may not exist at all.
The unique solution is only possible when m = n.
The solution of this matrix equation is important to many
numerical problems, ranging from the solution of a large
system of linear equations to linear mathematical
programming, and from data interpolation to finding solutions
to finite element problems.
−1
The inverse of A is possible only if m = n. If the inverse A
does not exist, then the linear system is under-determined or
there are no unique solutions (or even no solution at all). In
−1
order to find the solutions, we multiply both sides by A ,
−1 −1 (4.4)
A Au = A b,

and we obtain the solution

−1 (4.5)
u = A b.

A special case of the above equation is when b = λu, and

this becomes an eigenvalue problem. An eigenvalue λ and
cor-responding eigenvector v of a square matrix A satisfy

Av = λv, (4.6)
or

(A − λI)v = 0. (4.7)

Eigenvalues have the interesting connections with the matrix,

n
X

tr(A) =aii = λ1 + λ2 + ... + λn, (4.8)

i=1
and
nY

det(A) =λi. (4.9)

i=1

For a symmetric square matrix, the two eigenvectors for two

distinct eigenvalues λi and λj are orthogonal vTv = 0.
4.1 Linear systems 37

Mathematically speaking, a linear system can be solved

in principle using the Cramer’s rule,
ui = det Ai ,i = 1, 2, ..., n, (4.10)
det A
where the matrix Ai is obtained by replacing the i-th column
by the column vector b. For three linear equations with three
unknown u1, u2 and u3,

a11u1 + a12u2 + a13u3 = b1,

a21u1 + a22u2 + a23u3 = b2,

a31u1 + a32u2 + a33u3 = b3, (4.11)
its solution vector is given by the following Cramer’s rule
a a a a
u1 = 1 b2 22 23 , u2 = 1 21 b2 23 ,
a a a a
b1
b3 a
32
12 33
13 a a
31
11 b1
b3 a
33
13

21 a22 ,
a
a a
u3 = 1 b2

a
31
11 32
12 a
b1 b3

where a a
12 13
a
11

= (4.12)
a a a
31 32 33
a a a
21 22 23 .

Though it is straightforward to extend to any dimensions in

theory, in practice this is not an easy task because the
calcula-tion of the determinant of a large matrix is not easy.
Though, the Cramer’s rule is good for theorem-proof, but it is
not good for numerical implementation. A far better method is
to use the inverse matrix.
−1
Finding the inverse A of a square n × n matrix A is not an
easy task either, especially when the size of the matrix is large
3
and it usually requires the algorithm complexity of O(n ). In fact,
many solution methods intend to avoid the calculations
38 Chapter 4. System of Linear Equations

−1
of finding the inverse A if possible. There are many ways of
solving the linear equations, but they fall into two categories:
direct algebraic methods and iteration methods. The former
intends to find the solution by elimination, decomposition of the
matrix, and substitutions, while the later involves certain
iterations to find the approximate solutions. The choice of these
methods depends on the characteristics of the matrix A, the
size of the problem, computational time, the type of problem,
and the required solution quality.

4.2 Gauss Elimination

The basic idea of Gauss elimination is to transform the square matrix into
a triangular matrix by elementary row operations, so that the simplified
triangular system can be solved by direct back substitution. For the linear
system
a a a
a21 22 23 ... 2n u2 b2
a a a ... a1n
11 12 13 u1 b1
.
. .. ..

an1 .
a
n2
a
n3 ... a
nn un
. = bn
. , (4.13)

the aim in the first step is to try to make all the coefficients in
the first column (a21, ..., an1) become zero except the first
element by elementary row operations. This is based on the
principle that a linear system will remain the same if its rows
are multiplied by some non-zero coefficients or any two rows
are interchanged or any two (or more) rows are combined
through addition and substraction.
To do this, we first divide the first equation by a11 (we can
always assume a11 6= 0, if not, we re-arrange the order of the
equations to achieve this). We now have
a1n b1
1 a12 a13 ... u2
a11 a11
a

a11 a11
2n b2

a21 a22 a23 ...

u1
. . = . (4.14)
.. ..

. .
.
a nn un bn

an1 an2 an3 ...

4.2 Gauss Elimination 39

Then multiplying the first row by −a 21 and adding it to the

second row, multiplying the first row by −ai1 and adding it to
the i-th row, we finally have
a a a
0 1
22 a11
a11
... 2n a11 n u2 b2 a11 a11 ... a11 a11

u
1
a12 a13 a1n b1
−. − a21a1 . = −. .
a21a12 21b1

.. ..

..
1a 12 1 1n
0 a ... a un
an1bn

n2 nn
an an a

a11
a11
a11

bn
−
− −
We then repeat the same procedure for the third row to the
n-th row, the final form of the linear system should be in the
following generic form
α α α
0 22 23 ... 2nu2 β2
α α α ... α
11 12 13 1n u1 β1
.. .
.. .. ..
.
0 0 0 ... α
nn un
. = βn
. , (4.15)

where α1j = a1j /a11, α2j = a2j − a1j a21/a11(j = 1, 2, ..., n),
..., β1 = b1/a11, β2 = b2 − a21b1/a11 and others. From the above
form, we see that un = βn/αnn because there is only one
unknown un in the n-th row. We can then use the back
substitution to obtain un−1 and up to u1. Therefore, we have
un = βn ,
α
nn
n
1
u = (β j X
i αii i− αij xj ), (4.16)
=i+1
where i = n − 1, n − 2, ..., 1. Obviously, in our present case,
α11 = ... = αnn = 1. Let us look at an example.

Example 4.1: For the linear system

3 2 −5 6u2 = 9,
−1 32 4u1 21
−2 1 0 5u3 12
4 5 6 0u 3

− −
4

−
40 Chapter 4. System of Linear Equations
we first divide the first row by a = 2, we have . 11

3 −2 −5 6u2 = 9
1 1/2 3/2 2u 21/2
1
−2 1 0 5u3 12
4 5 6 0u 3

− −
4

Multiplying the first row by 3 and subtracting it from the second

row, and carrying out similar row manipulations for the other rows, we
have
0 7/2 19/2 0 u2 45/7
1 −1/2 3/2 2 u1 21/2
0 9u3 = − 33 .
0 −3
0 3 12 8u 45

− −
4

− −
For the second row, we repeat this procedure again, we have
0 −1 19/7 0 u2 45/7
1 1/2 3/2 2 u1 21/2
0 9u3 = .
0 0
−3
141/7 8u
−33
450/7
0

−
4

−
−
After the same procedure for the third row, we have
−
0 1 19/7 0 u2 45/7
1 1/2 3/2 2 u1 21/2
0 3 u3 =
0 −1 −11 .
0 0 0 367/7u 1101/7
4

The fourth row gives that u4 = 3. Using the back substitution, we

have u3 = 2 from the third row. Similarly, we have u 2 = −1 and u1
= 1. So the solution is
u2 1
u1 1
u3 = − .
2
u 3
4

We have seen from the example, there are many float-point

calculations even for the simple system of three linear equa-
tions. In fact, the full Gauss elimination is computationally
extensive with an algorithmic complexity of O(2n3/3).
4.3 Gauss-Jordan Elimination 41

4.3 Gauss-Jordan Elimination

Gauss-Jordan elimination is a variant of Gauss elimination
which solves a linear system and, at the same time, can also
compute the inverse of a square matrix. The first step is to
formulate an augmented matrix from A, b and the unit matrix
I (with the same size of A). That is
|
a21 ... a2n b2 | 0 1 ... 0
a ... a 1 0 ... 0
11 1n |b1 |
|| .. ..
..

B = [A b I] = a n1 .
...
a
nn .
bn
0 . 0 ...
1 ,(4.17)
| |

where the notation A|b denotes the augmented form of two

matrices A and b. The aim is to reduce B to the following form
by elementary row reductions in the similar way as those carried out in
Gauss elimination.
1 ... 0| | 0 ... 0
a2 n
0 u2 a21
1 0 ... 0 u1 a110 ... 0
a1 n
. | | .
| | 1
. .
.
.
A− ],(4.18)
. . . = [I u
0 ... 1 u a n01 ... a nn0
n
0
| |
−1 0
where A = [a ij ] is the inverse. This is better demonstrated
by an example.

Example 4.2: In order to solve the following system

Au = 2 2 5 u2 = 2 = b,
1 2 3 u1 5
− 4
0 5u3 14
−
−

we first write it in an augmented form

1 2 3 | 5 |1 0 0
B= −2 2 5 | −2 | 0 1 0.

4 0 −5 | 14 | 0 0 1
42 Chapter 4. System of Linear Equations
By elementary row operations, this could be changed into

1 0 0 | 1 | 5 − 5 − 2

0 − 7 7 7
B = [I|u|A ] = 0 1 0 | 5 | −7 14 14 ,
1 5 17 11
4 4 3

0 1 2

| − | − −
0 7 7 7

which gives
u= , −1 10 17 11 .
5 A =
1 1 10 −10 −4
2 14
−8 8 6
−
− −
−1
We can see that both the solution u and the inverse A are
obtained in Gauss-Jordan elimination.
The Gauss-Jordan elimination is not quite stable numeri-
cally. In order to get better and stable schemes, common
prac-tice is to use pivoting. Basically, pivoting is a scaling
procedure by dividing all the elements in a row by the
element with the largest magnitude or norm. If necessary,
rows can be exchanged so the the largest element is moved
so that it becomes the lead-ing coefficient, especially on the
diagonal position. This makes all the scaled elements to be
in the range [−1, 1]. Thus, excep-tionally large numbers are
removed, which makes the scheme more numerically stable.
An important issue in both Gauss elimination and Gauss-
Jordan elimination is the non-zero requirement of leading co-
efficients such as a11 6= 0. Fora11, it is possible to re-arrange
the equations to achieve this requirement. However, there is
no guarantee that other coefficients such as a 22 − a21a12/a11
should be nonzero. If it is zero, there is a potential difficulty due to the
dividing by zero. In order to avoid this problem, we can use other
methods such as the pivoting method and LU decomposition.
4.4 LU Factorization 43

4.4 LU Factorization
Any square matrix A can be written as the product of two
triangular matrices in the form

A = LU, (4.19)

where L and U are the lower and upper triangular matrices,

respectively. A lower (upper) triangular matrix has elements only on the
diagonal and below (above). That is
β
β21 22 ... 0
β 0... 0
11
β β
.. .
. .

L= βn1
. n2 ... nn , (4.20)
and

.
.. − .
α ... α 1 α
11 1,n 1,n
U= 0.... αn 1,n 1 αn . 1,n . (4.21)
0... − − −

0 α
nn

The linear system Au = b can be written as two step

Au = (LU)u = L(Uu) = b, (4.22)

or

Uu = v, Lv = b, (4.23)
which are two linear systems with triangular matrices only,
and these systems can be solved by forward and back
substitutions. The solutions of vi are given by
1 i−1
b 1 X

v1 = β , vi = βii (bi − βij vj ), (4.24)

11
j=1

where i = 2, 3, ..., n. The final solutions ui are then given by

1 n
v n
u = , X
n αnn ui = αii
(v
i−
j
αij uj ), (4.25)
=i+1
44 Chapter 4. System of Linear Equations

where i = n − 1, ..., 1.
For triangular matrices such as L, there are some interesting
properties. The inverse of a lower (upper) triangular matrix is also a
lower (upper) triangular matrix. The determinant of the
triangular matrix is simply the product of its diagonal entries.
That is n
Y

det(L) = |L| = βii = β11β22...βnn. (4.26)

i=1
More interestingly, the eigenvalues of a triangular matrix are the
diagonal entries: β11, β22, ..., βnn. These properties are
convenient in determining the stability of an iteration scheme.
But there is another issue here and that is how to decom-
pose a square matrix A = [aij ] into L and U. As there are n(n
+ 1)/2 coefficients αij and n(n + 1)/2 coefficients βij , so there
2
are n + n unknowns. For the equation (4.19), we know that
2 2
it could provide only n equations (as there are only n
coefficients aij ). They are
i
β α =a ,
X

ik kj ij (i < j), (4.27)

k=1
j=i
X
β α =a , (i = j), (4.28)
ik kj ij
k=1
and
j
X
β α =a , (i > j), (4.29)
ik kj ij
k=1
which again form another system of n equations.
2 2
As n + n > n , there are n free coefficients. Therefore,
the factorization or decomposition is not uniquely
determined. We have to impose some extra conditions.
Fortunately, we can always set either α ii = 1 or βii = 1 where i
= 1, 2, ..., n. If we set β ii = 1, we can use the Crout’s
algorithm to determine αij and βij . We have the coefficients
for the upper triangular matrix
i−1
X
α =a β α , (4.30)
ij ij − ik kj
k=1
4.5 Iteration Methods 45

for (i = 1, 2, ..., j) and j = 1, 2, ..., n. For the lower triangular

matrix, we have
1 j−1
β = β α ),
X

ij α (aij − ik kj
(4.31)
jj
k=1

for i = j + 1, j + 2, ..., n.
The same issue appears again, that is, all the leading
coef-ficients αii must be non-zero. For sparse matrices with
many zero entries, this often causes some significant
problems nu-merically. Better methods such as iteration
methods should be used in this case.

4.5 Iteration Methods

−1
For a linear system Au = b, the solution u = A b gener-ally
involves the inversion of a large matrix. The direct in-version
becomes impractical if the matrix is very large (say, if n >
1000000). Many efficient algorithms have been developed
for solving such systems. Jacobi and Gauss-Seidel iteration
methods are just two examples.

4.5.1 Jacobi Iteration Method

The basic idea of the Jacobi-type iteration method is to de-
compose the n × n square matrix A into three simple matrices
A = D + L + U, (4.32)

where D is a diagonal matrix. L and U are the strictly lower

and upper triangular matrices, respectively. Here the ‘strict’
means that the lower (or upper) triangular matrices do not
include the diagonal elements. That is say, all the diagonal
elements of the triangular matrices are zeros.
It is worth pointing out that here the triangular matrices L
and U are different from those in the LU decomposition
where it requires a matrix product. In comparison with the LU
decomposition where LU = A, we have used the simple
46 Chapter 4. System of Linear Equations

additions here and this makes the decomposition an easier task.

Using this decomposition, the linear system Au = b becomes

Au = (D + L + U)u = b, (4.33)
which can be written as the iteration procedure
(n+1) (n) (4.34)
Du = b − (L + U)u .
This can be used to calculate the next approximate solution
u(n+1) from current estimate u(n). As the inverse of any diag-
onal matrix D =diag[dii] is easy, we have
(n) (4.35)
[b − (L + U)u ].
u(n+1) = D−1
Writing in terms of the elements, we have
(n+1) 1 (n)
X
u = 6
i dii [bi − a u
ij j ], (4.36)
j =i

where dii = aii are the diagonal elements only.

(0)
This iteration usually starts from an initial guess u (usu-
(0)
ally, u = 0). However, this iteration scheme is only stable
under the conditions that the square matrix is strictly diago-
nally dominant. That is to require that
j X
6
n

|aii| > |aij |, (4.37)

=1,j =i

for all i = 1, 2, ..., n.

In order to show how the iteration works, let us at look at
an example by solving the following linear system
5u1 + u2 − 2u3 = 5,

u1 + 4u2 = −10,
(4.38)
2u1 + 2u2 − 7u3 = −9.
We know its exact solution is = 3
u = u2 . (4.39)
u1 2
u3 − 1
4.5 Iteration Methods 47

Now let us solve the system by the Jacobi-type iteration method.

Example 4.3: We first write the above simple system as the

compact matrix form Au = b or
1 4 0u2 = 10 .
5 1 −2 u1 5
2 2 7 u3 −9

− −
Then let us decompose the matrix A as

5 1 −2

2 2 −7
= 0 4 0 + 1 0 0 + 0 0 0 .
5 0 0 0 0 0 0 1 −2
0 0 7 2 2 0 0 0 0

The inverse of D is simply .

−1
D = 0 1/4 0
1/5 0 0
0 0 − 1/7

The Jacobi-type iteration formula is

(n+1) −1 (n)
u = D [b − (L + U)u ]
or
u2 =0 1/4 0 101 0 0 u2 .
u1 1/5 0 0 " 5 − 0 1 −2 u1 #
u3 n+1 0 0 − 9 2 2 0 u3

− −

(0) T
If we start from the initial guess u = 0 0 0 , we have
u(1) 2.5 , u(2) 2.7500 , u(3) 3.0036 ,

1 2.0143 1.8929
≈ − ≈
1 .2857 − ≈ 0.8571
1.0755
 48 Chapter 4. System of Linear Equations
2 3.0077 .
u(4) 2.9732, u(5)
.0309 1.9820
≈
− 0.9684
−
≈ 1.0165

We can see that after 5 iterations, the approximate solution is

T
quite near the true solution u = 2 −3 1 .

This is an issue here. If we interchange the second row

(equation) and the third row (equation), then the new
diagonal matrix is
5 0 0
02 0 ,

0 0 0
which has no inverse as it is singular. This means the order
of the equations is important to ensure that the matrix is
diago-nally dominant.
Furthermore, if we interchange the first equation (row) and second
equation (row), we have an equivalent system
5 1 2u2 = −5 .
1 4 0u1 10
2 2 − 7u3 − 9
−

Now the new decomposition becomes

1 4 0

2 2 −7
= 0 1 0 + 5 0 0 + 0 0 −2 ,
1 0 0 0 0 0 0 4 0
0 0 72 2 00 0 0
−

which gives the following iteration formula 2u2

u2 = 0 1 0 −5 5 0 .
u1 1 0 0 " 10 0 4 0 u1 #
u3 n+10 0 1 9 − 2 2 − u3
7 0 n
− −
4.5 Iteration Methods T 49
0
(0) 0 again, we have , ...
Starting from u = 0
−
u(1) = 5 , u(2) = 57 .5714 , u(3) = 154.71
−10 30 −240.28
1.2857 0.1429 9.16

−
We can see that it diverges. So what is the problem? How can
the order of the equation affect the results so significantly?
There are two important criteria for the iteration to con-
−1
verge correctly, and they are: the inverse of D must exist and
the spectral radius of the right matrix must be less than 1. The
−1
first condition is obvious, if D does not exist (say, when any of
the diagonal elements is zero), then we cannot carry out the
iteration process at all. The second condition requires
−1 −1 (4.40)
ρ(D ) ≤ 1,ρ[D (L + U)] ≤ 1,
where ρ(A) is the spectral radius of the matrix A. From the
diagonal matrix D, its largest absolute eigenvalue is 1. So
ρ(D−1) = max(|λi|) = 1 seems to be no problem. How about
the following matrix?
N = D−1(L + U) = 5 0 −2 . (4.41)
0 4 0
2/7 − 2/7 0

−
The three eigenvalues of N are λ i = 4.590, −4.479, −0.111.
So its spectral radius is ρ(N) = max(|λ i|) = 4.59 > 1. The
itera-tion scheme will diverge.
If we revisit our earlier example, we have
/5 0 0 1 1 −1
−1
1 −1
D = 0
0
1/4
0
0 , eig(D ) =
1/7 5
, ,
4 7
, (4.42)
and −

−1 −
N = D (L + U) = 1/4 0 0 , (4.43)
0 1/5 2/5
2/7 − 2/7 0

−
50 Chapter 4. System of Linear Equations
whose eigenvalues are

λ1 = 0.4739, λ2,3 = −0.2369 ± 0.0644i. (4.44)

So we have
−1 ρ(N) = 0.4739 < 1. (4.45)
ρ(D ) = 1/4 < 1,

That is why the earlier iteration procedure is convergent.

4.5.2 Gauss-Seidel Iteration

(n+1)
In the Jacobi-type iterations, we have to store both u
(n) ( n)
and u as we have to use all the u j values to compute the
value at the next level t = n + 1, this means that we cannot
use the running update when the new approximate has just
been computed
u
(n+1)
→u ,
(n) (j = 1, 2, ...).
j j

If the vector size u is large (it usually is), then we can devise
other iteration procedure to save memory using the running
update. So only one vector storage is needed.
The Gauss-Seidel iteration procedure is such an iteration
procedure to use the running update and also provides an
ef-ficient way of solving the linear matrix equation Au = b. It
uses the same decomposition as the Jacobi-type iteration by
splitting A into
A = L + D + U, (4.46)

but the difference from the Jacobi method is that we use L +

D instead of D for the inverse so that the running update is
possible. The n-th step iteration is updated by

(n+1) (n) (4.47)

(L + D)u = b − Uu ,
or
(n+1) −1 (n) (4.48)
u = (L + D) [b − Uu ].
This procedure, starting from an initial vector u (0), stops if a
prescribed criterion is reached.
4.6 Nonlinear Equation 51

It is worth pointing out that the Gauss-Seidel iteration re-

quires the same criteria of convergence as the Jacobi-type
iter-ation method. The inverse of the matrix must exist, and
the largest spectral radius must be less than 1.

4.5.3 Relaxation Method

The above Gauss-Seidel iteration method is still slow, and
the relaxation method provides a more efficient iteration
procedure. A popular method is the successive over-
relaxation method which consists of two steps

(n) (n) (4.49)

v = (L + D + U)u − b,
and
(4.50)
u(n+1) = u(n) − ω(L + D)−1v(n),
where 0 < ω < 2 is the overrelaxation parameter. If we
combine the above equations and re-arrange, we have

(n+1) (n) (n) (4.51)

u = (1 − ω)u + ωu˜ ,
(n) −1 (n)
where u˜ = (L + D) (b − Uu ) is the standard Gauss-
Seidel procedure. Therefore, this method is essentially the
weighted average between the previous ieration and the
succes-sive Gauss-Seidel iteration. Clearly, if ω = 1, then it
reduces to the standard Gauss-Seidel iteration method.
Broadly speaking, a small value of 0 < ω < 1 corresponds
to under-relaxation with slower convergence while 1 < ω < 2
leads to over-relaxation and faster convergence. It has been
proved theoretically that the scheme will not converge if ω <
0 or ω > 2.

4.6 Nonlinear Equation

Sometimes, the algebraic equations we meet are nonlinear,
and direct inversion is not the best technique. In this case,
more elaborate techniques should be used.
52 Chapter 4. System of Linear Equations

4.6.1 Simple Iterations

The nonlinear algebraic equation

A(u)u = b(u), or F(u) = A(u)u − b(u) = 0, (4.52)

can be solved using a simple iteration technique
(n) (n+1) (n) n = 0, 1, 2, ... (4.53)
A(u )u = b(u ),

(n+1) (n)
until ku − u k is sufficiently small. Iterations require a
(0)
starting vector u . This method is also referred to as the
successive substitution.
If this simple method does not work, the relaxation method
can be used. The relaxation technique first gives a tentative
new approximation u∗ from A(u(n))u∗ = b(u(n)), then we use
(n)
u
(n+1)
= ωu∗ + (1 − ω)u , ω ∈ (0, 1], (4.54)

where ω is a prescribed relaxation parameter.

4.6.2 Newton-Raphson Method

The nonlinear equation (4.52) can also be solved using the
Newton-Raphson procedure. We approximate F(u) by a
(n)
linear function R(u; u ) in the vicinity of an existing
(n)
approximation u to u:
(n) (n) (n) (n)
R(u; u ) = F(u ) + J(u )(u −u ), J(u) = rF, (4.55)
where J is the Jacobian of F(u) = (F1, F2, ..., FM )T . For u =
(u1, u2, ..., uM )T , we have
Jij = ∂Fi . (4.56)

∂uj
To find the next approximation u(n+1) from R(u(n+1); u(n)) = 0,
one has to solve a linear system with J as the coefficient matrix
(4.57)
u(n+1) = u(n) − J−1F(u(n)),
(n+1) (n)
under a given termination criterion ku −u k≤ .
 Part II

Mathematical Optimization
Chapter 5

Unconstrained
Optimization

5.1 Univariate Functions

The simplest optimization problem without any constraints is

probably the search of the maxima or minima of a univariate
function f (x). For unconstrained optimization problems, the
optimality occurs at the critical points given by the stationary
0
condition f (x) = 0. However, this stationary condition is just
0
a necessary condition, but it is not sufficient. If f (x∗) = 0
and
00 0
f (x∗) > 0, it is a local minimum. Conversely, if f (x∗) = 0 and f
00 0
(x∗) < 0, then it is a local maximum. However, if f (x∗) =
00
0 but f (x) is indefinite (both positive and negative) when x →
x∗, then x∗ corresponds to a saddle point. For example,
3 0
f (x) = x has a saddle point x∗ = 0 because f (0) = 0 but f
00 00 00
changes sign from f (0+) > 0 to f (0−) < 0.
Example 5.1: For example, in order to find the maximum or
minimum of an univariate function f (x)

f (x) = xe
−x2
, −∞ < x < ∞ , (5.1)

we have to find first the stationary point x∗ when the first

deriva-
55
56 Chapter 5. Unconstrained Optimization
0
tive f (x) is zero. That is

df (x ) 2 2
∗ −x 2 −x (5.2)
dx =e ∗ − 2x∗ e ∗ = 0.
∗
2
Since exp(−x∗ ) 6= 0, we have
√2
. (5.3)
x∗ = ± 2
From the basic calculus we knowthat the maximum requires
√
00 00
f (x∗) ≤ 0 while minimum requires f (x∗) ≥ 0. At = 2/2,
x∗
we have
00 2 −x 2 √
f (x∗) = (4x∗ − 6)x∗e ∗ = 2e−1/2 < 0, (5.4)
−2
1 −1/2
so this point corresponds to a maximum f (x ∗) = e . Simi-
00 √
2

larly, x = −1 √ 2/2, f (x ) = 2 2e−1/2 > 0, we have a minimum

∗ ∗
f (x∗) = − 2 e−1/2.

Since a maximum of a function f (x) can be converted into a

minimum of A − f (x) where A is usually a large positive
number (though A = 0 will do). For example, we know the
−x
maximum of f (x) = e 2 , x ∈ (−∞, ∞) is 1 at x∗ = 0. This
problem can be converted to a minimum problem −f (x). For
this reason, the optimization problems can be either
expressed as minima or maxima depending the context and
convenience of finding the solutions.

5.2 Multivariate Functions

T
For functions of multivariate x = (x 1, ..., xn) , the optimiza-
tion can be expressed in the same way as the univariate
opti-mization problems.
minimize/maximize f (x). (5.5)
x <n
For a quadratic function f (x) it can be expanded using
Taylor series about a point x = x∗ so that x = x∗ + u
1 2 T
 2 u (5.6)
f (x+ u) = f (x∗)+ uG1(x∗)+ G2(x∗+ u)u+...,
5.2 Multivariate Functions 57

where G1 and G2 are the gradient vector and the Hessian

ma-trix, respectively. is a small parameter, and u is a vector.
1 T T
For example, f (x) = 2 x Ax + k x + b, we have
T 1 2 T
k+ 2 u Au + ..., (5.7)
f (x∗ + u) = f (x∗) + u
where 1
T T (5.8)
f (x∗) = 2 Ax∗ + k x∗ + b.
x∗
Thus, in order to study the local behaviour of a quadratic func-
tion, we only need to study G1 and G2. In addition, for sim-
plicity, we can take b = 0 as it is a constant vector anyway.
At a stationary point x∗, the first derivatives are zero or
G1(x∗) = 0, therefore, equation (5.6 becomes
12uT G u. (5.9)
f (x∗ + u) ≈ f (x∗) + 2
2
If G2 = A, then
Av = λv (5.10)
forms an eigenvalue problem. For an n ×n matrix A, there will
be n eigenvalues λj (j = 1, ..., n) with n eigenvectors v. As we
have seen earlier that A is symmetric, these eigenvectors are
orthonormal. That is,
T (5.11)
vi vj = δij .
Near any stationary point x∗, if we take uj = vj as the local
coordinate systems, we then have

1 2

2 λj , (5.12)
f (x∗ + vj ) = f (x∗) +
which means that the variations of f (x), when x moves away
from the stationary point x∗ along the direction vj , are char-
acterised by the eigenvalues. If λj > 0, | | > 0 will leads to
| f | = |f (x) − f (x∗)| > 0. In other words, f (x) will increase as
| | increases. Conversely, if λj < 0, f (x) will decrease as | | > 0
increases. Obviously, in the special case λj = 0, the
58 Chapter 5. Unconstrained Optimization

function f (x) will remain constant along the corresponding di-

rection of vj .

Example 5.2: We know that the function f (x, y) = xy has a saddle point at
(0, 0). It increases along the x = y direction and decreases along x = −y
T T
direction. From the above analysis, we know that x ∗ = (x∗, y∗) = (0, 0)
and f (x∗, y∗) = 0. We now have

|
1
| f (x∗ + u) ≈ f (x∗) + 2

where
A =f (x ) = =
∂2f
2
∂ f
∂2f
∂ f
2 !
2 0 1
∂x2 ∂x∂y
r ∗ .
∂x∂y ∂y
2 1 0
The eigenvalue problem is simply
Av = λj vj , (j = 1, 2),
or j
−1 λj = 0,

λ −
1

whose solutions are

λj = ±1.
For λ1 = 1, the corresponding eigenvector is
√
v1 = 2 /2
!
.

2/2
√

Similarly, for λ2 = −1, the eigenvector is !

√

2/2
v = √ .
2 − 2/2
Since A is symmetric, v1 and v2 are orthonormal. Indeed this
is the case because kv1k = kv2k = 1 and
√2 √2 √2 √2
T
v v = × + × (− ) = 0.
1 2
2 2 2 2
5.3 Gradient-Based Methods 59
Thus, we have

1 2

f ( vj ) = λj , (j = 1, 2). (5.13)
2
As λ1 = 1 is positive, f increases along the direction v1 =
√
2 (1 1)T which is indeed along the line x = y. Similarly, for
2 √
2 T
λ2 = −1, f will decrease along v2 = 2 (1 −1) which is exactly
along the line x = −y. As there is no zero eigenvalue, the
function will not remain constant in the region around (0, 0).

5.3 Gradient-Based Methods

The gradient-based methods are iterative methods that
exten-sively use the information of the gradient of the
function during iterations. For the minimization of a function f
(x), the essence of this class of method is

(n+1) (n) (n) (5.14)

x =x + αg(rf, x ),

where α is the step size which can vary during iterations. g(rf
) is a function of the gradient rf . Different methods use
(n)
different form of g(rf, x ).

5.3.1 Newton’s Method

We know that Newton’s method is a popular iterative method
for finding the zeros of a nonlinear univariate function of f (x) on
the interval [a, b]. It can be modified for solving optimization
problems because it is equivalent to find the zero of the first
0
derivative f (x) once the objective function f (x) is given.
For a given function f (x) which are continuously differ-
entiable, we have the Taylor expansion about a known point
x = xn (with x = x − xn)

T 1 T 2
f (x) = f (xn)+(rf (xn)) x+ 2 x r f (xn)Δx+..., (5.15)
60 Chapter 5. Unconstrained Optimization
which is minimized near a critical point when x is the solution

of the following linear equation

2 (5.16)
rf (xn) + r f (xn)Δx = 0.

This leads to −1
x = xn − G rf (xn),
(5.17)
2
where G = r f (xn) is the Hessian matrix. If the iteration
(0)
procedure starts from the initial vector x (usually taken to
be a guessed point in the domain), then Newton’s iteration
formula for the nth iteration is
(5.18)
x(n+1) = x(n) − G−1(x(n))f (x(n)).
It is worth pointing out that if f (x) is quadratic, then the
solution can exactly be found in a single step. However, this
method is not efficient for non-quadratic functions.
In order to speed up the convergence, we can use a smaller
step size α ∈ (0, 1] so that we have modified Newton’s method
(5.19)
x(n+1) = x(n) − αG−1(x(n))f (x(n)).
It sometimes might be time-consuming to calculate the
Hes-sian matrix for second derivatives. A good alternative is
−1
to use an identity matrix G = I , and we have the quasi-
Newton method
(n+1) (n) (n) (5.20)
x = x − αI rf (x ),
which is essentially the steepest descent method.

5.3.2 Steepest Descent Method

The essence of this method is to find the lowest possible ob-
(n)
jective function f (x) from the current point x . From the
(n)
Taylor expansion of f (x) about x , we have
(n+1) (n) (n) (n) T
f (x ) = f (x + s) ≈ f (x + (rf (x )) s, (5.21)
(n+1) (n)
where s = x −x is the increment vector. Since we try to find a
lower (better) approximation to the objective function,
5.3 Gradient-Based Methods 61

it requires that the second term on the right hand is negative.

That is
(n) (n) T (5.22)
f (x + s) − f (x ) = (rf ) s < 0.
T
From vector analysis, we know the inner product u v of two
vectors u and v is largest when they are parallel but in opposite
T
directions. Therefore, (rf ) s becomes the smallest when
(n) (5.23)
s = −αrf (x ),
where α > 0 is the step size. This the case when the direc-
tion s is along the steepest descent in the negative gradient
direction. As we seen earlier, this method is a quasi-Newton
method.
The choice of the step size α is very important. A very
small step size means slow movement towards the local
mini-mum, while a large step may overshoot and
subsequently makes it move far away from the local
(n)
minimum. Therefore, the step size α = α should be
different at each iteration step and should be chosen so that
(n+1) (n) (n)
it minimizes the objective func-tion f (x ) = f (x , α ).
Therefore, the steepest descent method can be written as
(n+1) (n) (n) (n) T (n)
f (x ) = f (x ) − α (rf (x )) rf (x ). (5.24)

In each iteration, the gradient and the step size will be calcu-
lated. Again, a good initial guess of both the starting point
and the step size is useful.

Example 5.3: Let us minimize the function

2 2
f (x1, x2) = 10x 1 + 5x1x2 + 10(x2 − 3) ,
where
(x1, x2) = [−10, 10] × [−15, 15],
using the steepest descent method starting with the initial
(0) T
x = (10, 15) . We know that the gradient
T
rf = (20x1 + 5x2, 5x1 + 20x2 − 60) ,
62 Chapter 5. Unconstrained Optimization

therefore
(0) T
rf (x ) = (275, 290) .
In the first iteration, we have
!
(1) (0)
x =x − α0 275
290 .

(1)
The step size α0 should be chosen such that f (x ) is at the
minimum, which means that
2
f (α0) = 10(10 − 275α0)
2
+5(10 − 275α0)(15 − 290α0) + 10(12 − 290α0) ,
should be minimized. This becomes an optimization problem for
a single independent variable α0. All the techniques for
univariate optimization problems such as Newton’s method can
be used to find α0. We can also obtain the solution by setting
df
= −159725 + 3992000α = 0,
dα0 0

whose solution is α0 ≈ 0.04001.

At the second step, we have
!
(1) T (2) (1) −3.078
rf (x ) = (−3.078, 2.919) , x =x − α1 .
2.919
The minimization of f (α1) gives α1 ≈ 0.066, and the new
location of the steepest descent is
(2) T
x ≈ (−0.797, 3.202) .
At the third iteration, we have
!
(2) T (3) (2) 0.060
rf (x ) = (0.060, 0.064) , x =x − α2 . 0.064

The minimization of f (α2) leads to α2 ≈ 0.040, and

thus x(3) ≈ (−0.8000299, 3.20029)T .
5.3 Gradient-Based Methods 63

Then, the iterations continue until a prescribed tolerance is met.

From calculus, we know that we can set the first partial
deriva-tives equal to zero

∂f = 20x + 5x = 0, ∂f = 5x + 20x − 60 = 0,
1 2 1 2
∂x1 ∂x2

we know that the minimum occurs exactly at

T T
x∗ = (−4/5, 16/5) = (−0.8, 3.2) .

The steepest descent method gives almost the exact

solution after only 3 iterations.
In finding the step size αn in the above steepest descent
method, we have used the stationary condition df (αn)/dαn =
0. Well, you may say that if we use this stationary condition for f
(α0), why not use the same method to get the minimum point of
f (x) in the first place. There are two reasons here. The first
reason is that this is a simple example for demonstrating how
the steepest descent method works. The second reason is
that even for complicated multiple variable f (x 1, ..., xp) (say
p = 500), then f (α n) at any step n is still an univariate
function, and the optimization of such f (αn) is much simpler
compared with the original multivariate problem.
From our example, we know that the convergence from
the second iteration to the third iteration is slow. In fact, the
steepest descent is typically slow once the local minimization
is near. This is because near the local minimization the
gradient is nearly zero, and thus the rate of descent is also
slow. If high accuracy is need near the local minimum, other
local search methods should be used.
It is worth pointing out that there are many variations of the
steepest descent methods. If the optimization is to find the
maximum, then this method becomes the hill-climbing method
because the aim is to climb up the hill to the highest peak.
64 Chapter 5. Unconstrained Optimization

5.4 Hooke-Jeeves Pattern Search

Many search algorithms such as the steepest descent method
experience slow convergence near the local minimum. They are
also memoryless because the pass information is not used to
produce accelerated move. The only information they use is the
(n)
current location x , gradient and value of the objective itself at
step n. If the past information such as the steps at n−1 and n is
properly used to generate new move at step n+ 1, it may speed
up the convergence. The Hooke-Jeeves pattern search method
is one of such methods that incorporate the past history of
iterations in producing a new search direction.
The Hooke-Jeeves pattern search method consists of two
moves: exploratory move and pattern move. The exploratory
moves explore the local behaviour and information of the ob-
jective function so as to identify any potential sloping valleys if
they exist. For any given step size (each coordinate direction
can have different increment) i(i = 1, 2, ..., p), exploration
movement performs from an initial starting point along each
coordinate direction by increasing or decreasing ± i, if the new
value of the objective function does not increase (for mini-
( n) ( n−1)
mization problem), that is f (x i ) ≤ f (x i ), the exploratory
move is considered as successful. If it is not successful, then
try a step in the opposite direction, and the result is updated
only if it is successful. When all the p coordinates have been
(n)
explored, the resulting point forms a base point x .
(n)
The pattern move intends to move the current base x
(n) (n−1)
along the base line (x −x ) from the previous
(historical) base point to the current base point. The move is
carried out by the following formula
(5.25)
x(n+1) = x(n) + [x(n) − x(n−1)].
Then x(n+1) forms a new temporary base point for further new
exploratory moves. If the pattern move produces improvement
(lower value of f (x)), the new base point x (n+1) is successfully
updated. If the pattern move does not lead to improvement or lower
value of the objective function, then the pattern move
5.4 Hooke-Jeeves Pattern Search 65
Hooke-Jeeves Pattern Search Algorithm

begin
T
Objective function f (x), x = (x1, ..., xp )
(0)
Initialize starting point x and increments i (i = 1, ..., p)
Initialize step reduction factor γ > 1 and tolerance > 0
while ( ||xn+1 − xn || ≥ )
for all i = 1 to p, by x
Perform exploratory search i ±i (n)
( n+1) )
Update until successful f (x ) ≤ f (x
If the move fails, try again using i = i /γ
end (for)
Perform pattern move:
(n+1) (n) (n−1)
(n)
−
x
x =x + (x (n +1) )
Update new base point x
n=n+1
end (while)
end

Figure 5.1: Hooke-Jeeves pattern search algorithm.

(n)
is discarded and new search starts from x , and new search
moves should use smaller step size by reducing increments
Di/γ when γ > 1 is the step reduction factor. Iterations continue
until the prescribed tolerance is met. The algorithm is sum-
marised in the pseudo code shown in Fig. 5.1.

Example 5.4: Let us minimize

2 2
f (x) = x 1 + 2x1x2 + 3x 2,

using the Hooke-Jeeves pattern search. Suppose we start

(0) T
from x = (2, 2) with an initial step size = 1 for both
(0)
coordinates x1 and x2. We know that f (x ) = 24.
(1) T
First, the exploratory move in x1 with x = (2 + 1, 2)
(1)
produces f (x ) = 33 > 24. So this move is not successful, we
(1) T
discard it, and try the opposite direction x = (2 − 1, 2) . It
(1)
gives f (x ) = 17 < 24, and it is a good move. So we keep it
and there is no need to reduce the step size = 1. Now we try
66 Chapter 5. Unconstrained Optimization

(1) T
the other coordinate along x2, we know that x = (1, 2 − 1)
(1)
is a good move as it gives f (x ) = 6 < 24. Therefore, the
(1) T
base point is x = (1, 1) .
(2) (1)
We then perform the pattern move by using x =x +
(1) (0)
(x − x ), we have
! ! !
(2) 1 2 0
x =2 1 − 2 = 0 .

(2)
This pattern move produces f (x ) = 0 and it is a successful
move. This is indeed the optimal solution as the minimum
T
occurs exactly at x∗ = (0, 0) .
Chapter 6

Linear Mathematical
Programming

6.1 Linear Programming

Linear programming is a powerful mathematical modelling
tech-nique which is widely used in business planning,
engineering design, oil industry, and many other optimization
applications. The basic idea in linear programming is to find
the maximum or minimum under linear constraints.
For example, an Internet service provider (ISP) can pro-vide
two different services x1 and x2. The first service is, say, the
fixed monthly rate with limited download limits and band-width,
while the second service is the higher rate with no down-load
limit. The profit of the first service is αx 1 while the sec-ond is
βx2, though the profit of the second product is higher
β > α > 0, so the total profit is

P (x) = αx1 + βx2,β/α > 1, (6.1)

which is the objective function because the aim of the ISP

company is to increase the profit as much as possible. Suppose
the provided service is limited by the total bandwidth of the ISP
company, thus at most n1 = 16 (in 1000 units) of the first and at
most n2 = 10 (in 1000 units) of the second can be

67
68 Chapter 6. Linear Mathematical Programming

provided per unit of time, say, each day. Therefore, we have

x1 ≤ n1,x2 ≤ n2. (6.2)

If the management of each of the two service packages take

the same staff time, so that a maximum of n = 20 (in 1000
units) can be maintained, which means

x1 + x2 ≤ n. (6.3)

The additional constraints are that both x1 and x2 must be

non-negative since negative numbers are unrealistic. We
now have the following constraints

0 ≤ x1 ≤ n1,0 ≤ x2 ≤ n2. (6.4)

The problem now is to find the best x1 and x2 so that the profit
P is a maximum. Mathematically, we have
maximize P (x1, x2) = αx1 + βx2,
2
(x1,x2) N
subject to x + x ≤ n,
1 2
0 ≤x ≤n ,0≤x ≤n . (6.5)
1 1 2 2

Example 6.1: The feasible solutions to this problem can be

graphically represented as the inside region of the polygon
OABCD as shown in Fig. 6.1.
As the aim is to maximize the profit P , thus the optimal
solution is at the extreme point B with (n − n 2, n2) and P =
α(n−n2)+βn2. For example, if α = 2, β = 3, n 1 = 16, n2 = 10,
and n = 20, then the optimal solution is x 1 = n − n2 = 10 and
x2 = n2 = 10 with the total profit P = 2×(20−10)+3×10 = 50
thousand pounds.
Since the solution (x1 and x2) must be integers, an
interesting thing is that the solution is independent of β/α if and
only if β/α > 1. However, the profit P does depend on α and β.
The number of feasible solutions are infinite if x 1 and x2 are
real numbers. Even for x1, x2 ∈ N are integers, the number of
6.1 Linear Programming 69

x2
6
infeasible region
A (n−n2, n2)

(0, n2) B
P P = αx1+ βx2
feasible solutions C

D -

0 (n1, 0) (n, 0) x1
Figure 6.1: Schematic representation of linear programming. If
α = 2, β = 3, n1 = 16, n2 = 10 and n = 20, then the optimal
solution is at B(10, 10).

feasible solutions are quite large. Therefore, there is a need

to use a systematic method to find the optimal solution.
In order to find the best solution, we first plot out all the
constraints as straight lines, the feasible solution which satis-
fying all the constraints form the inside region of the polygon
OABCD. The vertices of the polygon form the set of the ex-
treme points. Then, we plot the objective function P as a
family of parallel lines (shown as dashed lines) so as to find
the maximum value of P . Obviously, the highest value of P
corresponds to the case when the objective line goes
through the extreme point B. Therefore, x 1 = n − n2 and x2 =
n2 at the point B is the best solution.
The current example is simple because it has only two deci-
sion variables and three constraints, which can be solved easily
using a graphic approach. Many real-world problems involve
hundreds of or thousands of decisional variables, and graphic
approach is simply incapable of dealing with such complexity.
Therefore, a formal approach is needed. Many powerful meth-
ods have been developed over the last several decades, and
the most popular method is probably the simplex method.
70 Chapter 6. Linear Mathematical Programming

6.2 Simplex Method

The simplex method was introduced by George Dantzig in
1947. The simplex method essentially works in the following
way: For a given linear optimization problem such as the ex-
ample of the ISP service we discussed earlier, it assumes
that all the extreme points are known. If the extreme points
are not known, the first step is to determine these extreme
points or to check whether there any feasible solutions. With
known extreme points, it is easy to test whether an extreme
point is optimal or not using the algebraic relationship and
the objec-tive function. If the test for optimality is not passed,
then move to an adjacent extreme point to do the same test.
This process stops until an optimal extreme point is found or
the unbounded case occurs.

6.2.1 Basic Procedure

Mathematically, the simplex method first transforms the con-
straint inequalities into equalities by using slack variables.
To convert an inequality such as
5x1 + 6x2 ≤ 20, (6.6)
we can use a new variable x3 or s1 = 20 − 5x1 − 6x2 so that
the original inequality becomes an equality

5x1 + 6x2 + s1 = 20, (6.7)

with an auxiliary non-negativeness condition

s1 ≥ 0. (6.8)

Such a variable is referred to as a slack variable.

Thus, the inequalities in our example
x1 + x2 ≤ n, 0 ≤ x1 ≤ n1, 0 ≤ x2 ≤ n2, (6.9)

can be written, using three slack variables s1, s2, s3, as the fol-
lowing equalities
x1 + x2 + s1 = n, (6.10)
6.2 Simplex Method 71
x1 + s2 = n1, x2 + s3 = n2, (6.11)

and

xi ≥ 0 (i = 1, 2), sj ≥ 0(j = 1, 2, 3). (6.12)

The original problem (6.5) becomes
maximize P (x) = αx + βx + 0s + 0s + 0s ,
1 2 1 2 3
5
x<
1 1 1 0 0 x2 n
subject to 1 00 1 0 x1 = n1 ,
0 1 0 0 1 s2 n2

s
3

xi ≥ 0,
(i = 1, 2, ..., 5), (6.13)
which has two control variables (x 1, x2) and three slack vari-
ables x3 = s1, x4 = s2, x5 = s3.
In general, a linear programming problem can be written
as the standard form
maximize p T
X

x <nf (x) = Z =αixi = α x,

i=1

subject to Ax = b, xi ≥ 0 (i = 1, ..., p), (6.14)

T
where A is an q × p matrix, b = (b1, ..., bq ) , and
T T (6.15)
x = [xp xs] = (x1, ..., xm, s1, ..., sp−m) .

This problem has p variables, and q equalities and all p vari-

ables are non-negative. In the standard form, all constraints are
expressed as equalities and all variables including slack vari-
ables are non-negative. Suppose q = 500, even the simplest
integer equalities xi + xj = 1 where i, j = 1, 2, ..., 500, would give
500
a huge number of combinations 2 . Thus the number of basic
500
feasible solutions or extreme points will be the order of 2 ≈3
150
×10 , which is larger than the number of particles in the
whole universe. This huge number of extreme points neces-
sitates a systematic and efficient search method. The simplex
72 Chapter 6. Linear Mathematical Programming

method is a powerful method to carry out such mathematical

programming task.
A basic solution to the linear system Ax = b of q linear
equations in p variables in the standard form is usually obtained
by setting p−q variables equal to zero, and subsequently
solving the resulting q × q linear system to get a unique solution
of the remaining q variables. The q variables (that are not
bound to zero) are called the basic variables of the basic
solution. The p − q variables at zero are called non-basic
variables. Any basic solution to this linear system is referred to
as a basic feasible solution (BFS) if all its variables are non-
negative. The important property of the basic feasible solutions
is that there is a unique corner point (extreme point) for each
basic feasible solution, and there is at least one basic feasible
solution for each corner or extreme point. These corner or
extreme points are points on the intersection of two adjacent
boundary lines such as A and B in Fig. 6.1. Two basic feasible
solutions are said to be adjacent if they have q −1 basic
variables in common in the standard form.

6.2.2 Augmented Form

The linear optimization problem is usually converted into the following
standard augmented form or the canonical form
0 −A ! x! = b! , (6.16)
1 Z 0
αT

with the objective to maximize Z . In this canonical form, all the

constraints are expressed as equalities for all non-negative variables. All
the right hand sides for all constraints are also non-negative, and each
constraint equation has a single ba-sic variable. The intention of writing
in this canonical form is to identify basic feasible solutions, and move
from one ba-sic feasible solution to another via a so-called pivot
operation. Geometrically speaking, this means to find all the corner or
ex-treme points first, then evaluate the objective function by going
6.2 Simplex Method 73

through the extreme points so as to determine if the current

basic feasible solution can be improved or not.
In the framework of the canonical form, the basic steps of
the simplex method are: 1) to find a basic feasible solution to
start the algorithm. Sometimes, it might be difficult to start, this
may either implies there is no feasible solution or it is nec-
essary to reformulate the problem in a slightly different way by
changing the canonical form so that a basic feasible solution
can be found; 2) to see if the current basic feasible solution can
be improved (even marginally) by increasing the non-basic
variables from zero to non-negative values; 3) stop the pro-cess
if the current feasible solution cannot be improved, which
means that is optimal. If the current feasible solution is not
optimal, then move to an adjacent basic feasible solution. This
adjacent basic feasible solution can be obtained by changing
the canonical form via elementary row operations.
The pivot manipulations are based on the fact that a
linear system will remain an equivalent system by multiplying
a non-zero constant on a row and adding it to the other row.
This procedure continues by going to the second step and
repeat-ing the evaluation of the objective function. The
optimality of the problem will be reached or stop the iteration
if the solu-tion becomes unbounded in the case that you can
improve the objective indefinitely.

6.2.3 A Case Study

Now we come back to our example, if we use α = 2, β = 3,
n1 = 16, n2 = 10 and n = 20, we then have
Z
1 −2 −3 0 0 0 x1 0
0 1 0 1
1 1 0
0 0 1
0x2
0
= 20 , s1
(6.17)
16

2
0 0 1 0 0 1 s 10

s3

where x1, x2, s1, ..., s3 ≥ 0. Now the first step is to identify a
corner point or basic feasible solution by setting non-isolated
74 Chapter 6. Linear Mathematical Programming

variables x1 = 0 and x2 = 0 (thus the basic variables are s1,

s2, s3). We now have
s = 20, s = 16, s = 10. (6.18)
1 2 3

The objective function Z = 0, which corresponds to the cor-

ner point O in Fig. 6.1. In the present canonical form, the
corresponding column associated with each basic variables has only one
non-zero entry (marked by a box) for each constraint equality, and all
other entries in the same column are zero. The non-zero value usually
converts into 1 if it is not unity. This is shown as follows:
Z x1 x2 s1 s2 s3 (6.19)
1 −2 −3 0 0 0
0
0
1
1
1
0
1
0
0
1
0
0

0 0 1 0 0 1

When we change the set or the bases of basic variables

from one set to another, we will aim to convert to a similar
form using pivot row operations. There are two ways of
numbering this matrix. One way is to call the first row [1 −2
−3 0 0 0] as the zeroth row, so that all other rows correspond
to their corresponding constraint equation. The other way is
simply to use its order in the matrix, so [1 −2 −3 0 0 0] is
simply the first row. We will use these standard notations.
Now the question is if we can improve the objective by
increasing one of the non-basic variables x 1 and x2? Obviously,
if we increase x1 by a unit, then Z will also increase by 2 units.
However, if we increase x 2 by a unit, then Z will increase by 3
units. Since our objective is to increase Z as much as possible,
so we choose to increase x 2. As the requirement of the non-
negativeness of all variables, we cannot increase x 2 without
limit. So we increase x2 while holding x1 = 0, we have
s = 20 − x , s = 16, s = 10 − x . (6.20)
1 2 2 3 2
Thus, the highest possible value of x2 is x = 10 when s1 = s3 =
0. If x2 increases further, both s1 and s3 will become
negative, thus it is no longer a basic feasible solution.
6.2 Simplex Method 75

The next step is to use either by setting x 1 = 0 and s1 = 0 as

non-basic variables or by setting x 1 = 0 and s3 = 0. Both cases
correspond to the point A in our example, so we simply choose
x1 = 0 and s3 = 0 as non-basic variables, and the basic
variables are thus x2, s1 and s2. Now we have to do some pivot
operations so that s3 will be replaced by x2 as a new basic
variable. Each constraint equation has only a single basic
variable in the new canonical form. This means that each
column corresponding to each basic variable should have only
a single non-zero entry (usually 1). In addition, the right hand
sides of all the constraints are non-negative and increase the
value of the objective function at the same time. In order to
convert the third column for x2 to the form with only a single
non-zero entry 1 (all other coefficients in the column should be
zero), we first multiply the fourth row by 3 and add it to the first
row, and the first row becomes

Z − 2x1 + 0x2 + 0s1 + 0s2 + 3s3 = 30. (6.21)

Then, we multiply the fourth row by −1 and add it to the

second row, we have
0Z + x1 + 0x2 + s1 + 0s2 − s3 = 10. (6.22)

So the new canonical form becomes

Z
1 −2 0 0 0 3 x1 30
0 1 0 1 0 − 1x2 s1
= 10 ,
16
(6.23)
0 1 0 0 1 0
2
0 0 1 0 0 1 s 10

s3

where the third, fourth, and fifth columns (for x 2, s1 and s2,
respectively) have only one non-zero coefficient. All the right
hand sides are non-negative. From this canonical form, we
can find the basic feasible solution by setting non-basic
variables equal to zero. This is to set x 1 = 0 and s3 = 0. We
now have the basic feasible solution
x2 = 10, s1 = 10, s2 = 16, (6.24)
76 Chapter 6. Linear Mathematical Programming

which corresponds to the corner point A. The objective Z = 30.

Now again the question is whether we can improve the ob-
jective by increasing the non-basic variables. As the objective
function is
Z = 30 + 2x − 3s , (6.25)
1 3
Z will increase 2 units if we increase x 1 by 1, but Z will de-
crease −3 if we increase s3. Thus, the best way to improve
the objective is to increase x1. The question is what the limit
of x1 is. To answer this question, we hold s3 at 0, we have

s1 = 10 − x1, s2 = 16 − x1, x2 = 10. (6.26)

We can see if x1 can increase up to x1 = 10, after that s1

becomes negative, and this occurs when x1 = 10 and s1 =
0. This also suggests us that the new adjacent basic
feasible solution can be obtained by choosing s 1 and s3 as
the non-basic variables. Therefore, we have to replace s 1
with x1 so that the new basic variables are x1, x2 and s2.
Using these basic variables, we have to make sure that
the second column (for x1) has only a single non-zero entry.
Thus, we multiply the second row by 2 and add it to the first
row, and the first row becomes

Z + 0x1 + 0x2 + 2s1 + 0s2 + s3 = 50. (6.27)

We then multiply the second row by −1 and add it to the third

row, we have

0Z + 0x1 + 0x2 − s1 + s2 + s3 = 6. (6.28)

Thus we have the following canonical form

Z
1 0 0 2 0 1 x1 50
0 1 0 1 0 0 0 0 1 1
− 1x2 s1
= 10 ,
6
(6.29)
−
s

2
0 0 1 0 0 1 10

s3
6.2 Simplex Method 77

whose basic feasible solution can be obtained by setting

non-basic variables s1 = s3 = 0. We have

x1 = 10, x2 = 10, s2 = 6, (6.30)

which corresponds to the extreme point B in Fig. 6.1. The

objective value is Z = 50 for this basic feasible solution. Let
us see if we can improve the objective further. Since the
objective becomes
Z = 50 − 2s − s , (6.31)
1 3

any increase of s1 or s3 from zero will decrease the objective

value. Therefore, this basic feasible solution is optimal. In-
deed, this is the same solution as that obtained from the
graph method. We can see that a major advantage is that we
have reached the optimal solution after searching a certain
number of extreme points, there is no need to evaluate the
rest of the extreme points. This is exactly why the simplex
method is so efficient.
The case study we used here is relatively simple, but it is
useful to show how the basic procedure works in linear pro-
gramming. For more practical applications, there are well-
established software packages which will do the work for you
once you have set up the objective and constraints properly.
Chapter 7

Nonlinear Optimization

As most of the real world problems are nonlinear, nonlinear

mathematical programming thus forms an important part of
mathematical optimization methods. A broad class of nonlin-
ear programming problems is about the minimization or
maxi-mization of f (x) subject to no constraints, and another
impor-tant class is the minimization of a quadratic objective
function subject to nonlinear constraints. There are many
other nonlin-ear programming problems as well.
Nonlinear programming problems are often classified ac-
cording to the convexity of the defining functions. An inter-
esting property of a convex function f is that the vanishing of
the gradient rf (x∗) = 0 guarantees that the point x∗ is a
global minimum or maximum of f . If a function is not convex
or concave, then it is much more difficult to find global
minima or maxima.

7.1 Penalty Method

For the simple function optimization with equality and
inequal-ity constraints, a common method is the penalty
method. For the optimization problem
minimize T n
f (x), x = (x , ..., x ) ∈< ,
n
x< 1 n
subject to φi(x) = 0, (i = 1, ..., M ),

79
 80 Chapter 7. Nonlinear Optimization
ψj (x) ≤ 0, (j = 1, ..., N ), (7.1)

the idea is to define a penalty function Π(x, µi, νj ) so that the

constrained problem can be transformed into an
unconstrained problem. Now we define
M (x) + N (x),(7.2)
Π(x, µi, ν j ) = f (x) +µ i φi 2 ν j ψj 2

X X
i=1 j=1

where µi 1 and νj ≥ 0.
For example, in order to solve the following problem of the
Gill-Murray-Wright type
minimize f (x) = 100(x − b) 2 + 1,
x<

subject to g(x) = x − a ≥ 0, (7.3)

where a > b is a given value, we can define a penalty

function Π(x) using a penalty parameter µ 1. We have
Π(x, µ) = f (x) + µ g(x)T g(x)
2
= 100(x − b) + 1 + µ (x − a) , (7.4)
2 2

2
where the typical value for µ is 2000 ∼ 10000.
This method essentially transforms a constrained problem
0
into an unconstrained one. From the stationary condition Π (x)
= 0, we have
(7.5)
200(x∗ − b) − µ(x∗ − a) = 0,
which gives
200b + µa
x = . (7.6)
∗ 200 + µ
For µ → ∞, we have x∗ → a. For µ = 2000, a = 2 and b = 1,
we have x∗ ≈ 1.9090. This means the solution depends on
the value of µ, and it is very difficult to use extremely large values
without causing extra computational difficulties.
7.2 Lagrange Multipliers 81

7.2 Lagrange Multipliers

Another powerful methods without the above limitation of us-
ing large µ is the method of Lagrange multipliers. If we want
to minimize a function f (x)
T
minimizef (x), ∈ n , (7.7)
x <n x = (x1, ..., xn) <
subject to the following nonlinear equality constraint

g(x) = 0, (7.8)

then we can combine the objective function f (x) with the

equality to form a new function, called the Lagrangian

Π = f (x) + λg(x), (7.9)

where λ is the Lagrange multiplier, which is an unknown

scalar to be determined.
This essentially converts the constrained problem into an
unconstrained problem for Π(x), which is exactly the beauty
of this method. If we have M equalities,

gj (x) = 0,(j = 1, ..., M ), (7.10)

then we need M Lagrange multipliers λj (j = 1, ..., M ). We thus

have
M X

Π(x, λj ) = f (x) + λj gj (x). (7.11)

j=1
The requirement of stationary conditions leads to
∂Π M ∂g (i = 1, ..., n), (7.12)
= ∂f +λ j
j,
X
∂xi ∂xi j=1 ∂xi
and ∂Π

(j = 1, ..., M ). (7.13)
∂λj = gj = 0,
These M + n equations
∂Π
will determine the n-component of x
and M Lagrange multipliers. As = λj , we can consider λj
∂gj
82 Chapter 7. Nonlinear Optimization

as the rate of the change of the quantity Π as a functional of g j .

Example 7.1: To solve the optimization problem

maximize 2
f (x, y) = xy ,
(x,y) <2

subject to the condition

2 2
g(x, y) = x + y − 1 = 0.
We define
2 2 2
Π = f (x, y) + λg(x, y) = xy + λ(x + y − 1).
The stationary conditions become
∂Π 2
= y + 2λx = 0,
∂x
∂Π
= 2xy + 2λy = 0,

and
∂Π 2 2
= x + y − 1 = 0.
∂λ
The condition xy+λy = 0 implies that y = 0 or λ = −x. The
2
case of y = 0 can be eliminated as it leads to x = 0 from y +
2 2
2λx = 0, which does not satisfy the last condition x + y = 1.
Therefore, the only valid solution is

λ = −x.
From the first stationary condition, we have
2 2 2 2
y − 2x = 0, or y = 2x .
Substituting this into the third stationary condition, we have
2 2
x − 2x − 1 = 0,
which gives
x = ±1.
7.3 Kuhn-Tucker Conditions 83
So we have four stationary points

√ √ q
√
P1(1, 2), P2(1, − 2), P3(−1, 2), P4(−1, − (2)).
The values of function f (x, y) at these four points are
f (P1) = 2, f (P2) = 2, f (P3) = −2, f (P4) = −2.
√ √
Thus, the function reaches its maxima at (1, 2) and (1, − 2).
The Lagrange multiplier for this case is λ = −1.

7.3 Kuhn-Tucker Conditions

There is a counterpart of the Lagrange multipliers for the non-
linear optimization with constraint inequalities. The Kuhn-Tucker
conditions concern the requirement for a solution to be
optimal in nonlinear programming.
For the nonlinear optimization problem
minimize f (x),
x <n
subject to φ (x) = 0, (i = 1, ..., M ),
i
ψj (x) ≤ 0, (j = 1, ..., N ). (7.14)
If all the functions are continuously differentiable, at a local
minimum x∗, there exist constants λ0, λ1, ..., λq and µ1, ..., µp
such that
M
X
N X

λ0rf (x∗) + µirφi(x∗) + λj rψj (x∗) = 0, (7.15)

i=1 j=1
and
(j = 1, 2, ..., N ), (7.16)
ψj (x∗) ≤ 0, λj ψj (x∗) = 0,
where λj ≥ 0, (i = 0, 1, ..., N ). The constants satisfy the fol-
lowing condition
X
N X
M
λj + |µi| ≥ 0. (7.17)
j=0 i=1
This is essentially a generalised method of the Lagrange
mul-tipliers. However, there is a possibility of degeneracy
when λ0 = 0 under certain conditions.
84 Chapter 7. Nonlinear Optimization

7.4 No Free Lunch Theorems

The methods used to solve a particular problem depend largely
on the type and characteristics of the optimization problem it-
self. There is no universal method that works for all problems,
and there is generally no guarantee to find the optimal solu-tion
in global optimization problems. In fact, there are several so-
called Wolpert and Macready’s ‘No Free Lunch Theorems’ (NLF
theorems) which state that if any algorithm A outper-forms
another algorithm B in the search for an extremum of a cost
function, then algorithm B will outperform A over other cost
functions. NFL theorems apply to the scenario (either de-
terministic or stochastic) where a set of continuous (or discrete
or mixed) parameter θ maps the cost functions into a finite set.
Let nθ be the number of values of θ (either due to discrete
values or the finite machine precisions), and n f be the number
of values of the cost function. Then, the number of all possi-ble
n
combinations of cost functions is N = n fθ which is finite, but
usually huge. The NFL theorems prove that the average
performance over all possible cost functions is the same for all
search algorithms.
Mathematically, if P (sym |f, m, A) denotes the
performance, based on probability theory, of an algorithm A
iterated m times on a cost function f over the sample s m, then
we have the averaged performance for two algorithms
y y
P (sm |f, m, A) = P (sm |f, m, B), (7.18)
X X
f f

x y x y
where sm = {(s m(1), s m(1)), ..., (s m(m), s m(m))} is a time-
ordered set of m distinct visited points with a sample of size
m. The interesting thing is that the performance is indepen-
dent of algorithm A itself. That is to say, all algorithms for
optimization will give the same performance when averaged
over all possible functions. This means that the universally
best method does not exist.
Well, you might say, there is no need to formulate new al-gorithms
because all algorithms will perform equally well. The
7.4 No Free Lunch Theorems 85

truth is that the performance is measured in the statistical

sense and over all possible functions. This does not mean all
algorithms perform equally well over some specific functions.
The reality is that no optimization problems require averaged
performance over all possible functions. Even though, the NFL
theorems are valid mathematically, their influence on parame-
ter search and optimization is limited. For any specific set of
functions, some algorithms perform much better than others. In
fact, for any specific problem with specific functions, there
usually exist some algorithms that are more efficient than oth-
ers if we do not need to measure their average performance.
The main problem is probably how to find these better algo-
rithms for a given particular type of problem.
On the other hand, we have to emphasize on the best
es-timate or sub-optimal solutions under the given conditions
if the best optimality is not achievable. The knowledge about
the particular problem concerned is always helpful for the
ap-propriate choice of the best or most efficient methods for
the optimization procedure.
The optimization algorithms we discussed so far are con-
ventional methods and deterministic as there is no randomness
in these algorithms. The main advantage of these algorithms is
that they are well-established and benchmarked, but the dis-
advantages are that they could be trapped in a local optimum
and there is no guarantee that they will find the global opti-mum
even if you run the algorithms for infinite long because the
diversity of the solutions is limited.
Modern optimization algorithms such as genetic algorithms and
simulated annealing often involve a certain degree of ran-domness
so as to increase the diversity of the solutions and also to avoid
being trapped in a local optimum. The randomness makes sure
that it can ‘jump out’ any local optimum. Statis-tically speaking it
can find the global optima as the number of iterations approaches
infinite. The employment of randomness is everywhere, especially
in the metaheuristic methods such as particle swarm optimization
and simulated annealing. We will study these methods in detail in
the rest chapters of this book.
 Part III

Metaheuristic Methods
Chapter 8

Tabu Search

Most of the algorithms discussed so far, especially, the

gradient-based search methods, are local search methods. The
search usually starts with a guess, and tries to improve the
quality of the solutions. For unimodal functions, the convexity
guaran-tees the final optimal solution is also a global optimum.
For multimodal objectives, it is likely the search will get stuck at
a local optimum. In order to get out a local optimum, certain
variations with randomness should be used. A typical exam-ple
is the genetic algorithm which is a global search algorithm.
Another excellent example is the simulated annealing which is
a global search method and guarantees to reach the global op-
timality as computing time approaches infinity. In reality, it finds
the global optimality very efficiently.
The algorithms we will discuss in the rest of the book are
metaheuristic methods, here meta- means ‘beyond’ or ‘higher
level’ and heuristic means ‘to find’ or ‘to discover by trial and
error’. These metaheuristic methods include the Tabu search,
ant colony optimization, particle swarm optimization, and sim-
ulated annealing. First, let us study the Tabu search.

8.1 Tabu Search

Tabu search, developed by Fred Glover in 1970s, is the search
strategy that uses memory and search history as its integrated

89
90 Chapter 8. Tabu Search

component. As most of the successful search algorithms such

as gradient-based methods do not use memory, it seems that at
first it is difficult to see how memory will improve the search ef-
ficiency. Thus, the use of memory poses subtleties that may not
be immediately visible on the surface. This is because mem-ory
could introduce too many degrees of freedom, especially for the
adaptive memory use, which makes it almost impossible to use
the rigorous theorems-and-proof approach to establish the
convergence and efficiency of such algorithms. Therefore,
even though Tabu search works so well for certain problems, it
is difficult to analyse mathematically why it is so. Consequently,
Tabu search remains a heuristic approach. Two important is-
sues that are still under active research are how to use the
memory effectively and how to combine with other algorithms
to create more superior new generation algorithms.
Tabu search is an intensive local search algorithm and
the use of memory avoids the the potential cycling of local
solu-tions so as to increase the search efficiency. The recent
tried or visited solutions are recorded and put into a Tabu list
and new solutions should avoid those in the Tabu list.
The Tabu list is an important concept in Tabu search, and
it records the search moves as the recent history, and any
new search move should avoid this list of previous moves,
this will inevitably save time as previous moves are not
repeated. Over a large number of iterations, this Tabu list
could save tremen-dous amount of computing time and thus
increase the search efficiency significantly.
Let us illustrate the concept of the Tabu list through an
example. Suppose we want to go through each route (straight-
line) once and only once without repetition (see Fig. 8.1).

Example 8.1: If we start the travel from, say, point E, we first

go along EA and reach point A. Now put the route EA in the
Tabu list which is
Tabu list ={EA} ,
where we do not distinguish between EA and AE, though such
difference is important for directed graphs. When we reach point
8.1 Tabu Search 91

As sB

Es sD sC

Figure 8.1: Passing through each line once and only once.

A, the next move can only be AB because EA is in the Tabu list

so going back is not allowed. We now update the Tabu list as

Tabu list ={EA, AB} .

At point B, there are four possible routes, but only three (BE,
BD, BC) are permissible routes because AB is now in the
updated Tabu list. We see that the length of the Tabu list is
increasing and flexible. Suppose we randomly choose the
route BD to go to point D. The Tabu list now becomes

Tabu list ={EA, AB, BD} .

Now at point D, we can either go along BC or along DE, but

we cannot go back along BD. So we randomly choose DE to
reach point E.
Tabu list ={EA, AB, BD, DE} .
At point E, we have three possible routes, but only EB is permis-
sible as the other two routes are forbidden. So we have to choose
route EB to go back to point B, and the current Tabu list becomes

Tabu list ={EA, AB, BD, DE, EB} .

Again at point B, there are four routes, but only BC is

permissible and all other three routes are in the Tabu list and
thus not allowed. So we have to go along BC to reach point
C. The updated Tabu list is
Tabu list ={EA, AB, BD, DE, EB, BC} .
92 Chapter 8. Tabu Search

Now the only permissible route at point C is CD to go to point

D. Here we complete the task of going through each branch
once and only once. If we now update the Tabu list, we get

Tabu list ={EA, AB, BD, DE, EB, BC, CD} ,

which is the complete route map for this task.

Here we have used the whole history as the Tabu list. There
no reason why we must use the whole history, we can use it if
the computer memory is not an issue. More often, we only use
part of the history as an ongoing Tabu list, usually with a fixed
length. If we use a fixed length n = 5 in our example, then the
last two steps, the Tabu list should be replaced by

Tabu list ={AB, BD, DE, EB, BC} ,

and
Tabu list ={BD, DE, EB, BC, CD} .

We can see that the Tabu list serves as a way of mimick-

ing the way we humans tackle the problem by ‘trial and error’
(heuristically) and by memorizing the steps we have tried.
This is probably why the Tabu search is much more efficient
than memoryless search methods. Furthermore, the
advantage of a fixed length Tabu list is that it saves memory
and is easy to implement, but it has the disadvantage of
incomplete informa-tion. Effective usage of memory in Tabu
search is still an area of active research. In general, a Tabu
list should be flexible with enough information and should be
easy for implementation in any programming language.
Tabu search was originally developed together with the in-teger
programming, a special class of linear programming. The Tabu list in
combination with integer programmming can re-duce the computing
time by at least two orders for a given problem, comparing the solely
standard integer programming. The Tabu search can also be used along
with many other algo-rithms.
 8.2 Travelling Salesman Problem 93
5s s4 5s s4

2 2
s s
s3 s3
1s 1s
Figure 8.2: Travelling salesman problem for five cities and a
possible route (not optimal).

8.2 Travelling Salesman Problem

Travelling salesman problem (TSP) is an optimization problem

for a given number of cities (say, n) and their locations. The
cities are represented as nodes in a graph and the distances
between two cities are represented as the weight of a directed
edge or route between the two cities. The aim is to find a path
that visits each city once, and returning to the starting city,
minimizing the total distance travelled. This is a very difficult
problem as its only known solution to find the shortest path
requires a solution time that grows exponentially with the
problem size n. In fact, the travelling salesman problem is an
NP-hard problem, which means that there are no known
algorithms (of polynomial time) that exist.
For simplicity, we now look at the travelling salesman
prob-lem with only five cities: city 1 at (0,0), city 2 at (5,2),
city 3 at (10,0), city 4 at (10,10), and city 5 at (0,10). Here
the unit is 1000 km. There are many possible routes. Is the
route 2-3-1-5-4-2 optimal? If not, how to prove this?
Let us first try an exhaustive search. The Euclidean dis-
tance between any two cities i and j is given by

q
2 2
dij = (xi − xj ) + (yi − yj ) ), (i, j = 1, 2, ..., 5), (8.1)

where (xi, yi) are the Cartesian coordinates of city i. This will
94 Chapter 8. Tabu Search

give the following distance matrix d

05.39 10.00 14.14 10.00

5.39 0 39 9.43 9.43
d = 10.00 5.39 5.0 10.00 14.14 , (8.2)
14.14 9.43 10.00 0 10.00

10.00 9.43 14.14 10.00 0

where the diagonal elements are zero because the distance

of a city to itself is zero.
In order to avoid any potential problem in finding the
short-est distance, we now set the distance dii = ∞ for i = 1,
2, .., 5. This leads to
∞ 5.39 10.00 14.14 10.00
5.39 ∞ 5.39 9.43 9.43
d = 10.00 5.39 ∞ 10.00 14.14 . (8.3)
14.14 9.43 10.00 10.00

9.43 14.14 ∞
10.00 10.00
∞
As there are n = 5 cities, there are 5! = 120 ways of visiting
these cities such as 2-5-3-1-4-2, 5-4-1-2-3-5 (see Fig. 8.3)
and so on and so forth. For example, the route 2-5-3-1-4-2
has a total distance

9.43 + 14.14 + 10.00 + 14.14 + 9.43 ≈ 57.14, (8.4)

while the route 5-4-1-2-3-5 has a total distance

10.00 + 14.14 + 5.39 + 5.39 + 14.14 ≈ 49.06. (8.5)

By searching over all the 120 combinations and calculating

the total distance for each case, we can finally find the
shortest route is 1-2-3-4-5-1 (or their ordered permutations
such as 2-3-4-5-1-2 and 4-5-1-2-3-4) with a distance 5.39 +
5.39 + 10.00 + 10.00 + 10.00 ≈ 40.78.
For n = 5, we can use the exhaustive search because n!
= 120 is small. Now suppose n = 50, the number of possible
combinations is 50! ≈ 3.04 × 1064 while for n = 100 cities,
100! ≈ 9.3 × 10157. Even with all modern computers working
8.3 Tabu Search for TSP 95
5s s45 s s4

2 2
s s
s3
1s s31 s
Figure 8.3: Two different routes: 2-5-3-1-4-2 and 5-4-1-2-3-5.

in parallel, it would takes the time much longer than the age
of the universe to do such a huge number of calculations. So
we have to use other methods.

8.3 Tabu Search for TSP

Now let us use Tabu search for the travelling salesman problem.
As there is no known efficient method for solving such problems,
we use a method of systematic swapping. For example, when n =
3 cities, any order is an optimal 1-2-3 or 2-3-1 and others. For n =
four cities, suppose we know that optimal route is 1-2-3-4-1 (its
ordered permutation 2-3-4-1-2 and others are the same route).
However, initially we start randomly from, say, 2-4-3-1-2. A simple
swap between cities 3 and 4 leads to 2-3-4-1-2 which is the
optimal solution. Of course, we do not know which two cities
should be swapped, so we need a systematic approach by
swapping any two cities i and j. Furthermore, in order to swap any
two cities, say, 2 and 4, we can either simple swap them or we can
use a more systematical approach by swapping 2 and 3, then 3
and 4. The latter version takes more steps but it does provide a
systematic way of implementing the algorithms. Thus, we will use
this latter approach.
For simplicity, we still use the same example of the five cities.
Suppose we start from a random route 2-5-3-1-4-2 (see Fig. 8.3). In order
to swap any two cities among these five cities, we use the following swap
indices by swapping two ad-
96 Chapter 8. Tabu Search
jacent cities 3 2 4 5
1
2 1 3 4 5
swap = 2 4 3 , (8.6)
1
1 2 3 5
5
4

5 2 3 4 1

where the first row swaps the first two cities and the fifth row
swap the first and last (fifth) cities. In order to avoid repetition of
recent swaps, we use a Tabu list for the above index matrix

Tabu list = 0 0 0 0 0 , (8.7)

where the first element corresponds to the first row of the

swap index matrix. Now we also use a fixed length memory,
say, m = 2 which means that we only record two steps in
recent history.
From the initial route= [2 5 3 1 4] with an initial distance d
= 57.14, let us try the new route by swapping the first two
cities or route = route[swap(1)]. We have

route = [5 2 3 1 4], (8.8)

whose total distance is 48.96 which is shorter than d = 57.14

so we update the new shortest distance as d = 48.96. Here
we can update the Tabu list, but it is better to update it when
we find a better route after going through all the five swaps.
Now we generate the new route by swapping the second
two pair or route = route [swap(2)]. We now have

route = [2 3 5 1 4], (8.9)

whose the total distance is 53.10 which is larger than d so

dis-card this attempt and still use d = 48.96.
Similarly, try the next swap, we get

route = [2 5 1 3 4], (8.10)

with a distance 48.87, which is shorter than d = 48.96, so we

update d = 48.87. The next swap leads to route=[2 5 3 4 1]
8.3 Tabu Search for TSP 97
5s s45 s s4

2 2
s s
s3
1s s31 s
Figure 8.4: Current best route after first generation of swaps
and the final optimal route.

with a distance 53.10, so we discard it. The fifth swap gives

route=[4 5 3 1 2] with a distance 48.96 which is longer than d
= 48.87, so we also discard.
After running through all the possible five swaps, the cur-
rent best route is
route = [2 5 1 3 4], (8.11)

with d = 48.87. Now we have to update the Tabu list and it

becomes

Tabu list = 0 0 m 0 0 = 0 0 2 0 0, (8.12)

here the entry corresponding to the third swap is recorded.

Now we have to start all the five sequential swaps again
from this current best route=[2 5 1 3 4]. The first swap leads
to route=[5 2 1 3 4] with a distance of 44.82 which is shorter
than 48.87. So the new distance is now d = 44.82. The
second swap, we got the new route

route = [2 1 5 3 4], (8.13)

whose distance is 48.96 so we discard it. As the next entry in

the Tabu list is 2, we will not carry out this swap. Similarly, the
next swap leads to route=[2 5 1 4 3] with a distance of 48.96 so
we discard it. Finally, the fifth swap gives the route=[4 5 1 3 2]
with a distance of 44.82 which is the same as the first swap,
98 Chapter 8. Tabu Search

so there is no need to update. After all the four permissible

swaps, the current best route

route = [5 2 1 3 4], (8.14)

and the new Tabu list is

Tabu list = 2 0 1 0 0 . (8.15)

where the third element is reduced from 2 to 1, which means

the history is running and we can use the swap again later
when this entry becomes zero.
Starting yet again from the current best route=[5 2 1 3 4]
after the two generation swaps (see Fig. 8.4), the first swap
is not allowed because the first entry in the Tabu list is 2. So
we start the second swap which leads to

route = [5 1 2 3 4], (8.16)

which has a total distance d = 40.78 and this is the global

optimal route for this five cities (see Fig. 8.4).
The next swap leads to route=[5 2 1 4 3] with a distance
of 53.10. As we continue, there is no further improvement in
the solution quality. As we do not know whether the current
opti-mal solution is the global optimality or not, we will
continue the search until a prescribed number of iterations or
after certain number of iterations with no further
improvement in solution quality. In this present case, the
global optimal solution is the route =[5 1 2 3 4] or 5-1-2-3-4-5
or its ordered permutations such as 2-3-4-5-1-2 and others.
We have seen that the Tabu search only takes about 10
steps to reach the global optimal route, and this is much
more effi-cient comparing with the 120 steps by the
exhaustive method. In addition, the Tabu list avoided at least
3 possible repeti-tions, saving about 1/3 of the moves. In
general, for problems with large n, the computing time saved
using Tabu search will be more significant.
Chapter 9

Ant Colony
Optimization

From the Tabu search, we know that we can improve the search
efficiency by using memory. Another way of improving the
efficiency is to use the combination of randomness and memory.
The randomness will increase the diversity of the solutions so as to
avoid being trapped in local optima. The memory does not means
to use simple history records. In fact, there are other forms of
memory using chemical messenger such as pheromone which is
commonly used by ants, honeybees, and many other insects. In
this chapter, we will discuss the nature-inspired ant colony
optimization (ACO), which is a metaheuristic method.

9.1 Behaviour of Ants

Ants are social insects in habit and they live together in or-
ganized colonies whose population size can range from about 2
to 25 millions. When foraging, a swarm of ants or mo-bile
agents interact or communicate in their local environment. Each
ant can lay scent chemicals or pheromone so as to com-
municate with others, and each ant is also able to follow the
route marked with pheromone laid by other ants. When ants
find a food source, they will mark it with pheromone and also

99
100 Chapter 9. Ant Colony Optimization

mark the trails to and from it. From the initial random for-
aging route, the pheromone concentration varies and the
ants follow the route with higher pheromone concentration,
and the pheromone is enhanced by increasing number of
ants. As more and more ants follow the same route, it
becomes the favoured path. Thus, some favourite route
(often the shortest or more efficient) emerges. This is
actually a positive feedback mecha-nism.
Emerging behaviour exists in an ant colony and such emer-
gence arises from simple interactions among individual ants.
Individual ants act according to simple and local information
(such as pheromone concentration) to carry out their activities.
Although there is no master ant overseeing the entire colony
and broadcasting instructions to the individual ants, organized
behaviour still emerges automatically. Therefore, such emer-
gent behaviour is similar to other self-organized phenomena
which occur in many processes in nature such as the pattern
formation in animal skins (tiger and zebra skins).
The foraging pattern of some ant species (such as the army
ants) can show extraordinary regularity. Army ants search for
◦
food along some regular routes with an angle of about 123
apart. We do not know how they manage to follow such reg-
ularity, but studies show that they could move in an area and
build a bivouac and start foraging. On the first day, they for-age
in a random direction, say, the north and travel a few hundred
meters, then branch to cover a large area. The next day, they
◦
will choose a different direction, which is about 123 from the
direction on the previous day and cover a large area. On the
following day, they again choose a different direction about
◦
123 from the second day’s direction. In this way, they cover the
whole area over about 2 weeks and they move out to a
different location to build a bivouac and forage again.
The interesting thing is that they do not use the angle of
360◦/3 = 120◦ (this would mean that on the fourth day, they will
search on the empty area already foraged on the first day). The
beauty of this 123◦ angle is that it leaves an angle of about 10 ◦
from the direction on the first day. This means they cover
9.2 Ant Colony Optimization 101

Ant Colony Optimization

begin
T
Objective function f (x), x = (x1, ..., xn )
[or f (xij ) for routing problem where (i, j) ∈ {1, 2, ..., n}]
Define pheromone evaporate rate γ
while ( criterion )
for loop over all n dimensions (or nodes)
Generate new solutions
Evaluate the new solutions
Mark better locations/routes with pheromone δφij
Update pheromone: φij ← (1 − γ)φij + δφij
end for
Daemon actions such as finding the current
best end while
Output the best results and pheromone distribution
end

Figure 9.1: Pseudo code of ant colony optimization.

the whole circle in 14 days without repeating (or covering an

previously-foraged area). This is an amazing phenomenon.

9.2 Ant Colony Optimization

Based on these characteristics of ant behaviour, scientists have
developed a number of powerful ant colony algorithms with
important progress made in recent years. Marco Dorigo pio-
neered the research in this area in 1992. In fact, we only use
some of the nature or the behaviour of ants and add some new
characteristics, we can devise a class of new algorithms.
The basin steps of the ant colony optimization (ACO) can
be summarized as the pseudo code shown in Fig. 9.1.
Two important issues here are: the probability of choosing a
route, and the evaporation rate of pheromone. There are a few
ways of solving these problems although it is still an area of
active research. Here we introduce the current best methods.
For a network routing problem, the probability of ants at a
102 Chapter 9. Ant Colony Optimization

particular node i to choose the route from node i to node j is

given by
β
φα d
p = ij ij
ij n α β , (9.1)
P φ d
i,j=1 ij ij
where α > 0 and β > 0 are the influence parameters, and their
typical values are α ≈ β ≈ 2. φij is the pheromone concentra-tion
on the route between i and j, and d ij the desirability of the same
route. Some a priori knowledge about the route such as the
distance sij is often used so that d ij ∝ 1/sij , which implies that
shorter routes will be selected due to the travelling time is
shorter, and thus the pheromone concentration is higher.
This probability formula reflects the fact that ants would
normally follow the paths with higher pheromone concentra-
tion. In the simpler case when α = β = 1, the probability of
choosing a path by ants is proportional to the pheromone
concentration on the path. The denominator normalizes the
probability so that it is in the range between 0 and 1.
The pheromone concentration can change with time due
to the evaporation of pheromone. Furthermore, the
advantage of pheromone evaporation is that it could avoid
the system being trapped in local optima. If there is no
evaporation, then the path randomly chosen by the first ants
will become the pre-ferred path as the attraction of other
ants by their pheromone. For a constant rate γ of pheromone
decay or evaporation, the pheromone concentration usually
varies with time exponen-tially
−γt (9.2)
φ(t) = φ e , 0
where φ0 is the initial concentration of pheromone and t is
the time. If γt 1, then we have φ(t) ≈ (1 − γt)φ 0. For the
unitary time increment t = 1, the evaporation can be
t+1 t
approximated by φ ← (1 − γ)φ . Therefore, we have the
simplified pheromone update formula:

t+1 t t (9.3)
φij = (1 − γ)φij + δφij ,
where γ ∈ [0, 1] is the rate of pheromone evaporation. The
increment δφtij is the amount of pheromone deposited at time
9.3 Double Bridge Problem 103

t along route i to j when an ant travels a distance L. Usually

t
δφ ij ∝ 1/L. If there are no ants on a route, then the
pheromone deposit is zero.
There are other variations to these basic procedures. A
pos-sible acceleration scheme is to use some bounds of the
pheromone concentration and only the ants with the current
global best solution(s) are allowed to deposit pheromone. In
addition, cer-tain ranking of solution fitness can also be used.
These are hot topics of current research.

9.3 Double Bridge Problem

A standard test problem for ant colony optimization is the
sim-plest double bridge problem with two branches (see Fig.
9.2) where route (2) is shorter than route (1). The angles of
these two routes are equal at both point A and point B so
that the ants have equal chance (or 50-50 probability) of
choosing each route randomly at the initial stage at point A.
Initially, fifty percent of the ants would go along the longer
route (1) and the pheromone evaporates at a constant rate,
but the pheromone concentration will become smaller as
route (1) is longer and thus takes more time to travel
through. Conversely, the pheromone concentration on the
shorter route will increase steadily. After some iterations,
almost all the ants will move along the shorter routes. Figure
9.3 shows the initial snapshot of 10 ants (5 on each route
initially) and the snapshot after 5 iterations (or equivalent to
50 ants have moved along this section). Well, there are 11
ants, and one has not decided which route to follow as it just
comes near to the entrance. Almost all the ants (well, about
90% in this case) move along the shorter route.
Here we only use two routes at the node, it is straightfor-
ward to extend it to the multiple routes at a node. It is ex-pected
that only the shortest route will be chosen ultimately. As any
complex network system is always made of individual nodes,
this algorithms can be extended to solve complex rout-ing
problems reasonably efficiently. In fact, the ant colony al-
104 Chapter 9. Ant Colony Optimization

(1)
=⇒ A (2) B=⇒

Figure 9.2: Test problem for routing performance: route (2) is

shorter than route (1).
r r
r
r r
r
r r
r r r
r r r rr r r r
r r r
(a) (b)

Figure 9.3: Route selection via ACO: (a) initially, ants choose
each route with 50-50 probability, and (b) almost all ants
move along the shorter route after 5 iterations.

gorithms have been successfully applied to the Internet

routing problem, travelling salesman problem, combinatorial
optimiza-tion problem, and other NP-hard problems.

9.4 Multi-Peak Functions

As we know that ant colony optimization has successfully solved
NP-hard problems such as travelling salesman problems, it can
also be extended to solve the standard optimization problems of
multimodal functions. The only problem now is to figure out how
the ants will move on a n-dimensional hyper-surface. For simplicity,
we now make the discussion using the 2-D case which can easily
be extended to higher dimensions. On a 2D landscape, ants can
◦ ◦
move in any direction or 0 ∼ 360 , but this will cause some problems.
How to update the pheromone
9.4 Multi-Peak Functions 105

at a particular point as there are infinite number of points.

One solution is to track the history of each ant moves and
record the locations consecutively, and the other approach is
to use a moving neighbourhood or window. The ants ‘smell’
the pheromone concentration of their neighbourhood at any
particular location.
In addition, we can limit the number of directions the ants
can move. For example, ants are only allowed to move left and
right, and up and down (only 4 directions). We will use this
quantized approach here, which will make the implementation
much simpler. Furthermore, the objective function or land-scape
can be encoded into virtual food so that ants will move the best
locations where the best food sources are. This will make the
search process even more simpler. This simplified algorithm is
called Virtual Ant Algorithm (VAA) developed by Xin-She Yang
and his colleagues in 2006, which has been successfully
applied to optimization problems in engineering.
The following Keane function with multiple peaks is a
stan-dard test function
f (x, y) = sin2(x − y) sin2(x + y) , (9.4)
p
x2 + y2
where
(x, y) ∈ [0, 10] × [0, 10].
This function without any constraint is symmetric and has two
highest peaks at (0, 1.39325) and (1.39325, 0). To make the
problem harder, it is usually optimized under two constraints:
x + y ≤ 15,xy ≥ 3 . (9.5)
4
This makes the optimization difficult because it is now nearly
symmetric about x = y and the peaks occur in pairs where
one is higher than the other. In addition, the true maximum is
f (1.593, 0.471) ≈ 0.365, which is defined by a constraint
boundary.
Figure 9.4 shows the surface variations of the multi-peaked
function. If we use 50 roaming ants and let them move around
106 Chapter 9. Ant Colony Optimization

Figure 9.4: Surface variations of the multi-peak function.

for 25 iterations, then the pheromone concentrations (also

equiv-alent to the paths of ants) are displayed in Fig. 9.4.
It is worth pointing out that ant colony algorithms are the
right tool for combinatorial and discrete optimization. They
have the advantages over other stochastic algorithms such
as genetic algorithms and simulated annealing in dealing
with dy-namical network routing problems.
For continuous decision variables, its performance is still
under active research. For the present example, it took about
1500 evaluations of the objective function so as to find the
global optima. This is not as efficient as other metaheuristic
methods, especially comparing with particle swarm optimiza-
tion. This is partly because the handling of the pheromone
takes time. Is it possible to eliminate the pheromone and just
use the roaming ants? The answer is yes. Particle swarm op-
timization is just the right kind of algorithm for such further
modifications. This is the topic of the next chapter.
Chapter 10

Particle Swarm
Optimization

10.1 Swarm Intelligence

Particle swarm optimization (PSO) was developed by Kennedy
and Eberhart in 1995, based on the swarm behaviour such as
fish and bird schooling in nature. Many algorithms (such as ant
colony algorithms and virtual ant algorithms) use the behaviour
of the so-called swarm intelligence. Though particle swarm op-
timization has many similarities with genetic algorithms and
virtual ant algorithms, but it is much simpler because it does not
use mutation/crossover operators or pheromone. Instead, it
uses the real-number randomness and the global communica-
tion among the swarm particles. In this sense, it is also easier
to implement as there is no encoding or decoding of the param-
eters into binary strings as those in genetic algorithms.
This algorithm searches a space of an objective function by
adjusting the trajectories of individual agents, called par-ticles,
as the piecewise path formed by positional vectors in a quasi-
stochastic manner. The particle movement has two ma-jor
components: a stochastic component and a deterministic
component. The particle is attracted toward the position of the
current global best while at the same time it has a ten-dency to
move randomly. When a particle finds a location that

107
108 Chapter 10. Particle Swarm Optimization

Possible
f x∗
directions i
:
-
f

H
particle i A H
HH
A Hj
? AU v
g∗

Figure 10.1: Schematic representation of the motion of a

∗
parti-cle in PSO, moving towards the global best g and the
∗
current best x i for each particle i.

is better than any previously found locations, then it updates

it as the new current best for particle i. There is a current
best for all n particles. The aim is to find the global best
among all the current best until the objective no longer
improves or after a certain number of iterations.

10.2 PSO algorithms

The essential steps of the particle swarm optimization can
be summarized as the pseudo code shown in Fig. 10.2. The
par-ticle movement is schematically represented in Fig. 10.1
∗ ∗
where x i is the current best for particle i, and g ≈ min /
max{f (xi)}, (i = 1, 2, ..., n) is the current global best.
Let xi and vi be the position vector and velocity for particle i,
respectively. The new velocity vector is determined by
t +1 t ∗ t ∗ t
vi =vi+α1 [g − x i] + β 2 [x i − x i]. (10.1)

where 1 and 2 are two random vectors, and each entry taking
the value between 0 and 1. The Hadamard product of two
matrices u v is defined as the entrywise product, that is [u v] ij
= uij vij .
The initial values of x(i, j, t = 0) can be taken as the
bound-ary values [a = min(xj ), b = max(xj )] and vti=0= 0.
10.3 Accelerated PSO 109

Particle Swarm Optimization

begin
T
Objective function f (x), x = (x1, ..., xp ) Initialize
locations xi and velocity vi of n particles.
t=0
Initial minimum f = min{f (x1), ..., f (xn )} (at t = 0) min

t=t+1
for loop over all n particles and all p dimensions
t +1
Generate new velocity v i using equation
t +1 t t +1
(10.1) Calculate new locations x i = x i + v i
Evaluate objective functions at new locations
t +1 t+1
x i Find the current minimum fmin
end for
Find the current best x∗i and current global best
g∗ end while
Output the results x∗i and g∗
end

Figure 10.2: Pseudo code of particle swarm optimization.

The parameters α and β are the learning parameters or

acceleration constants, which can typically be taken as, say,
α ≈ β ≈ 2. The new position can then be updated by
t+1 t t+1 (10.2)
xi = x i + vi .

Although v can be any values, it is usually bounded in some

range [0, vmax].

10.3 Accelerated PSO

There are many variations which extend the standard algo-rithm,
and the most noticeably the algorithms use inertia func-tion θ(t) so
t t
that v i is replaced by θ(t)v i where θ takes the values between 0
and 1. In the simplest case, the inertia function can be take as a
constant, say, θ ≈ 0.5 ∼ 0.9. This is equivalent to introduce a
virtual mass to stabilize the motion of the particles, and thus the
algorithm is expected to converge more quickly.
110 Chapter 10. Particle Swarm Optimization

The standard particle swarm optimization uses both the

current global best g∗ and the individual best x∗i. The rea-son of
using the individual best is primarily to increase the diversity in the
quality solution, however, this diversity can be simulated using the
randomness. Subsequently, there is no compelling reason for using the
individual best. A simplified version which could accelerate the
convergence of the algorithm is to use the global best only. Thus, in the
accelerated particle swarm optimization, the velocity vector is generated
by
t − 0.5) + − x ), (10.3)
vt+1 = v + α(
∗
β(g
i i i

where is a random variable with values from 0 to 1. The

update of the position is simply
t+1 t t+1 (10.4)
xi = x i + vi .
In order to increase the convergence even further, we can also write the
update of the location in a single step
= (1− β)x + βg∗ + α( − 0.5). (10.5)
xt+1
i i

This simpler version will give the same order of

convergence. The typical values for this accelerated PSO
are α ≈ 0.1 ∼ 0.4 and β ≈ 0.1 ∼ 0.7, though α ≈ 0.2 and β ≈
0.5 are recommended for most unimodal objective functions.
A further accelerated PSO is to reduce the randomness
as iterations proceed. This mean that we can use a
monotonically decreasing function such as
−γt t
α = α0e , or α = α0γ , (γ < 1), (10.6)

where α0 ≈ 0.5 ∼ 1 is the initial value of the randomness

parameter. t is the number of iterations or time steps. γ < 1 is
a control parameter. For example, in our implementation
given later in this chapter, we will use

α = 0.7 ,
t (10.7)

where t ∈ [0, 10]. The implementation of the accelerated PSO in

Matlab and Octave is given later. Various studies show
10.4 Multimodal Functions 111

Figure 10.3: Multiple peaks with a global maximum at

(0.7634, 0.7634). We have used λ = 1.

that PSO algorithms can outperform genetic algorithms and

other conventional algorithms for solving many optimization
problems. This is partially due to that fact that the broadcast-
ing ability of current best estimates gives a better and
quicker convergence towards the optimality. However, this
algorithm is almost memoryless since it does not record the
movement paths of each particle, and it is expected that it
can be further improved using short-term memory in the
similar fashion as that in Tabu search.

10.4 Multimodal Functions

Multiple peak functions are often used to validate new algo-
rithms. We can construct the following function with multiple
peaks,
2 2 2
sin(x + y )
f (x, y) = ,λ > 0, (10.8)
2 e−λ(x−y)
p +y
x2
112 Chapter 10. Particle Swarm Optimization

where (x, y) ∈ [0, 5] × [0, 5]. Obviously, for a minimization

problem, we can write it as
2 2 2
sin(x + y )
f (x, y) = − ,λ > 0. (10.9)
p 2 2
x +y e−λ(x−y)
Here we have designed our own test function, thought it is a
standard practice to benchmark new algorithms against the
test functions in the literature. In fact, various studies have
already benchmarked the PSO algorithms against standard
test functions. Later, we will also use the Michaelewicz
function in our implementation as a simple demo.
The reason we devised our own function is that this mul-
tiple peak function has a global maximum in the domain and
its location is independent of λ > 0. If λ is very small, say,
λ = 0.01, then it is very difficult to find the optimality for
many algorithms. Rosenbrock’s banana function has similar
properties.
The first derivatives of f are
∂f 2
e−λ(x−y)
= 2 2
∂x p 2 2 {2x cos(x + y )
x +y
−[ x 2 2 (10.10)
+ 2λ(x − y)] tan(x + y )},
2 2
x +y
∂f = e−λ(x−y)2 2 2
cos(x + y )
2 2
p {2y
y∂y x2 + y2
−[ 2 2 + 2λ(x − y)] tan(x + y )}. (10.11)
x +y
As the function is symmetric in terms of x and y, the max-
ima must be on the line of x = y and are determined by
∂f = 0, ∂f = 0. (10.12)
∂x ∂y
By using x = y and x > 0, we have
2 1
2x cos(2x ) −
2 (10.13)
2x sin(2x ) = 0,
10.5 Implementation 113

Figure 10.4: Initial locations and final locations of 40 particles

after 10 iterations. The point marked with corresponds to the
best estimate (x∗, y∗).

or
2 2 2 (10.14)
sin(2x ) = 4x cos(2x ),
which has multiple solutions. The global maximum occurs at
(10.15)
x∗ = y∗ ≈ 0.7634.
It is worth pointing out that the solution x∗ = y∗ is
independent of λ as long as λ > 0.
If we use 40 particles, the new locations of these
particles after 10 iterations (generations) are shown in Figure
10.4. The final optimal solution at t = 10 is shown on the right
(with the best location marked with ).

10.5 Implementation
The accelerated particle swarm optimization has been imple-
mented using both Matlab and Octave. If you type the follow-
ing program and save it as, say, pso simpledemo.m, then
launch Matlab or Octave and change to the directory where
the file was saved. After typing in >pso simpledemo, it will
find the global optimal solution in less a minute on most
modern per-sonal computers.
114 Chapter 10. Particle Swarm Optimization

% The Particle Swarm Optimization

% (written by X S Yang, Cambridge University)
% Usage: pso(number_of_particles,Num_iterations)
% eg: best=pso_demo(20,10);
% where best=[xbest ybest zbest] %an n by 3 matrix
% xbest(i)/ybest(i) are the best at ith iteration

function [best]=pso_simpledemo(n,Num_iterations)
% n=number of particles
% Num_iterations=total number of iterations
if nargin<2, Num_iterations=10; end
if nargin<1, n=20; end
% Michaelewicz Function f*=-1.801 at [2.20319, 1.57049]
% Splitting two parts to avoid a long line for
printing str1=‘-sin(x)*(sin(x^2/3.14159))^20’;
str2=‘-sin(y)*(sin(2*y^2/3.14159))^20’;
funstr=strcat(str1,str2);
% Converting to an inline function and
vectorization f=vectorize(inline(funstr));
% range=[xmin xmax ymin ymax];
range=[0 4 0 4];
% ----------------------------------------------------
% Setting the parameters: alpha, beta
% Random amplitude of roaming particles alpha=[0,1]
% alpha=gamma^t=0.7^t;
% Speed of convergence (0->1)=(slow-
>fast) beta=0.5;
% ----------------------------------------------------
% Grid values of the objective function
% These values are used for
visualization only Ngrid=100;
dx=(range(2)-range(1))/Ngrid;
dy=(range(4)-range(3))/Ngrid;
xgrid=range(1):dx:range(2);
ygrid=range(3):dy:range(4);
[x,y]=meshgrid(xgrid,ygrid);
z=f(x,y);
% Display the shape of the function to be
optimized figure(1);
surfc(x,y,z);
% ---------------------------------------------------
best=zeros(Num_iterations,3); % initialize history
10.5 Implementation 115

% ----- Start Particle Swarm Optimization -----------

% generating the initial locations of n
particles [xn,yn]=init_pso(n,range);
% Display the paths of particles in a figure
% with a contour of the objective function
figure(2);
% Start iterations
for i=1:Num_iterations,
% Show the contour of the function
contour(x,y,z,15); hold on;
% Find the current best location
(xo,yo) zn=f(xn,yn);
zn_min=min(zn);
xo=min(xn(zn==zn_min));
yo=min(yn(zn==zn_min));
zo=min(zn(zn==zn_min));
% Trace the paths of all roaming particles
% Display these roaming particles
plot(xn,yn,‘.’,xo,yo,‘*’); axis(range);
% The accelerated PSO with alpha=gamma^t
gamma=0.7; alpha=gamma.^i;
% Move all the particles to new locations
[xn,yn]=pso_move(xn,yn,xo,yo,alpha,beta,range);
drawnow;
% Use "hold on" to display paths of
particles hold off;
% History
best(i,1)=xo; best(i,2)=yo; best(i,3)=zo;
end %%%%% end of iterations
% ----- All subfunctions are listed here -----
% Intial locations of n particles
function [xn,yn]=init_pso(n,range)
xrange=range(2)-range(1);
yrange=range(4)-range(3);
xn=rand(1,n)*xrange+range(1);
yn=rand(1,n)*yrange+range(3);
% Move all the particles toward (xo,yo) function
[xn,yn]=pso_move(xn,yn,xo,yo,a,b,range)
nn=size(yn,2); %a=alpha, b=beta xn=xn.*(1-b)
+xo.*b+a.*(rand(1,nn)-0.5); yn=yn.*(1-b)
+yo.*b+a.*(rand(1,nn)-0.5);
[xn,yn]=findrange(xn,yn,range);
116 Chapter 10. Particle Swarm Optimization

Figure 10.5: Michaelewicz function with a global minimum at

about (2.202, 1.571).

% Make sure the particles are within the range

function [xn,yn]=findrange(xn,yn,range)
nn=length(yn);
for i=1:nn,
if xn(i)<=range(1), xn(i)=range(1); end
if xn(i)>=range(2), xn(i)=range(2); end
if yn(i)<=range(3), yn(i)=range(3); end
if yn(i)>=range(4), yn(i)=range(4); end
end

Let us look at the Michaelewicz function (for two indepen-dent

variables)
2
x2 2y
f (x, y) = −{sin(x)[sin( 2m 2m
π )] + sin(y)[sin( π )] }, (10.16)
where m = 10. The stationary conditions fx = fy = 0 require that
4m x2 x2
− π x sin(x) cos( π ) − cos(x) sin( π ) = 0, (10.17)
and
2 2
8m 2y 2y
− π y sin(x) cos( π ) − cos(y) sin( π ) = 0. (10.18)
10.6 Constraints 117

Figure 10.6: Initial locations and final locations of 20 particles

after 10 iterations. The point marked with corresponds to the
best estimate (x∗, y∗).

∗
The solution at (0, 0) is trivial, and the minimum f ≈−1.801
occurs at about (2.202,1.571) (see Fig. 10.5).
If we run the program, we will get the global optimum after
about 200 evaluations of the objective function (for 20 particles
and 10 iterations). The results are shown in Fig. 10.6.

10.6 Constraints
The implementation we discussed in the previous section is for
unstrained problems. For constrained optimization, there are
many ways to implement the constraint equalities and inequal-
ities. However, we will only discussed two approaches: direct
implementation and transform to unconstrained optimization.
The simplest direct implementation is to check all the new
particle locations to see if they satisfy all the constraints. The
new locations are discarded if the constraints are not met, and
new locations are replaced by newly generated locations until
all the constraints are met. Then, the new solutions are eval-
uated using the standard PSO procedure. In this way, all the
new locations should be in the feasible region, and all infeasible
solutions are not selected. For example, in order to maximize f
(x) subjected to a constraint g(x) ≤ 0, the standard PSO
118 Chapter 10. Particle Swarm Optimization

as discussed earlier is used, however, the new locations x i of

the n particles are checked at each iteration so that they
must satisfy g(xi) ≤ 0. If any xi does not satisfy the constraint,
it is then replaced by a new (different) location x˜ i which
satisfies the constraint.
Alternatively, we can transform it to an unconstrained prob-
lem by using penalty method or Lagrange multipliers as dis-
cussed in Chapter 7. Using a penalty parameter ν 1 in our
simple example here, we have the following penalty function

2 (10.19)
Π(x, ν) = f (x) + νg(x) .

For any fixed value of ν which will determine the accuracy of

the corresponding solutions, we can then optimize Π as a
standard unstrained optimization problem.
There are other variations of particle swarm optimization,
and PSO algorithms are often combined with other existing
algorithms to produce new hybrid algorithms. In fact, it is still
an active area of research with many new studies published
each year.
Chapter 11

Simulated Annealing

11.1 Fundamental Concepts

Simulated annealing (SA) is a random search technique for
global optimization problems, and it mimics the annealing pro-
cess in material processing when a metal cools and freezes
into a crystalline state with minimum energy and larger crystal
size so as to reduce the defects in metallic structures. The an-
nealing process involves the careful control of temperature and
cooling rate (often called annealing schedule).
The application of simulated annealing into optimization
problems was pioneered by Kirkpatrick, Gelatt and Vecchi in
1983. Since then, there have been extensive studies. Unlike
the gradient-based methods and other deterministic search
meth-ods which have the disadvantage of be trapped into
local min-ima, the main advantage of the simulated
annealing is its ability to avoid being trapped in local minima.
In fact, it has been proved that the simulated annealing will
converge to its global optimality if enough randomness is
used in combination with very slow cooling.
Metaphorically speaking, this is equivalent to dropping some
bouncing balls over a landscape, and as the balls bounce and
loose energy, they settle down to some local minima. If the balls
are allowed to bounce enough times and loose energy slowly
enough, some of the balls will eventually fall into the global

119
120 Chapter 11. Simulated Annealing

lowest locations, hence the global minimum will be reached.

The basic idea of the simulated annealing algorithm is to
use random search which not only accepts changes that im-
prove the objective function, but also keeps some changes
that are not ideal. In a minimization problem, for example,
any better moves or changes that decrease the cost (or the
value) of the objective function f will be accepted, however,
some changes that increase f will also be accepted with a
probabil-ity p. This probability p, also called the transition
probability, is determined by
δE
− , (11.1)
p=e kT

where k is the Boltzmann’s constant, and T is the tempera-

ture for controlling the annealing process. δE is the change
of the energy level. This transition probability is based on the
Boltzmann distribution in physics. The simplest way to link
δE with the change of the objective function δf is to use

δE = γδf, (11.2)

where γ is a real constant. For simplicity without losing gen-

erality, we can use k = 1 and γ = 1. Thus, the probability p
simply becomes
δf
− . (11.3)
p(δf, T ) = e T

Whether or not we accept a change, we usually use a

random number r as a threshold. Thus, if p > r or
− δf
p=e T > r, (11.4)

it is accepted.

11.2 Choice of Parameters

Here the choice of the right temperature is crucially important.
For a given change δf , if T is too high (T → ∞), then p → 1,
which means almost all changes will be accepted. If T is too low
(T → 0), then any δf > 0 (worse solution) will rarely be
11.2 Choice of Parameters 121

accepted as p → 0 and thus the diversity of the solution is lim-

ited, but any improvement δf will almost always be accepted. In
fact, the special case T → 0 corresponds to the gradient-based
method because only better solutions are accepted, and system
is essentially climbing up or descending along a hill. Therefore,
if T is too high, the system is at a high energy state on the
topological landscape, and the minima are not easily reached. If
T is too low, the system may be trapped in a local minimum (not
necessarily the global minimum), and there is not enough
energy for the system to jump out the local mini-mum to explore
other potential global minima. So the proper temperature should
be calculated.
Another important issue is how to control the annealing
or cooling process so that the system cools down gradually
from a higher temperature to ultimately freeze to a global
minimum state. There are many ways of controlling the
cooling rate or the decrease of the temperature.
Two commonly used annealing schedules (or cooling
sched-ules) are: linear and geometric cooling. For a linear
cooling process, we have T = T0 − βt or T → T − δT , where T 0
is the initial temperature, and t is the pseudo time for iterations.
β is the cooling rate, and it should be chosen in such as way
that T → 0 when t → tf (maximum number of iterations), this
usually gives β = T0/tf .
The geometric cooling essentially decreases the temperature
by a cooling factor 0 < α < 1 so that T is replaced by αT or
t
T (t) = T0α , t = 1, 2, ..., tf . (11.5)

The advantage of the second method is that T → 0 when t →

∞, and thus there is no need to specify the maximum
number of iterations tf . For this reason, we will use this
geometric cooling schedule. The cooling process should be
slow enough to allow the system to stabilise easily. In
practise, α = 0.7 ∼ 0.9 is commonly used.
In addition, for a given temperature, multiple evaluations of
the objective function are needed. If too few evaluations, there
is a danger that the system will not stabilise and subse-
122 Chapter 11. Simulated Annealing

Simulated Annealing Algorithm

begin
T
Objective function f (x), x = (x1, ..., xp )
(0)
Initialize initial temperature T0 and initial guess x
Set final temperature Tf and max number of iterations N
Define cooling schedule T 7→αT , (0 < α < 1)
while ( T > Tf and n < N )
Move randomly to new locations: xn+1 = xn + randn
Calculate δf = fn+1(xn+1) − fn (xn )
Accept the new solution if better
if not improved
Generate a random number r
Accept if p = exp[−δf /kT ] > r
end if
Update the best x∗ and f∗
end while
end

Figure 11.1: Simulated annealing algorithm.

quently will not converge to its global optimality. If too many

evaluations, it is time-consuming, and the system will usually
converge too slowly as the number of iterations to achieve
sta-bility might be exponential to the problem size.
Therefore, there is a balance of the number of evaluations
and solution quality. We can either do many evaluations at a
few temper-ature levels or do few evaluations at many
temperature levels. There are two major ways to set the
number of iterations: fixed or varied. The first uses a fixed
number of iterations at each temperature, while the second
intends to increase the number of iterations at lower
temperatures so that the local minima can be fully explored.

11.3 SA Algorithm
The pseudo code of the simulated annealing algorithm is shown
in Fig. 11.1. In order to find a suitable starting temperature T 0,
we can use any information about the objective function. If we
11.4 Implementation 123
2

-1

-2

-2 -1 0 1 2
Figure 11.2: Contour of Rosenbrock function with a global
min-imum f∗ = 0 at (1, 1), and locations of the final 10
particles at the end of the simulated annealing.

know the maximum change max(δf ) of the objective function,

we can use this to estimate an initial temperature T0 for a
≈ −max(δf )

given probability p0. That is T0 ln p0 . If we do not know the

possible maximum change of the objective function, we can
use a heuristic approach. We can start evaluations from a
very high temperature (so that almost all changes are
accepted) and reduce the temperature quickly until about
50% or 60% the worse moves are accepted, and then use
this temperature as the new initial temperature T0 for proper
and relatively slow cooling processing.
For the final temperature, it should be zero in theory so that
no worse move can be accepted. However, if T f → 0, more
unnecessary evaluations are needed. In practice, we simply
−10 −5
choose a very small value, say, T f = 10 ∼ 10 , depending
on the required quality of the solutions and time constraints.

11.4 Implementation
Based on the guidelines of choosing the important parameters
such as cooling rate, initial and final temperatures, and the bal-
anced number of iterations, we can implement the simulated
124 Chapter 11. Simulated Annealing

Figure 11.3: 500 evaluations during the simulated annealing.

The final global best is marked with .

annealing using both Matlab and Octave. The implemented

Matlab and Octave program is given below. Some of the
sim-ulations of the related test functions are explained in
detail in the examples.

Example 11.1: For Rosenbrock’s function

2 2 2
f (x, y) = (1 − x) + 100(y − x ) ,
we know that its global minimum f ∗ = 0 occurs at (1, 1) (see
Fig. 11.2). This is a standard test function and quite tough for
most algorithms. However, using the program given below, we
can find this global minimum easily and the 500 evaluations
during the simulated annealing are shown in Fig. 11.3.

% Find the minimum of a function by Simulated Annealing

% Programmed by X S Yang (Cambridge University)
% Usage: sa_simpledemo
disp(‘Simulating ... it will take a minute or so!’);
% Rosenbrock test function with f*=0 at
(1,1) fstr=‘(1-x)^2+100*(y-x^2)^2’;
% Convert into an inline function
f=vectorize(inline(fstr));
% Show the topography of the objective
function range=[-2 2 -2 2];
11.4 Implementation 125

xgrid=range(1):0.1:range(2);
ygrid=range(3):0.1:range(4);
[x,y]=meshgrid(xgrid,ygrid);
surfc(x,y,f(x,y));
% Initializing parameters and settings
T_init = 1.0; % initial temperature
T_min = 1e-10; % finial stopping temperature
% (eg., T_min=1e-10)
F_min = -1e+100; % Min value of the function
max_rej=500; % Maximum number of rejections
max_run=100; % Maximum number of runs
max_accept = 15; % Maximum number of accept
k = 1; % Boltzmann constant
alpha=0.9; % Cooling factor
Enorm=1e-5; % Energy norm (eg, Enorm=1e-8)
guess=[2 2]; % Initial guess
% Initializing the counters
i,j etc i= 0; j = 0;
accept = 0; totaleval = 0;
% Initializing various values
T = T_init;
E_init = f(guess(1),guess(2)); E_old
= E_init; E_new=E_old; best=guess; %
initially guessed values % Starting
the simulated annealling
while ((T > T_min) & (j <= max_rej) &
E_new>F_min) i = i+1;
% Check if max numbers of run/accept are met
if (i >= max_run) | (accept >= max_accept)
% Cooling according to a cooling schedule
T = alpha*T;
totaleval = totaleval + i;
% reset the counters
i = 1; accept = 1;
end
% Function evaluations at new
locations ns=guess+rand(1,2)*randn;
E_new = f(ns(1),ns(2));
% Decide to accept the new
solution DeltaE=E_new-E_old;
% Accept if improved
if (-DeltaE > Enorm)
126 Chapter 11. Simulated Annealing

Figure 11.4: Egg crate function with a global minimum f∗ = 0

at (0, 0).

best = ns; E_old = E_new;

accept=accept+1; j = 0;
end
% Accept with a small probability if not improved
if (DeltaE<=Enorm & exp(-DeltaE/(k*T))>rand );
best = ns; E_old =
E_new; accept=accept+1;
else
j=j+1;
end
% Update the estimated optimal solution
f_opt=E_old;
end
% Display the final results disp(strcat(‘Obj
function :’,fstr)); disp(strcat(‘Evaluations :’,
num2str(totaleval))); disp(strcat(‘Best
location:’, num2str(best))); disp(strcat(‘Best
estimate:’, num2str(f_opt)));

Example 11.2: Using the above program, we can also simulate a

more complicated function, often called the egg crate function,
whose global minimum f∗ = 0 is at (0, 0). The objective function
11.4 Implementation 127

Figure 11.5: The paths of particles during simulated annealing.

for two variables is

2 2 2 2
f (x, y) = x + y + 25[sin (x) + sin (y)],

in the domain (x, y) ∈ [−5, 5] × [−5, 5]. The landscape of the egg
crate function is shown in Fig. 11.4, and the paths of par-ticles
during simulated annealing are shown in Fig. 11.5. It is worth
pointing that the random generator we used in the program
rand(1,2)<1/2 leads to discrete motion along several major di-
rections, which may improve the convergence for certain class of
functions. However, for the Rosenbrock test function, this dis-crete
approach does not work well. For continuous movement in all
directions, simply use the random function rand(1,2) that is given in
this simple demo program. It would takes about 2500 evaluations
to get an accuracy with three decimal places.
Chapter 12

Multiobjective
Optimization

All the optimization problems we discussed so far have only a

single objective. In reality, we often have to optimize multiple
objectives simultaneously. For example, we may want to im-
prove the performance of a product while trying to minimize the
cost at the time. In this case, we are dealing with multiob-jective
optimization problems. Many new concepts have to be
introduced for multiobjective optimization.

12.1 Pareto Optimality

Any multiobjective optimization problem can generally be
writ-ten as
minimize/maximize f (x) = [f1(x), f2(x), ..., fp(x)],
x <n
subject to g (x) ≥ 0, j = 1, 2, ..., M, (12.1)
j

hk (x) = 0, k = 1, 2, ..., N, (12.2)

T
where x = (x1, x2, ..., xn) is the vector of decision variables.
In some formulations used in the literature, inequalities g j (j =
1, ..., N ) can also include any equalities because an equality
φ(x) = 0 can be converted into two inequalities φ(x) ≤ 0 and
129
130 Chapter 12. Multiobjective Optimization

φ(x) ≥ 0. However, for clarity, we list here the equalities and

inequalities separately.
n
The space F = < spanned by the vectors of decision
p
vari-ables x is called the search space. The space S = <
formed by all the possible values of objective functions is
called the solution space or objective space. Comparing with
the sin-gle objective function whose solution space is (at
most) <, the solution space for multiobjective optimization is
considerably much larger. In addition, as we know that we
are dealing with multiobjectives f (x) = [fi], for simplicity, we
can write fi as f (x) without causing any confusion.
Multiobjective optimization problems, unlike a single
objec-tive optimization problem, do not necessarily have an
optimal solution that minimizes all the multiobjective
functions simul-taneously. Often, different objectives may
conflict each other and the optimal parameters of some
objectives usually do not lead to optimality of other
objectives (sometimes make them worse). For example, we
want first-class quality service on our holidays and at the
same time we want to pay as little as pos-sible. The high-
quality service (one objective) will inevitably cost much more,
and this is in conflict with the other objective (minimize cost).
Therefore, among these often conflicting objectives, we
have to choose some tradeoff or a certain balance of
objectives. If none of these are possible, we must choose a list
of preferences so that which objectives should be achieved
first. More im-portantly, we have to compare different objectives
and make a compromise. This usually requires a formulation of
a new evaluation modelling problem, and one of the most
popular approaches to such modelling is to find a scalar-valued
func-tion that represents weighted combinations or preference
order of all objectives. Such a scalar function is often referred to
as the preference function or utility function. A simple way to
construct this scalar function is to use the weighted sum

p
X

u(f1(x), ..., fp(x)) =αifi(x), (12.3)

i=1
12.1 Pareto Optimality 131

where αi are the weighting coefficients.

n
A point x∗ ∈ < is called a Pareto optimal solution or
non-inferior solution to the optimization problem if there is no
n
x ∈ < satisfying fi(x) ≤ fi(x∗), (i = 1, 2, ..., p). In other
words, x∗ is Pareto optimal if there exists no feasible vector
(of decision variables in the search space) which would
decrease some objectives without causing an increase in at
least one other objective simultaneously.
Unlike the single objective optimization with a single opti-
mal solution, multiobjective optimization will lead to a set of
∗
solutions, called the Pareto optimal set P , and the decision
vectors x∗ for the set solutions are thus called non-dominated. The
set x∗ in the search space that corresponds to the Pareto optimal
solutions is also an efficient set in literature. The set (or plot) of the
objective functions of these non-dominated deci-sion vectors in the
Pareto optimal set forms the so-called Pareto
front P (or Pareto frontier). Now let us define the dominance
of a vector x (or any vectors u and v).
T
A vector u = (u1, .., un) ∈ F is said to dominate vector
T
v = (v1, ..., vn) if and only if u i ≤ vi for ∀i ∈ {1, ..., n} and ∃i
∈ {1, ..., n} : ui < vi. This ‘partial less’ relationship is denoted
by
uv, (12.4)

which is equivalent to

∀i ∈ {1, ..., n} : ui ≤ vi ∧ ∃i ∈ {1, ..., n} : ui < vi. (12.5)

Similarly, we can define another dominance relationship

uv ⇐⇒ uv ∨ u = v. (12.6)

Using these notations, the Pareto front P can be defined as the

set of non-dominated solutions so that
0 0
P = {s ∈ S ∃/ s ∈ S : s s}, (12.7)

or in term of the Pareto optimal set in the search space

∗ 0 0 (12.8)
P = {x ∈ F ∃/ x ∈ F : f (x ) f (x)}.
132 Chapter 12. Multiobjective Optimization

Furthermore, a point x∗ ∈ F is called a non-dominated

so-lution if no solution can be found that dominates it. A
vector is called ideal if it contains the decision variables that
correspond to the optima of objectives when each objective
is considered separately.
The identification of the Pareto front is not an easy task, and
it often requires a parametric analysis, say, by treating all but
one objective, say, fi in a p-objective optimization problem so
that fi is a function of f1, ..., fi−1, fi+1, ..., and fp. By maximizing
the fi when varying the values of the other p − 1 objectives so
that the solutions will trace out the Pareto front.

Example 12.1: For example, we have four Internet service

providers A, B, C and D. We have two objectives to choose
their service 1) as cheap as possible, and 2) higher
bandwidth. They are listed below:

IP provider Cost (£) Bandwidth (Mb)

A 20 12
B 25 16
C 30 8
D 40 16

From the table, we know that option C is dominated by A and B

because both objectives are improved (low cost and faster). Option
D is dominated by B. Thus, solution C is an inferior solu-tion, so is
D. Both solutions A and B are non-inferior solutions or non-
dominated solutions. However, which solution ( A or B) to choose is
not easy as provider A outperforms B on the first objective
(cheaper) while B outperforms A on another objective (faster). In
this case, we say these two solutions are incomparable. The set of
the non-dominated solutions A and B forms the Pareto front which
is a mutually incomparable set.

For a minimization problem with two objectives, the basic concepts

of non-dominated set, Pareto front, and ideal vectors are shown in Fig.
12.1.
12.2 Weighted Sum Method 133
f2 ideal vectors for f1

dominated set
x0

Pareto
front non-dominated
f1

Figure 12.1: Non-dominated set, Pareto front and ideal vectors

in a minimization problem with two objectives f1 and f2.

12.2 Weighted Sum Method

The classical three-objective functions are commonly used
for testing multi-objective optimization algorithms. These
func-tions are
2 2 (12.9)
f (x, y) = x + (y − 1) ,
1
2 2 (12.10)
f2(x, y) = (x − 1) + y + 2,
2 2 (12.11)
f3(x, y) = x + (y + 1) + 1,
where (x, y) ∈ [−2, 2] × [−2, 2].
Many solution algorithms intend to combine the multi-
objective functions into one scalar objective using the weighted
sum of objective functions
F (x) = αf (x) + βf (x) + ... + γf (x). (12.12)
1 2 N

The important issue arises in assigning the weighting coeffi-

cients (α, β, ..., γ) because the solution is strongly dependent
on the chosen weighting coefficients.
If we combine all the three functions into one f (x, y)
using weighted sums, we have
f (x, y) = αf1 + βf2 + γf3, α + β + γ = 1. (12.13)

The stationary point is determined by

∂f = 0, ∂f = 0, (12.14)
∂x ∂y
134 Chapter 12. Multiobjective Optimization

Figure 12.2: Three-objective functions with the global mini-

mum at x∗ = β, y∗ = α − γ.

which lead to
2α + 2β(x − 1) + 2γ = 0, (12.15)
and
2α(y − 1) + 2βy + 2γ(y + 1) = 0. (12.16)
The solutions are
(12.17)
x∗ = β,y∗ = α − γ.

This implies that x∗ ∈ [0, 1] and y∗ ∈ [−1, 1]. Consequently,

f1 ∈ [0, 5], f2 ∈ [2, 4] and f3 ∈ [1, 6]. In addition, the solution
or the optimal location varies with the weighting coefficients
α, β and γ. In the simplest case α = β = γ = 1/3, we have

1 (12.18)
x∗ = ,y∗ = 0.
3

This location is marked with a solid bar in Figure 12.2.

Now the original multiobjective optimization problem has been
transformed into a single objective optimization problem. Thus, the
solution methods for solving single objective prob-lems are all valid. For
example, we can use the particle swarm
12.2 Weighted Sum Method 135

Figure 12.3: Final locations of 40 particles after 5 iterations.

The optimal point is at (1/3, 0) marked with ◦.

to find the optimal for given parameters α, β and γ. Figure

12.3 shows the final locations of 40 particles at t = 5
iterations. We can see that the particles converge towards
the true optimal location marked with ◦.
However, there is an important question to be answered.
The combined weighted sum transforms the optimization prob-
lem into a single objective, this is not necessarily equivalent to
the original multiobjective problem because the extra weight-ing
coefficients could be arbitrary, while the final solutions still
depend on these coefficients. Furthermore, there are so many
ways to construct the weighted sum function and there is not
any easy guideline to choose which form is the best for a given
problem. When there is no rule to follow, the simplest choice
obviously is to use the linear form. But there is no reason why
the weighted sum should be linear. In fact, we can use other
combinations such as the following quadratic weighted sum

N
X 2 2 2
Π(x) = αifi (x) = α1f1 (x) + ... + αN fN (x), (12.19)
i=1

and the others.

136 Chapter 12. Multiobjective Optimization

Another important issue is that how to choose the

weighting coefficients as the solutions depend on these
coefficients. The choice of weighting coefficients is
essentially to assign a prefer-ence order by the decision
maker to the multiobjectives. This leads to a more general
concept of utility function (or pref-erence function) which
reflects the preference of the decision maker(s).

12.3 Utility Method

The weighted sum method is essentially a deterministic

value method if we consider the weighting coefficients as
the ranking coefficients. This implicitly assumes that the
consequence of each ranking alternative can be
characterized with certainty. This method can be used to
explore the implications of alter-native value judgement.
Utility method, on the other hand, considers uncertainty in
the criteria values for each alterna-tive, which is a more
realistic method because there is always some degree of
uncertainty about the outcome of a particular alternative.
Utility (or preference) function can be associated with risk
attitude or preference. For example, if you are offered a choice
between a guaranteed £500 and a 50/50 chance of zero and
£1000. How much are you willing to pay to take the gamble?
The expected payoff of each choice is £500 and thus it is fair to
pay 0.5 × 1000 + (1 − 0.5) × 0 = £500 for such a gamble. A risk-
seeking decision maker would risk a lower payoff in order to
have a chance to win a higher prize, while a risk-averse
decision maker would be happy with the safe choice of £500.
For a risk-neutral decision maker, the choice is indifferent
between a guaranteed £500 and the 50/50 gamble since both
choices have the same expected value of £ 500. In reality, the risk
preference can vary from person to person and may depend on the type
of problem. The utility function can have many forms, and one of the
simplest is the exponential utility (of
12.3 Utility Method 137
f2

utility (U )
A

Pareto front increase

feasible region
f1

Figure 12.4: Finding the Pareto solution with maximum utility

in a maximization problem with two objectives.

representing preference)
u(x) = , (12.20)
1 − e−(x−xa )/ρ
1 − e−(xb −xa )/ρ
where xa and xb are the lowest and highest level of x, and ρ
is called the risk tolerance of the decision maker.
The utility function defines combinations of objective
values f1, ..., fp which a decision maker finds equally
acceptable or indifference. So the contours of the constant
utility are referred to as the indifference curves. The
optimization now becomes the maximization of the utility. For
a maximization problem with two objectives f 1 and f2, the
idea of the utility contours (indifference curves), Pareto front
and the Pareto solution with maximum utility (point A) are
shown in Fig. 12.4. When the utility function touches the
Pareto front in the feasible region, it then provides a
maximum utility Pareto solution (marked with A).
For two objectives f1 and f2, the utility function can be
con-structed in different ways. For example, the combined
product takes the following form
α β , (12.21)
U (f1 , f2) = kf f
1 2
where α and β are non-negative exponents and k a scaling
138 Chapter 12. Multiobjective Optimization

factor. The aggregated utility function is given by

U (f1, f2) = αf1 + βf2 + [1 − (α + β)]f1f2. (12.22)

There are many other forms. The aim of utility function con-
p
structed by the decision maker is to form a mapping U : < 7→
< so that the total utility function has a monotonic and/or
convexity properties for easy analysis. It will also improve
the quality of the Pareto solution(s) with maximum utility. Let
us look at a simple example.

Example 12.2: For the simple two-objective optimization

prob-lem:
maximize
f
2 1
(x, y) = x + y,f2(x, y) = x,
(x,y) <
subject to
x + αy ≤ 5,x ≥ 0, y ≥ 0,
where 0 < α < 1. Let us use the simple utility function

U = f1f2,

which combines the two objectives. The line connecting the two
corner points (5, 0) and (0, 5/α) forms the Pareto front (see Fig.
12.5). It is easy to check that the Pareto solution with maximum
utility is U = 25 at A(5, 0) when the utility contours touch the
Pareto front with the maximum possible utility.
The complexity of multiobjective optimization makes the
construction of the utility function a difficult task as it can be
constructed in many ways. A general and yet widely-used
utility function is often written in the following additive form
maximize K p
παu ,
XX

x <nU (fk (x)) = j i ijk (12.23)

j=1 i=1

where U (fk ) is the expected utility of alternative k. p is the

number of objectives and K is the number of possible
scenarios. πj is the probability assigned to scenario j. uijk is
the value of a single criterion utility function for objective i,
scenario j and alternative k.
 12.4 Metaheuristic Search 139

Pareto front
B
B
U= f1f2
feasible set

Figure 12.5: Pareto front is the line connecting A(5, 0) and

B(0, 5/α). The Pareto solution with maximum utility is U ∗ =
25 at point A.

12.4 Metaheuristic Search

So far, we have seen that finding solutions of multiobjective
optimization is usually difficult, even by the simple weighted
sum method and utility function. However, there are other
promising methods that work well for multiobjective optimiza-
tion problems, especially the metaheuristic methods such as
simulated annealing and particle swarm optimization.
Here we will use the particle swarm optimization to find
solutions for multiobjective optimization problems. The basic
idea is to modify the standard PSO algorithms so that they
can deal with the multiobjectives at the same time. This is
better demonstrated by an example.
From the multimodal test function (10.8), we know that it
has a single global maximum. If we extend its domain to a
large set, then it will have two global maxima. Now we have
2 2 2
sin(x + y )
f (x, y) = −λ(x−y)
p 2 2 e , (x, y) ∈[−5, 5]×[−5, 5],
x +y
where λ > 0. For λ = 0.05, the function is shown in Fig. 12.6.
This function has two equal global maxima at (0.7634, 0.7634)
and (−0.7634, −0.7634). If we run the standard PSO, it will only
find one of the two global optima. In order to find all
140 Chapter 12. Multiobjective Optimization

Figure 12.6: Multimodal function with two equal global max-

ima.

the (two) global optima, we have to modify the PSO. In fact,

there are many ways to do this and one of the simplest way
to achieve this is to use the recursive method.
In the recursive method, we call the standard PSO many
times. As the initial starting point is a random guess, it will find
one optimum each time we call it. If we call the PSO re-
cursively, statistically speaking, all the optima will be founded.
For our test function here, we called the accelerated PSO for
1000 times. It found one peak at (0.7634,0.7634) in 497 runs,
the other peak in 498 runs, and trapped at local peaks in 5
runs. Two snapshots of the 1000 runs were shown in Fig. 12.7
so that all optima are found statistically.
The implementation of this recursive PSO using Matlab and
Octave is given below. Since it is a recursive process, it will
take a few minutes (depending on the speed of the computer).

% The Recursive PSO for Multiobjective Optimization

% (written by X S Yang, Cambridge University)
% Usage: pso_multi(number_of_runs)
% eg: best=pso_multi(100);
12.4 Metaheuristic Search 141

Figure 12.7: Recursive PSO runs converge to two global peaks.

% where best <=> locations of global optima

function [best]=pso_multi(Num_run)
% Num_run=number of recursive iterations (=10 default)
if nargin<1, Num_run=10; end
disp(‘Running PSO recursively!’);
disp(‘Please wait for a few minutes ...’);
n=20; % number of particles;
Num_steps=10; % number of pseudo time steps
% This function has two global optima f*=0.851
% at (0.7634,0.7634) and (-0.7634,-0.7634).
fstr=‘sin(x^2+y^2)/sqrt(x^2+y^2)*exp(-0.05*(x-y)^2)’
% Converting to an inline function
f=vectorize(inline(fstr));
% range=[xmin xmax ymin ymax];
range=[-5 5 -5 5];
% -------------------------------------------------
% Grid values of the objective function
% These values are used for
visualization only Ngrid=100;
dx=(range(2)-range(1))/Ngrid;
dy=(range(4)-range(3))/Ngrid;
xgrid=range(1):dx:range(2);
142 Chapter 12. Multiobjective Optimization

ygrid=range(3):dy:range(4);
[x,y]=meshgrid(xgrid,ygrid);
z=f(x,y);
% Display the shape of the function to be
optimized surfc(x,y,z); drawnow;
% Run the PSO recursively for, say, 10 times
for i=1:Num_run,
best(i,:)=pso(f,range,n,Num_steps);
end
% ------------------------------------------------
% Standard Accelerated PSO (for finding maxima)
function [best]=pso(f,range,n,Num_steps)
% here best=[xbest ybest fbest]
% Speed of convergence (0->1)=(slow-
>fast) beta=0.5;
% ----- Start Particle Swarm Optimization --------
% generating the initial locations of n
particles [xn,yn]=init_pso(n,range);
% Iterations as pseudo time
for i=1:Num_steps,
% Find the current best location
(xo,yo) zn=f(xn,yn);
zn_max=max(zn);
xo=max(xn(zn==zn_max));
yo=max(yn(zn==zn_max));
zo=max(zn(zn==zn_max));
% Accelerated PSO with randomness:
alpha=gamma^t gamma=0.7; alpha=gamma.^i;
% Move all particle to new locations
[xn,yn]=pso_move(xn,yn,xo,yo,alpha,beta,range);
end %%%%% end of iterations
% Return the finding as the current best
best(1)=xo; best(2)=yo; best(3)=zo;
% -----------------------------------------------
% All subfunctions are listed here
% Intial locations of particles
function [xn,yn]=init_pso(n,range)
12.5 Other Algorithms 143

xrange=range(2)-range(1);
yrange=range(4)-range(3);
xn=rand(1,n)*xrange+range(1);
yn=rand(1,n)*yrange+range(3);
% Move all the particles toward to (xo,yo)
function [xn,yn]=pso_move(xn,yn,xo,yo,a,b,range)
nn=size(yn,2); %%%%% a=alpha, b=beta
xn=xn.*(1-b)+xo.*b+a.*(rand(1,nn)-0.5);
yn=yn.*(1-b)+yo.*b+a.*(rand(1,nn)-0.5);
[xn,yn]=findrange(xn,yn,range);
% Make sure the particles are inside the range
function [xn,yn]=findrange(xn,yn,range)
nn=length(yn);
for i=1:nn,
if xn(i)<=range(1), xn(i)=range(1); end
if xn(i)>=range(2), xn(i)=range(2); end
if yn(i)<=range(3), yn(i)=range(3); end
if yn(i)>=range(4), yn(i)=range(4); end
end

12.5 Other Algorithms

There are other powerful algorithms that we have not
addressed in this book. These include genetic algorithms,
virtual bee algo-rithms, harmony search, random-restart hill
climbing, dynamic programming, stochastic optimization,
evolution strategy and many other evolutionary algorithms.
For example, genetic al-gorithms (and their stochastic
various variants) have been used to solve many practical
optimization problems such as schedul-ing problems and
engineering design problems. Readers inter-ested in these
modern techniques can refer to more advanced literature.
The optimization algorithms we have discussed in this book
are mainly for the optimization problems with explicit objective
functions. However, in reality, it is often difficult to quantify what
we want to achieve, but we still try to optimize certain things
such as the degree of enjoyment of a quality service on
144 Chapter 12. Multiobjective Optimization

our holidays. In other cases, it is not possible and/or there is no

explicit form of objective function. For example, we want to
design an aircraft wing so that it is most aerodynamically
efficient. If we know nothing about the modern aircraft de-sign,
suppose we have to start from scratch, what shape it should be
and how to quantify the efficiency (lift or some ratio of power to
lift)? What is the explicit form of the objective function? This is a
very difficult task. One approach is to try various shapes by
carrying out experimental tests of these var-ious shapes.
Alternatively, we can use computer simulations (e.g.,
computational fluid dynamics) for a given shape. Start-ing from
an initial shape (say, an ellipsoid), what is the new shape we
should generate? We usually have to run many itera-tions of
computer simulations, changing design (both computer models
and physical models) and evaluating their performance
(objectives) again and again.
Whatever the objectives are, we have to evaluate the ob-
jectives many times. In most cases, the evaluations of the
ob-jective functions consume a lot of computational power
(which costs money) and design time. Any efficient algorithm
that can reduce the number of objective evaluations will save
both time and money. Although, we have mainly focused on
the opti-mization algorithms for objectives which are
explicitly known, however, these algorithms will still be
applicable to the cases where the objectives are not known
explicitly. In most cases, certain modifications are required to
suit a particular appli-cation. This is an exciting area of active
research, and more publications are emerging each year.
Bibliography

[1] Abramowitz M. and Stegun I. A., Handbook of Mathematical

Functions, Dover Publication, (1965).
[2] Adamatzky A., Teuscher C., From Utopian to Genuine Uncon-
ventional Computers, Luniver Press, (2006).
[3] Arfken G., Mathematical Methods for Physicists, Academic
Press, (1985).
[4] Atluri S. N., Methods of Computer Modeling in Engineering
and the Sciences, Vol. I, Tech Science Press, (2005).
[5] Bonabeau E., Dorigo M., Theraulaz G., Swarm Intelligence:
From Natural to Artificial Systems. Oxford University Press,
(1999)
[6] Bonabeau E. and Theraulaz G., Swarm smarts, Scientific
Amer-icans. March, 73-79 (2000).
[7] Bridges D. S., Computatability, Springer, New York, (1994).
[8] Chatterjee A. and Siarry P., Nonlinear inertia variation for dy-
namic adapation in particle swarm optimization, Comp. Oper.
Research, 33, 859-871 (2006).
[9] Coello C. A., Use of a self-adaptive penalty approach for engi-
neering optimization problems, Computers in Industry, 41, 113-
127 (2000).
[10] Courant R. and Hilbert, D., Methods of Mathematical Physics,
2 volumes, Wiley-Interscience, New York, (1962).
[11] Davis M., Computability and Unsolvability, Dover, (1982).
[12] De Jong K., Analysis of the Behaviour of a Class of Genetic
Adaptive Systems, PhD thesis, University of Michigan, Ann Ar-
bor, (1975).

145
146 BIBLIOGRAPHY

[13] Deb K., An efficient constraint handling method for genetic al-
gorithms, Comput. Methods Appl. Mech. Engrg., 186, 311-338
(2000).
[14] Deb. K., Optimization for Engineering Design: Algorithms and
Examples, Prentice-Hall, New Delhi, (1995).

[15] Dorigo M., Optimization, Learning and Natural Algorithms,

PhD thesis, Politecnico di Milano, Italy, (1992).

[16] Dorigo M. and St¨utzle T., Ant Colony Optimization, MIT Press,
Cambridge, (2004).
[17] El-Beltagy M. A., Keane A. J., A comparison of various opti-
mization algorithms on a multilevel problem, Engin. Appl. Art.
Intell., 12, 639-654 (1999).
[18] Engelbrecht A. P., Fundamentals of Computational Swarm In-
telligence, Wiley, (2005).
[19] Flake G. W., The Computational Beauty of Nature: Computer
Explorations of Fractals, Chaos, Complex Systems, and
Adapta-tion, Cambridge, Mass.: MIT Press, (1998).
[20] Forsyth A. R., Calculus of Variations, Dover (1960).
[21] Fowler A. C., Mathematical Models in the Applied Sciences,
Cambridge University Press, (1997).
[22] Gardiner C. W., Handbook of Stochastic Methods, Springer,
(2004).
[23] Gershenfeld N., The Nature of Mathematical Modeling, Cam-
bridge University Press, (1998).
[24] Gill P. E., Murray W., and Wright M. H., Practical optimization,
Academic Press Inc, (1981).
[25] Glover F., Heuristics for Integer Programming Using Surrogate
Constraints, Decision Sciences, 8, 156-166 (1977).
[26] Glover F., Tabu Search - Wellsprings and Challenges, Euro. J.
Operational Research, 106, 221-225 (1998).
[27] Glover F. and Laguna M., Tabu Search, Kluwer Academic
Pub-lishers, (1997).
[28] Goldberg D. E., Genetic Algorithms in Search, Optimization and
Machine Learning, Reading, Mass.: Addison Wesley (1989).
[29] Goodman R., Teach Yourself Statistics, London, (1957).
BIBLIOGRAPHY 147

[30] Holland J., Adaptation in Natural and Artificial Systems, Uni-

versity of Michigan Press, Ann Anbor, (1975).
[31] Jaeggi D., Parks G. T., Kipouros T., Clarkson P. J.: A multi-
objective Tabu search algorithm for constrained optimization
problem, 3rd Int. Conf. Evolutionary Multi-Criterion Optimiza-
tion, Mexico, 3410, 490-504 (2005).
[32] Jeffrey A., Advanced Engineering Mathematics, Academic
Press, (2002).
[33] John F., Partial Differential Equations, Springer, New York,
(1971).
[34] Keane A. J., Genetic algorithm optimization of multi-peak prob-
lems: studies in convergence and robustness, Artificial Intelli-
gence in Engineering, 9, 75-83 (1995).

[35] Kennedy J. and Eberhart R. C.: Particle swarm optimization.

Proc. of IEEE International Conference on Neural Networks,
Piscataway, NJ. pp. 1942-1948 (1995).
[36] Kennedy J., Eberhart R., Shi Y.: Swarm intelligence, Academic
Press, (2001).
[37] Korn G. A. and Korn T. M., Mathematical Handbook for Scien-
tists and Engineers, Dover Publication, (1961).
[38] Kirkpatrick S., Gelatt C. D., and Vecchi M. P., Optimization by
simulated annealing, Science, 220, No. 4598, 671-680 (1983).
[39] Kreyszig E., Advanced Engineering Mathematics, 6th Edition,
Wiley & Sons, New York, (1988).
[40] Kuhn H. W. and Tucker A. W., Nonlinear programming, Proc.
2nd Berkeley Symposium, pp. 481-492, University of
California Press, (1951).
[41] Marco N., Lanteri S., Desideri J. A., P´eriaux J.: A parallel
genetic algorithm for multi-objective optimization in CFD, in:
Evolutionary Algorithms in Engineering and Computer Science
(eds Miettinen K. et al), Wiley & Sons, (1999).
[42] Matlab info, http://www.mathworks.com
[43] Michaelewicz, Z., Genetic Algorithm + Data Structure = Evo-
lution Progamming, New York, Springer, (1996).
[44] Mitchell, M., An Introduction to Genetic Algorithms, Cam-
bridge, Mass: MIT Press, (1996).
[45] Octave info, http://www.octave.org
148 BIBLIOGRAPHY

[46] Pearson C. E., Handbook of Applied Mathematics, 2nd Ed,

Van Nostrand Reinhold, New York, (1983).
[47] Press W. H., Teukolsky S. A., Vetterling W. T., Flannery B. P.,
Numerical Recipe in C++: The Art of Scientific Computing,
Cambridge University Press, (2002).
[48] Riley K. F., Hobson M. P., and Bence S. J., Mathematical
Meth-ods for Physics and Engineering, 3rd Edition, Cambridge
Uni-versity Press (2006).
[49] Sawaragi Y., Nakayama H., Tanino T., Theory of Multiobjective
Optimization, Academic Press, (1985).
[50] Sirisalee P., Ashby M. F., Parks G. T., and Clarkson P. J.:
Multi-criteria material selection in engineering design, Adv.
Eng. Mater., 6, 84-92 (2004).
[51] Selby S. M., Standard Mathematical Tables, CRC Press, (1974).
[52] Smith D. R., Variation Methods in Optimization, New York,
Dover, (1998).
[53] Spall J. C., Introduction to Stochastic Search and optimiza-
tion: Estimation, Simulation, and Control, Wiley, Hoboken, NJ,
(2003).
[54] Swarm intelligence, http://www.swarmintelligence.org
[55] Weisstein E. W., http://mathworld.wolfram.com
[56] Wikipedia, http://en.wikipedia.com
[57] Wolpert D. H. and Macready W. G., No free lunch theorems
for optimization, IEEE Transaction on Evolutionary
Computation, 1, 67-82 (1997).
[58] Wylie C. R., Advanced Engineering Mathematics, Tokyo, (1972).
[59] Yang X. S., Engineering optimization via nature-inspired virtual
bee algorithms, Lecture Notes in Computer Science, 3562,
317-323 (2005).
[60] Yang X. S., Biology-derived algorithms in engineering optimiza-
tion (Chapter 32), in Handbook of Bioinspired Algorithms, edited
by Olarius S. & Zomaya A., Chapman & Hall / CRC, (2005).
[61] Yang X. S., Lees J. M., Morley C. T.: Application of virtual ant
algorithms in the otpimization of CFRP shear strengthened
pre-cracked structures, Lecture Notes in Computer Sciences,
3991, 834-837 (2006).
[62] Yang X. S., Applied Engineering Mathematics, Cambridge In-
ternational Science Publishing, (2007).
Index

p-norm, 11 global optimality, 121

gradient vector, 20 gradient-
Accelerated PSO, 109 based method, 59, 89
algorithmic complexity, 9
annealing, 119 annealing Hessian matrix, 20
schedule, 119 ant colony Hilbert-Schmidt norm, 13
optimization, 99 hill-climbing method, 63
Hooke-Jeeves, 64
bisection method, 27
Boltzmann distribution, 120 ideal vector, 132 inertia
function, 109 initial
canonical form, temperature, 120 integer
73 complexity progamming, 89
algorithmic, 7, 9 iteration method, 30, 45
computational, 7, 9 Gauss-Seidel, 50
linear, 9 relaxation method, 51
quadratic, 9
constrained optimization, 4 Jacobi iteration, 45
convexity, 22
cooling process, 121 Kuhn-Tucker condition, 83
cooling rate, 119
Lagrange multiplier, 81
decision-maker, 136 linear programming, 4,
double bridge problem, 103 67 linear system, 35
LU decomposition, 43
egg crate function, LU factorization, 43
126 eigenvalue, 14
eigenvector, 14 mathematical optimization, 3
exploratory move, 64 Matlab, 113
matrix norm, 13
feasible solution, 68 metaheuristic method, 89, 99,
Frobenius norm, 13 139 minimum energy, 119
multimodal function, 111
global minima, 24 multiobjective optimization, 129

149
150 INDEX

Newton’s method, 29, 59 random search, 120

Newton-Raphson, 29, 31, 52 recursive PSO, 140 risk-
No free lunch theorems, 84 averse, 136 risk-neutral,
non-dominated solution, 132 136 risk-seeking, 136
non-inferior solution, 131, 132 root-finding algorithm, 25
nonlinear equation, 51
nonlinear optimization, 4, 79
nonlinear programming, 22, SA, 119
79 NP-complete problem, 9 algorithm, 122
NP-hard problem, 10 implementation, 123
semi-definite matrix, 17
objective function, 6 simple iteration, 25 simplex
Octave, 113 method, 70, 73 simulated
optimality, 73, 129 annealing, 89, 119 slack
optimality criteria, 6 variable, 70 solution quality,
optimization, 4 122
linear, 4 spectral radius, 18
multivariate, 56 square matrix, 15
nonlinear, 4 stability, 122
univariate, 55 stationary point, 57
order notation, 7 steepest descent method, 60
big O, 7 Stirling’s series, 8
small o, 8 strong maxima, 6
strong minima, 6
P-problem, 9
Pareto front, 131, 132 Tabu list, 90
Pareto optimal solution, 131 Tabu search, 89, 98
Pareto optimality, 129 travelling salesman problem,
Pareto solution, 137 10, 93
particle swarm optimization, 89,
107 unconstrained optimization, 4,
pattern move, 64 pattern 55 utility function, 136 utility
search, 64 penalty method, 136
method, 79 positive
definite matrix, 17 vector norm, 11
preference function, 136 virtual ant algorithm, 105
pseudo code, 122 PSO,
107 weak maxima, 6
implementation, 113 weak minima, 6
multimodal, 111 weighted sum method, 133

quadratic function, 21, 60

quasi-Newton method, 61