See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/226005181

Semi-Infinite Programming and Applications to Minimax Problems

Article  in  Annals of Operations Research · January 2003

DOI: 10.1023/B:ANOR.0000004764.76984.30 · Source: DBLP

CITATIONS READS

40 241

2 authors:

Stanislav Zakovic Berç Rustem

ABN AMRO Imperial College London
22 PUBLICATIONS   191 CITATIONS    198 PUBLICATIONS   3,420 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Worst case analysis in finance and engineering View project

All content following this page was uploaded by Stanislav Zakovic on 08 June 2017.

The user has requested enhancement of the downloaded file.

 Annals of Operations Research 124, 81–110, 2002
 2002 Kluwer Academic Publishers. Manufactured in The Netherlands.

Semi-Infinite Programming and Applications to

Minimax Problems
STANISLAV ŽAKOVIĆ [email protected]
Centre for Process Systems Engineering, Imperial College of Science, Technology and Medicine,
London SW7 2BY, UK

BERC RUSTEM ∗ [email protected]

Department of Computing, Imperial College of Science, Technology and Medicine,
London SW7 2BZ, UK

Abstract. A minimisation problem with infinitely many constraints – semi-infinite programming problem
(SIP) is considered. The problem is solved using a two stage procedure that searches for global maximum
violation of the constraints. A version of the algorithm that searches for any violation of constraints is also
considered, and the performance of the two algorithm is compared. An application to solving minimax
problem (with and without coupled constraints) is given and a comparison with the algorithm for contin-
uous minimax of Rustem and Howe (2001) is included. Finally, we consider an application to chemical
engineering problems.

Keywords: minimax problems, multiple solutions

1. Introduction

In this paper the following semi-infinite problem is considered:

min f (x)
x∈X
s.t. G(x, y)
0 ∀y ∈ Y,


X = x ∈ Rn | gi (x)
0, i = 1, . . . , k , (1)


Y = y ∈ Rm | qi (y)
0, i = 1, . . . , l ,
G : Rn+m → RnG ,

where X and Y are nonempty compact subsets of Rn and Rm and f and G are continu-
ously differentiable on X and X × Y , respectively. The term semi-infinite programming
arises from the fact that G(x, y)
0 ∀y ∈ Y represent an infinite set of constraints on x.
This problem was first studied by John (1948) who gave necessary and sufficient
conditions for its solution. Since then, it has been extensively studied in the literature.
Other efforts in this direction include (Blankenship and Falk, 1976; Coope and Watson,
∗ Corresponding author.
82 ŽAKOVIĆ AND RUSTEM

1985; Gustafson, 1981; Polak and Mayne, 1976). The reader is referred to (Hettich and
Kortanek, 1993) and the references therein.
Certain minimax and saddle point problems, as well as problems of optimal en-
gineering design can be formulated as in (1). An application to computer-aided design
is presented in Mayne, Polak, and Trahan (1979), whereas in Kwak and Haug (1976)
parametric optimal design is considered. In such problems, f (x) in (1) may present the
cost function associated with the design, where x is a vector of control variables and y is
a vector of uncertain parameters. The formulation ensures that the resulting design will
fulfil the constraints G(x, y)
0 for all y ∈ Y .
Also, some semi-infinite programming can be transformed into semidefinite pro-
gramming problems. A way to do it is to transform the constraints as linear matrix
inequalities (LMI), as suggested in Boyd et al. (1994). Then, the problem can be solved
using the recently developed interior point methods for LMI problems. Although many
nonlinear convex constraints can be expressed as LMIs, our aim is to produce an algo-
rithm for the general class of nonlinear problems.
In this paper, we solve (1) using the algorithm based on the work of Blankenship
and Falk (1976) and Darlington et al. (1999), where a two-phase procedure to guarantee
feasibility for the general nonlinear constraints was employed.
We reformulate a min-max robust design problem in terms of (1) and demonstrate
the desirable robustness property of the corresponding solution. We compare these with
ordinary min-max solution and saddle point solution to establish the superiority of (1).
We also show that the computation of global optima is central to the robustness prop-
erty. A version of the algorithm that searches for any, instead of the global, violation of
constraints is also presented.
The algorithm begins at k = 0 with a finite subset Yk ⊂ Y of the set of all con-
straints in (1), solving the approximate problem:


Fk = x ∈ X | G(x, y)
0 ∀y ∈ Yk ,


xk = arg min f (x) | x ∈ Fk , (2)
x
vk = f (xk ).
We denote F = {x ∈ X | G(x, y)
0 ∀y ∈ Y } and v ∗ = minx∈F f (x).
The algorithm solves the kth auxiliary global optimisation problem – compute
ymax ∈ Y such that
G(xk , ymax )  G(xk , y) ∀y ∈ Y. (3)
Termination is declared when the inequality
G(xk , ymax )
0 (4)
is satisfied. Otherwise, the algorithm sets
Yk+1 = Yk ∪ {ymax }, k = k + 1, (5)
and solves (2) again.
RELAXATION PROCEDURE 83

We note that
F ⊂ · · · ⊂ Fk+1 ⊂ Fk ⊂ · · · ⊂ F0 , (6)
which implies
v ∗  · · ·  vk+1  vk  · · ·  v0 . (7)
To justify the stopping rule, we observe that if the solution of kth auxiliary problem
satisfies
G(xk , ymax )
0 ⇒ G(xk , y)
0 ∀y ∈ Y. (8)
Therefore
vk
v ∗
f (xk ) = vk , (9)
so that xk is a solution of (2), with Fk = F .

1.1. Global optimisation

For global optimisation in (3) we use a stochastic algorithm, developed by Rinnooy

Kan and Timmer (1987). The algorithm starts local optimisations from numerous dif-
ferent points, and keeps a list (Xmax ) of maxima encountered. As a stopping rule it
uses Bayesian estimate of the total number of maxima, based on the numbers of lo-
cal optimisations performed, and local solutions found. In step 4 we define xmax (xst ) =
arg maxx {f (x)} as a local maximum of f (x), obtained when search was started from xst .
The algorithm is outlined below.

Algorithm 1 Global Optimisation

1: Xmax = ∅, nloc = 0, nmax = 0
2: repeat
3: Choose a random starting point xst
4: xmax (xst ) = arg maxx {f (x)}
5: nloc = nloc + 1
6: if not (xmax (xst ) ∈ Xmax )) then
7: Xmax = Xmax ∪ xmax (xst )
8: nmax = nmax + 1
9: end if
10: ntot = nmax (nloc − 1)/(nloc − nmax − 2)
11: until {(ntot
nmax + 0.5).or.(f (xmax (xst ))  0)}
12: xglob = {xglob ∈ Xmax | f (xglob )  f (x), ∀x ∈ Xmax }

Step 11, where we check whether f (xmax (xst ))  0, is not a part of the original
Rinnooy Kan and Timmer (1987) algorithm. The price of computing the global opti-
mum can be, and usually is, very high. As we are searching for points which violate
constraints it might be cheaper to stop the global optimisation search at a point where
violation is encountered. This approach can bring benefits regarding the computational
84 ŽAKOVIĆ AND RUSTEM

time, especially when functions that are being globally optimised are very complex.
There may be setbacks, however, that by including local violations of constraints, we
solve the nonlinear programming program in (2) more times than necessary. A switch
is built in our algorithm, which allows users to decide, depending on the complexity of
functions f (x) and G(x, y), which version of the algorithm to use.
The constrained nonlinear programming problems which appear in (2), and also in
the above algorithm, are solved using standard nonlinear programming packages, in this
case NAG subroutine E04UCF (see (NAG Library) reference). The subroutine computes
the minimum of a general smooth function which may include simple bounds as well as
linear and nonlinear constraints.

1.2. Semi-infinite programming algorithm

The full semi-infinite algorithm (SIP), for solving (1) is given below. In computational
examples of later sections we compare the performance of the SIP algorithm which in-
cludes complete global optimisation (A1) with the algorithm that searches for violation
of constraints only (A2) and with the algorithm of Rustem and Howe (2001) (marked
with B).

Algorithm 2 Semi-Infinite Programming

1: k = 0, Y0 = {y0 }, Fk = {x ∈ X | G(x, y)
0 ∀y ∈ Yk }
2: phase 1
3: xk = arg minx {f (x) | x ∈ Fk }
4: if (infeasible) then
5: STOP: original problem infeasible
6: end if
7: phase 2
8: OK = true, Y = ∅
9: for j = 1, nG do
j
10: yk = arg maxy∈Y {Gj (xk , y)} (global)
j
11: if {Gj (xk , yk )  0} then
12: OK = false
13: = Y
Y  ∪ {ykj }
14: end if
15: end for
16: if (not OK) then
17: Yk+1 = Yk ∪ Y 
18: k =k+1
19: goto 2
20: else
21: optimal solution found
22: end if
RELAXATION PROCEDURE 85

The feasibility check in step 11 depends on the nature of the constraints, whether
they are hard or soft, and on the application being considered. Hard constraints indicate
situations where no constraint violation can be tolerated by the physical system. This
requires feasibility for all realizations of y, which may further increase the price of com-
putation. If strict feasibility cannot be obtained or guaranteed a penalty approach may be
used, allowing optimal solutions with the smallest possible violation of the constraints
G(x, y) within Y . This approach is further discussed in (Darlington et al., 1999).
In later sections we also consider a choice for y0 in step 1 of the above algorithm.
In cases when G(x, y) is convex with respect to y the maximum is obtained at the
boundaries of the feasible region and it seems that choosing x0 first and then obtaining
y0 = arg maxy G(x0 , y) may further decrease the computational time, as y0 computed in
such way may carry more information about the problem, then a randomly chosen one.
The paper is organised as follows. In section 2, an application to minimax prob-
lems is described. Computational results are reported in section 3. A comparison with
computation of minimax saddle points is presented in section 4 with some applications to
chemical engineering problems covered in section 5. The conclusions and final remarks
are presented in section 6.

2. Application to minimax problems

Semi-infinite programming can be easily applied to certain minimax problems. General

minimax problems can be expressed as:
min max f (x, y)
x∈X y∈Y
s.t. G(x, y)
0, G : Rn+m → RnG ,

 (10)
X = x | gi (x)
0, x ∈ Rn , i = 1, . . . , k ,


Y = y | qi (y)
0, y ∈ Rm , i = 1, . . . , l .

Such problems arise in numerous disciplines, including engineering (Mayne, Po-

lak, and Trahan, 1979), finance (Rustem and Howe, 2001), missile defence (Danskin,
1967; Monahan, 1996), n-person games (Rosen, 1965) and, in general, they are used by
the decision maker to assess the worst-case strategy of uncertainty or an opponent and
compute the optimal response to the worst case. This is very useful in optimal process
design, or decision making under uncertainty (Darlington et al., 1999; Halemane and
Grossmann, 1983; Kwak and Haug, 1976).
In the case of general functions f (x, y) and G(x, y), there does not appear to
be an algorithm for solving (10). There are methods which exploit convexity (with
respect to x) and concavity (with respect to y) of the objective function – Demyanov
in (Demyanov and Pevnyi, 1971) or an interior point approach (Zakovic, Pantelides,
and Rustem, 2000). In the absence of coupled constraints G(x, y) there is a penalty
approach by Sasai (1974) or a relaxation procedure by Shimizu and Aiyoshi (1980).
More recently, Panin (1981) presented a linearisation algorithm, and a quasi-Newton
86 ŽAKOVIĆ AND RUSTEM

second order algorithm (where X = Rn ) has been suggested by Rustem and Howe
(2001).
A typical minimax problem of the type discussed by Danskin (1967) has the form:
min max f (x, y) = v ∗ . (11)
x∈X y∈Y

If we set l < v ∗ , L > v ∗ then this problem can be rewritten as

min xn+1
x∈X,xn+1 ∈[l,L]
(12)
s.t. max f (x, y)
xn+1 .
y∈Y

To rewrite the above problem in the form of (1) we note that:

max f (x, y)
xn+1 ⇔ f (x, y)
xn+1 ∀y ∈ Y. (13)
y∈Y

So, problem (11) can be rewritten in the form of (1) as

min xn+1
x∈X,xn+1 ∈[l,L] (14)
s.t. f (x, y)
xn+1 ∀y ∈ Y.
Problem (10) is well defined and may, at first inspection, be associated with op-
timum design in the presence of uncertainty. However, the close relation of minimax
theory with worst case design does not necessarily mean that the two problems are equiv-
alent. For example, assume that f (x, y) in (10) presents a cost function of a chemical
plant, where x is a vector of control variables and y is a vector of uncertain parame-
ters. With formulation (10) the plant is actually optimised at the value of parameters
that minimises the cost function. Although the solution of the minimax problem (10)
provides a lower bound for the cost, it does not ensure that the resulting design will fulfil
the constraints G(x, y)
0, for arbitrary values y ∈ Y . To ensure strict feasibility we
perform the following transformation:
min xn+1
x∈X,xn+1 ∈[l,L]
s.t. max f (x, y) − xn+1
0, (15)
y∈Y
max G(x, y)
0.
y∈Y

As explained before, maxy∈Y f (x, y)
xn+1 ⇔ f (x, y)
xn+1 ∀y ∈ Y , and similarly

maxy∈Y G(x, y)
0 ⇔ G(x, y)
0 ∀y ∈ Y problem (15) is of the same form as the
semi-infinite programming problem (1).
We note that continuous minimax addresses issues of uniform distribution where
the probability associated with the uncertainty is the same for all values. An alterna-
tive would be to specify a discrete set of scenarios with associated probabilities. This
may address certain cases where the scenarios are no longer uniformly distributed. The
basic methodology is, however, a stochastic programming approach (e.g., see Kall and
Wallace (1994) and Prekopa (1995)).
RELAXATION PROCEDURE 87

3. Computational examples

In this section we report the experience with numerical examples. The algorithms de-
fined in the previous sections have been implemented using Fortran 90 on a Sun Ultra-
Sparc 60, using the 450 MHz UltraSparc II processor. The following sixteen examples
are taken from (Rustem and Howe, 2001). First seven are convex–concave functions
(i.e., f (x, y) convex in x and concave in y) and then the further nine examples are
convex–convex functions, where multiple maximisers are present, and we examine the
behaviour of the algorithm in presence of such points.
The results are obtained using the algorithm presented in this paper (marked with
A1 and A2) and the algorithm of Rustem and Howe (2001) (marked B).

3.1. Convex–concave case

Problem 1.
f (x, y) = 5x12 + 5x22 − y12 − y22 + x1 (−y1 + y2 + 5) + x2 (y1 − y2 + 3),
−5.0
y
5.0;
x0 = [10.0, −10.0], y0 = [5.0, −5.0];
A1, A2 x∗ = [−0.483331, −0.316669],
y∗ = [0.083346, −0.083346], (16)
(x∗ ) = −1.683333.
B x∗ = [−0.483357, −0.316643],
y∗ = [0.083334, −0.083334],
(x∗ ) = −1.683333.

Problem 2.
f (x, y) = 4(x1 − 2)2 − 2y12 + x12 y1 − y22 + 2x22 y2 ,
−5.0
yi
5.0, i = 1, 2;
x0 = [4.0,
 4.0], y0 = [5.0, 5.0],
A1, A2 x∗ = 1.695431, 3.573034 · 10−5 ,
 
y∗ = 0.718564, 1.264831 · 10−9 , (17)
(x∗ ) = 1.403883.
B x∗ = [1.695415,
 −0.011596], 
y∗ = 0.718608, 1.205366 · 10−4 ,
(x∗ ) = 1.403883.

Problem 3.
f (x, y) = x14 y2 + 2x13 y1 − x22 y2 (y2 − 3) − 2x2 (y1 − 3)2 ,
0.0
yi
3.0, i = 1, 2;
x0 = [−1.0, 1.0], y0 = [1.5, 1.5].
(18)
A1, A2 x∗ = [−1.180679, 0.912804],
y∗ = [2.098417, 2.666203],
(x∗ ) = −2.468775.
88 ŽAKOVIĆ AND RUSTEM

B x∗ = [−1.180671, 0.912835],
y∗ = [2.098489, 2.666036],
(x∗ ) = −2.468775.

Problem 4.

f (x, y) = (x1 − 1)2 + (x2 − 1)2 − (y1 − 1)2 − (y2 − 1)2 − (y3 − 1)2
+ y1 (x1 − 1) + y2 x1 x2 + y3 (x2 − 1),
−3.0
yi
3.0, i = 1, 3;
x0 = [−2.0, −1.0], y0 = [0.0, 0.0, 0.0].
A1, A2 x∗ = [0.418114, 0.418114],
(19)
y∗ = [0.709096, 1.087443, 0.709096],
(x∗ ) = −0.134834.
B x∗ = [0.418130, 0.418128],
y∗ = [0.709063, 1.087415, 0.709064],
(x∗ ) = −0.134834.

Problem 5.

f (x, y) = − y1 (x1 − 1) − y2 (x2 − 2) − y3 (x3 − 1)

− y12 − y22 − y32 + 2x12 + 3x22 + x32 ,
−1.0
yi
1.0, i = 1, 3;
x0 = [2.0, 2.0, 2.0], y0 = [1.0, 1.0, 1.0].
A1, A2 x∗ = [0.111112, 0.153846, 0.199981],
(20)
y∗ = [0.444444, 0.923077, 0.399964],
(x∗ ) = 1.345299.
B x∗ = [0.111111, 0.153839, 0.200005],
y∗ = [0.444444, 0.923079, 0.399999],
(x∗ ) = 1.345299.

Problem 6.
   
f (x, y) = y1 x12− x2 + x3 − x4 + 2 + y2 −x1 + 2x22 − x32 + 2x4 + 1
+ y3 2x1 − x2 + 2x3 − x42 + 5 − y12 − y22 − y32
+ 5x12 + 4x22 + 3x32 + 2x42 ,
−2.0
yi
2.0, i = 1, 3;
x0 = [−2.0, 2.0, −2.0, 2.0], y0 = [2.0, 2.0, 2.0].
A1, A2 x∗ = [−0.231560, 0.222809, −0.675524, −0.083804], (21)
y∗ = [0.619584, 0.353366, 1.477990],
(x∗ ) = 4.542970.
B x∗ = [−0.231563, 0.222815, −0.675510, −0.083760],
y∗ = [0.619530, 0.353504, 1.478010],
(x∗ ) = 4.542967.
RELAXATION PROCEDURE 89

Problem 7.
     
f (x, y) = y1 x12 − 1 + y2 x22 − 1 + y3 x32 − 1 + 5x42 + 5x52
− y12 − y22 − y32 − y42 − y52 + 2x1 x5 + 3x2 x4 + x3 x5
− x4 (y4 − y5 − 5) + x5 (y4 − y5 + 3),
−3.0
yi
3.0, i = 1, 5;
x0 = [−4.0, −4.0, −4.0, −4.0, −4.0],
y0 = [2.0, 2.0, 2.0, 2.0, 2.0]. (22)
A1, A2 x∗ = [1.425206, 1.661242, 1.258543, −0.974417, −0.734852],
y∗ = [0.515581, 0.879821, 0.291921, 0.119782, −0.119782],
(x∗ ) = −6.350915.
B x∗ = [1.425203, 1.661232, 1.258514, −0.974420, −0.734844],
y∗ = [0.515604, 0.879844, 0.291935, 0.119786, −0.119786],
(x∗ ) = −6.350915.

Results obtained using algorithms A1 and A2 are identical, as function f (x, y)

is concave with respect to y, so there is the unique maximiser for any x, therefore
f (x, ymax (yst )) = f (x, yglob ). CPU times used for the two methods do differ, as can
be seen from table 1, in which we present the number of iterations and CPU time used
for each of the problems.
We can see from the statistics in table 1 that the algorithm B performs much better
in terms of CPU time and in general needs less iterations to converge, compared to
algorithms A. It is worth pointing out that these examples were just used for preliminary
testing. Later we deal with problems which cannot be solved by algorithm B.

3.2. Convex–convex case

The next nine examples are convex–convex functions (i.e., f (x, y) convex in x and in y),
so the maximum values for f (x, y) are obtained at the vertices of the feasible region Y .
Also, the presence of multiple maxima is expected. We present nine examples and each
example is run twice – with algorithms A1 and A2. Results obtained using algorithm B
are also included.

Problem 8.
 2
f (x, y) = 32 12 x + (4 + x)y ,
−2.0
y
2.0; x0 = 3.0, y0 = 2.0;
A1 x∗ = −4.000000, y∗ = [−2.0]; [2.0],
(x∗ ) = 6.000000. (23)
A2 x∗ = −4.000000, y∗ = [−2.0]; [2.0],
(x∗ ) = 6.000000.
B x∗ = −4.000000, y∗ = −2.0,
(x∗ ) = 6.000000.
90 ŽAKOVIĆ AND RUSTEM

Table 1
Numerical results (1–7).
Problem no. A1 A2 B
1
CPU 0.5386 0.3702 0.1338
Number of iterations 9 9 7
2
CPU 0.9144 0.7417 0.1963
Number of iterations 10 10 19
3
CPU 1.8519 1.2919 0.0854
Number of iterations 16 16 10
4
CPU 0.8812 0.6978 0.0997
Number of iterations 8 8 6
5
CPU 1.2352 1.0895 0.0864
Number of iterations 8 8 6
6
CPU 2.1328 1.7837 0.1795
Number of iterations 16 15 15
7
CPU 10.5240 8.9569 0.4356
Number of iterations 46 46 16

Problem 9.
 
f (x, y) = 12 (2x2 + 4y1 + x1 y1 )2 + (x1 + 2x1 y1 + x2 y2 )2 ,
−5.0
yi
5.0, i = 1, 2;
x0 = [0.0,
 0.0], y0 = [5.0, 5.0]; 
A1 x∗ = −0.684932, 5.653575 · 10−17 ,
y∗ = [5.0, 5.0]; [5.0, −5.0],
(x∗ ) = 165.753425.  (24)
A2 x∗ = −0.684931, 1.873597 · 10−17 ,
y∗ = [5.0, 5.0]; [5.0, −5.0],
(x∗ ) = 165.753425. 
B x∗ = −0.684932, 9.983768 · 10−10 ,
y∗ = [5.0, 5.0],
(x∗ ) = 165.753425.

Problem 10.
 
f (x, y) = 12 (5x2 + 5y1 + 3x1 y1 )2 + (2x1 + 5x1 y1 + 3x2 y2 )2 ,
−5.0
yi
5.0, i = 1, 2;
x0 = [1.0, 1.0], y0 = [5.0, 5.0].
RELAXATION PROCEDURE 91

 
A1 x∗ = −0.393082, 4.298982 · 10−18 ,
y∗ = [5.0, 5.0]; [5.0, −5.0],
(x∗ ) = 238.797170. 
A2 x∗ = −0.393082, 5.386406 · 10−17 ,
y∗ = [5.0, 5.0]; [5.0, −5.0], (25)
(x∗ ) = 238.797170. 
B x∗ = −0.393082, −3.068550 · 10−11 ,
y∗ = [5.0, 5.0]; [5.0, −5.0],
(x∗ ) = 238.797170.

Problem 11.
 2  2 
f (x, y) = 12 x1 x2 − y1 (1 − x3 x4 ) + x2 x3 − y2 (2 + x4 x1 ) + y12 + y22 ,
−5.0
yi
5.0, i = 1, 2;
x0 = [1.0,
 1.0, 1.0, 1.0], y0 = [3.0, 3.0]. 
A1 x∗ = −2.078678, 1.283354 · 10−5 , 1.039332, 0.962151 ,
y∗ = [5.0, 5.0]; [5.0, −5.0]; [−5.0, 5.0]; [−5.0, −5.0],
(x∗ ) = 25.00.  (26)
A2 x∗ = −2.546941, 2.448681 · 10−5 , 1.273491, 0.785253 ,
y∗ = [5.0, 5.0]; [5.0, −5.0]; [−5.0, 5.0]; [−5.0, −5.0],
(x∗ ) = 25.00. 
B x∗ = −1.541658, 5.346038 · 10−5 , 0.770815, 1.297296 ,
y∗ = [5.0, 5.0]; [5.0, −5.0]; [−5.0, 5.0]; [−5.0, −5.0],
(x∗ ) = 25.00.

Problem 12.
 
f (x, y) = y12 + y1 x12 − x2 + x3 − x4 + 2
 
+ y22 + y2 −x1 + 2x22 − x32 + 2x4 − 10
 
+ y32 + y3 2x1 − x2 + 2x3 − x42 − 5
 2 
+ 5 x1 + x22 + x32 + x42 ,
−2.0
yi
2.0, i = 1, 3;
x0 = [10.0,
 1.0, 1.0, 10.0], y0 = [2.0, 2.0, 2.0]. 
A1 x∗ = 0.142856, −1.453767 · 10−6 , 0.142857, 0.428569 ,
y∗ = [2.0, −2.0, −2.0], (27)
(x∗ ) = 44.428571. 
A2 x∗ = 0.142856, −1.453767 · 10−6 , 0.142857, 0.428569 ,
y∗ = [2.0, −2.0, −2.0],
(x∗ ) = 44.428571. 
B x∗ = 0.142857, 1.232060 · 10−6 , 0.142857, 0.428571 ,
y∗ = [2.0, −2.0, −2.0],
(x∗ ) = 44.428571.
92 ŽAKOVIĆ AND RUSTEM

Problem 13.
 
f (x, y) = y12 + y1 x12 − x2 + x3 − x4 + 2
 
+ y22 + y2 −x1 + 2x22 − x32 + 2x4 − 10
 
+ y32 + y3 2x1 − x2 + 2x3 − x42 − 5
 
+ y42 + 5y4 x12 + x22 + 5x32 + 5x42 ,
−2.0
yi
2.0, i = 1, 4;
x0 = [1.0,
 −1.0, −1.0, 1.0], y0 = [2.0, 2.0, 2.0, 2.0]. 
A1 x∗ = 8.333370 · 10−2 , 6.558254 · 10−7 , 0.142856, 0.428571 ,
y∗ = [2.0, −2.0, −2.0, 2.0], (28)
(x∗ ) = 48.488095. 
A2 x∗ = 8.333322 · 10−2 , −6.398135 · 10−8 , 0.142857, 0.428571 ,
y∗ = [2.0, −2.0, −2.0, 2.0],
(x∗ ) = 48.488095. 
B x∗ = 8.333322 · 10−2 , −6.398134 · 10−8 , 0.142857, 0.428571 ,
y∗ = [2.0, −2.0, −2.0, 2.0],
(x∗ ) = 48.488095.

Problem 14.
 
f (x, y) = y12 + y1 x12 − 2.2x2 + x3 − 10x4 + 10
 
+ y22 + y2 −2x1 + 2x22 − x32 + 3x4 − 10
 
+ y32 + y3 2x1 − x2 + 6x3 − x42 − 5
 
+ y42 + 5y4 x12 + x22 + 5x32 + 5x42 ,
−2.0
yi
2.0, i = 1, 4;
x0 = [0.0,
 0.0, 0.0, 0.0], y0 = [2.0, 2.0, 2.0, 2.0]. 
A1 x∗ = −1.193319 · 10−6 , 3.123732 · 10−4 , 0.792086, 1.079140 ,
y∗ = [2.0, −2.0, −2.0, 2.0]; [2.0, −2.0, −2.0, −2.0];
y∗ = [−2.0, −2.0, −2.0, 2.0]; [−2.0, −2.0, −2.0, −2.0], (29)
(x∗ ) = 42.564356. 
A2 x∗ = −7.086795 · 10−8 , 3.125216 · 10−4 , 0.7920858, 1.079140 ,
y∗ = [2.0, −2.0, −2.0, 2.0]; [2.0, −2.0, −2.0, −2.0];
y∗ = [−2.0, −2.0, −2.0, 2.0]; [−2.0, −2.0, −2.0, −2.0],
(x∗ ) = 42.564356. 
B x∗ = −9.582755 · 10−7 , 3.079998 · 10−4 , 0.792086, 1.079140 ,
y∗ = [2.0, −2.0, −2.0, 2.0],
(x∗ ) = 42.564356.
Problem 15.  
f (x, y) = y12 + y1 x12 − 2.2x2 + x3 − 10x4 + 10
 
+ y22 + y2 −2x1 + 2x22 − x32 + 3x4 − 10
 
+ y32 + y3 2x1 − x2 + 5.91x3 − x42 − 15
 
+ y42 + 5y4 x12 + x22 + 5x32 + 5x42 ,
−2.0
yi
2.0, i = 1, 4;
RELAXATION PROCEDURE 93

x0 = [1.0,
 1.0, 1.0, 1.0], y0 = [2.0, 2.0, 2.0, 2.0]. 
A1 x∗ = −1.422337 · 10−7 , 3.171196 · 10−6 , 0.779350, 1.077934 ,
y∗ = [2.0, −2.0, −2.0, 2.0]; [2.0, −2.0, −2.0, −2.0];
y∗ = [−2.0, −2.0, −2.0, 2.0]; [−2.0, −2.0, −2.0, −2.0],
(x∗ ) = 62.705785. 
A2 x∗ = −1.281341 · 10−7 , 3.165304 · 10−6 , 0.779350, 1.077934 ,
y∗ = [2.0, −2.0, −2.0, 2.0]; [2.0, −2.0, −2.0, −2.0];
y∗ = [−2.0, −2.0, −2.0, 2.0]; [−2.0, −2.0, −2.0, −2.0], (30)
(x∗ ) = 62.705785. 
B x∗ = 4.343094 · 10−6 , −1.037439 · 10−5 , 0.779351, 1.077912 ,
y∗ = [2.0, −2.0, −2.0, −2.0]; [−2.0, −2.0, −2.0, 2.0];
y∗ = [2.0, −2.0, −2.0, 2.0],
(x∗ ) = 62.705786.

Problem 16.
 
f (x, y) = 12 y12 + y22 + y32 + y42
 
+ y1 x12 − 2x2 + x3 − 10x4 + 2
 
+ y2 −2x1 + 2x22 − x32 + 3x4 − 5
 
+ y3 2x1 − x2 + 5x3 − x42 + 2
 
+ y4 x12 + x22 + x32 + x42 ,
−2.0
yi
2.0, i = 1, 4;
x0 = [0.1, 0.0, 1.0, 0.0], y0 = [2.0, 2.0, 2.0, 2.0].
A1 x∗ = [−0.746279, −1.063201, −0.275267, 0.440807],
y∗ = [2.0, 2.0, 2.0, 2.0]; [2.0, 2.0, −2.0, 2.0];
y∗ = [2.0, −2.0, 2.0, 2.0]; [2.0, −2.0, −2.0, 2.0];
y∗ = [−2.0, 2.0, 2.0, 2.0]; [−2.0, 2.0, −2.0, 2.0];
y∗ = [−2.0, −2.0, 2.0, 2.0]; [−2.0, −2.0, −2.0, 2.0], (31)
(x∗ ) = 11.914822. 
A2 x∗ = −0.476663, −1.043031, 3.791007 · 10−2 , 0.624094 ,
y∗ = [2.0, 2.0, 2.0, 2.0]; [2.0, 2.0, −2.0, 2.0];
y∗ = [2.0, −2.0, 2.0, 2.0]; [2.0, −2.0, −2.0, 2.0];
y∗ = [−2.0, 2.0, 2.0, 2.0]; [−2.0, 2.0, −2.0, 2.0];
y∗ = [−2.0, −2.0, 2.0, 2.0]; [−2.0, −2.0, −2.0, 2.0],
(x∗ ) = 11.412105.
B x∗ = [−0.746283, −1.063199, −0.275264, 0.440807],
y∗ = [−2.0, −2.0, −2.0, 2.0]; [2.0, −2.0, −2.0, 2.0];
[−2.0, 2.0, −2.0, 2.0]; [2.0, 2.0, −2.0, 2.0],
(x∗ ) = 11.914823.

Problems 8–10 have an infinite number of solutions. Problem 8, for example,

obtains its minimax value for x = −4, and f (x, y) does not depend on y anymore, so
any pair (x∗ = −4, y∗ = y) is the minimax value for f (x, y). Similarly, y∗ = [5, y2 ]
94 ŽAKOVIĆ AND RUSTEM

is the minimax solution for problems 9 and 10. This might cause problems both in the
global optimisation algorithm and computing the right descent direction for the function
f (x, y). Regarding the global optimisation algorithm, an IF statement can be added,
which would allow us to terminate the algorithm when a big number of local optima
occurs. In this implementation we stop the global optimisation algorithm when more
than 25 local solutions are encountered, and check for the solutions at the corners of
feasible region.
Invoking Caratheodory’s theorem (Rockafeller, 1972), the subgradient gk at xk can
be characterised by a convex combination of at most (m + 1) vectors ∇x f (xk , y) corre-
sponding to the maximisers y ∈ Y at xk . Thus, we have
 y y
 y
gk = λk ∇x f (xk , y); λk  0; λk = 1, (32)
y∈Y (xk ) y∈Y (xk )

where Y(xk ) ≡ {y | y = arg maxy f (xk , y)} and m = dim Y . Therefore, in the case of a
big number of maximisers we can choose m + 1 of them and compute the above linear
combination. More about this is said in the next section.
The CPU time and number of iterations for each of the problems 8–16 are presented
in table 2.
From the results shown in table 2 we see that, in general, the full algorithm A1
converges in smaller number of iterations. However, the algorithm A2 converges in less
time, as it does not search for global optimum, not until the last iteration. It is also worth
noting that algorithm B is much slower now, both in terms of CPU time and number of
iterations.

3.3. Estimating y0

When solving the approximate problem (2) we are required to give an estimate for y and
then solve the nonlinear programming problem vk = minx∈Fk f (x).
In cases when f (x, y) is convex with respect to y it may be better to do the global
optimisation first and compute y, for a given estimate of x, as we are likely to get an
initial estimate for y which will be included in the set of active constraints at the solution.
Problem 12, for example, has only one maximiser with respect to y, so finding x∗ is
equivalent to the following problem:

min x5
(33)
s.t. f (x, y∗ ) − x5
0,

where f (x, y) is same as in problem 12 and y∗ = [2, −2, −2]. The above is a simple
nonlinear programming problem with one constraint only. When previously solving
problem 12 we had an estimate y0 = [2, 2, 2]. Then at second iteration y∗ was found
and the solution was obtained at two iterations. Solving maxy f (x0 , y) we hope to obtain
y∗ which would allow us to solve problem 12 in only one iteration.
RELAXATION PROCEDURE 95

Table 2
Numerical results (8–16).
Problem no. A1 A2 B
8
CPU 0.0828 0.0524 1.8542
Number of iterations 2 2 15
9
CPU 0.1265 0.0933 2.9834
Number of iterations 2 3 23
10
CPU 0.1242 0.0903 2.7780
Number of iterations 2 3 29
11
CPU 0.2196 0.1051 1.2829
Number of iterations 3 3 22
12
CPU 0.1712 0.1284 0.7513
Number of iterations 2 3 14
13
CPU 0.2486 0.1819 0.4480
Number of iterations 2 3 6
14
CPU 0.6235 0.4114 7.5100
Number of iterations 3 3 36
15
CPU 0.3670 0.1652 1.9548
Number of iterations 3 3 26
16
CPU 1.5756 1.0143 17.2147
Number of iterations 4 4 51

Table 3 shows a comparison with the results obtained in this way (marked with *)
and the results obtained when we choose starting points y0 which are in the centre of the
feasible region.
In general, results in table 3 are as expected, as the algorithms marked with (*)
converge faster than the algorithms described earlier.

3.4. Global vs. local optimisation

In previous sections global optimisation has been used to find the maximum violation
of constraints. Even when the algorithm A2 is used, global search is still performed, at
last iteration, thus ensuring that all the constraints are satisfied. To demonstrate a failure
96 ŽAKOVIĆ AND RUSTEM

Table 3
Numerical results (8–16) (y0 is computed first).
Problem no. A1 A2 A1* A2*
8
CPU 0.1262 0.0509 0.1312 0.0599
Number of iterations 2 2 2 2
9
CPU 0.2007 0.1111 0.2072 0.1036
Number of iterations 3 5 2 4
10
CPU 0.3016 0.1169 0.1838 0.0969
Number of iterations 4 5 2 4
11
CPU 0.2310 0.1182 0.2180 0.1469
Number of iterations 3 3 2 2
12
CPU 0.1893 0.1216 0.1684 0.1225
Number of iterations 2 4 1 3
13
CPU 0.2772 0.2009 0.2152 0.1428
Number of iterations 2 2 1 1
14
CPU 0.7438 0.4039 0.9383 0.3805
Number of iterations 4 4 4 6
15
CPU 0.3728 0.3022 0.3129 0.1445
Number of iterations 3 3 2 2
16
CPU 1.6045 0.4846 0.9130 0.9076
Number of iterations 5 5 4 4

of the algorithm, when only local optimisation is used we consider problem 16 from the
previous section:
 
min max f (x, y) = 12 y12 + y22 + y32 + y42
x y  
+ y1 x12 − 2x2 + x3 − 10x4 + 2
 
+ y2 −2x1 + 2x22 − x32 + 3x4 − 5
 
+ y3 2x1 − x2 + 5x3 − x42 + 2 (34)
 2 
+ y4 x1 + x22 + x32 + x42 ,
−2.0
yi
2.0, i = 1, 4,
x0 = [0.1, 0.0, 1.0, 0.0],
y0 = [2.0, 2.0, 2.0, 2.0].
RELAXATION PROCEDURE 97

The solutions we obtain for this problem are:

Global optimization:
x∗ = [−0.746279, −1.063201, −0.275267, 0.440807],
(x∗ ) = 11.914822.
(35)
Localoptimization: 
x∗ = −0.568896, −0.854798, −6.872275 · 10−2 , 0.396452 ,
(x∗ ) = 10.432437.
Locally, 4 iterations are needed, whereas the global version converges in 6 itera-
tions. The local optimisation problem can be presented as follows:
min x5
x
s.t. f (x, yi ) − x5
0, i = 1, . . . , 4, (36)
y1 = [2, 2, 2, 2], y2 = [−2, −2, −2, 2],
y3 = [2, −2, 2, −2], y4 = [−2, −2, 2, −2].
The solution to this problem, as reported above is (x∗ ) = 10.432437. When the algo-
rithm checks for violation of constraints locally it obtains the point y5l = [2, −2, −2, 2]
which does not violate constraints, and therefore it terminates.
At that stage, global optimisation obtains the point y5g = [−2, −2, 2, 2] which
does violate the constraints and continues with iterations. At the end the global solution
of problem 16 is equivalent to the following problem:
min x5
x
s.t. f (x, yi ) − x5
0, i = 1, . . . , 6,
y1 = [2, 2, 2, 2], y2 = [−2, −2, −2, 2], (37)
y3 = [2, −2, 2, −2], y4 = [−2, −2, 2, −2],
y5 = [−2, −2, 2, 2], y6 = [2, 2, −2, 2].

4. Comparison with computation of minimax saddle points

In this section, we consider the general constrained minimax problem:

min max f (x, y)
x∈X y∈Y
s.t. G(x,
y)
0, G : Rn+m → RnG ,  (38)
X = x | gi (x)
0, x ∈ Rn , i = 1, . . . , k ,


Y = y | qi (y)
0, y ∈ Rm , i = 1, . . . , l ,
where it is assumed that the objective function f (x, y) is convex in x and concave in y.
The saddle point problem is to find a pair (x∗ , y∗ ) ∈ X × Y so that for every (x, y) ∈
X × Y the following inequality is satisfied:
f (x∗ , y)
f (x∗ , y∗ )
f (x, y∗ ). (39)
98 ŽAKOVIĆ AND RUSTEM

In previous two sections the results obtained solving minimax problems directly
(using algorithm B), or transforming them into problem (1) were identical. As mentioned
earlier, in general, transforming (10) into (1) does not necessarily yield same results, as
we show in the following examples from (Zakovic, Pantelides, and Rustem, 2000) where
they are solved using the saddle point algorithm C.
Solve
min max f (x, y) = 5x12 + 5x22 − y12 − y22
x y
+ x1 (−y1 + y2 + 5) + x2 (y1 − y2 + 3)
(40)
s.t. x12 + y12
0.2, x22 + y22
0.1;
− 0.2
yi
0.2, i = 1, 2.
The solution obtained using the saddle point algorithm is:
xs∗ = [−0.44490515, −0.30929350],
 
ys∗ = 4.53804920 · 10−2 , −6.58572397 · 10−2 . (41)
fs∗ = −1.67570237.
To solve (40) using algorithm A1 presented in this paper we reformulate the prob-
lem as:
min x3
x
s.t. f (x, y) − x3
0,
x12 + y12
0.2, x22 + y22
0.1;
∀yi ∈ [−0.2, 0.2], i = 1, 2.
Choose y0 = [0.1, −0.1] as a starting point for y and we arrive at the following
solution:
xr∗ = [−0.40000000, −0.24494897],
fr∗ = −1.62282651.
This solution differs from the saddle point solution. We start from y0 = ys∗ and solve
the following problem:
min x3
x
s.t. f (x, ys∗ ) − x3
0,
G(x, ys∗ )
0.
Solving this problem we obtain x (1) = xs∗ .
Next step is to solve the maxy subproblem for the constraints:
 
max G x (1) , y , −0.2
yi
0.2. (42)
y

Because both constraints are convex in y, the maximum is obtained at the end points of
Y space (−0.2 or 0.2) for every x. Therefore, a new constraint is added and the saddle
RELAXATION PROCEDURE 99

point solution xs∗ is no longer feasible, as there exists y ∈ [−0.2, 0.2] : G(xs∗ , y) > 0.
At the end of this procedure, once again we arrive at the same point:

xr∗ = [−0.40000000, −0.24494897],

fr∗ = −1.62282651.

The saddle point solution ensures that the constraints are fulfilled for the saddle
point, but does not guarantee feasibility for all y ∈ [−0.2, 0.2]. Therefore more con-
straints need to be added, before a new solution xr∗ = [−0.40000000, −0.24494897] is
computed. Although this solution is worse than the saddle point, it ensures feasibility
of x∗ for all feasible y.
We present more examples, where we compare the saddle point solutions with
those obtained using the SIP formulation.

Problem 17.

f (x, y) = 5x12 + 5x22 − y12 − y22

+ x1 (−y1 + y2 + 5) + x2 (y1 − y2 + 3),
x12 + y12
0.2, x22 + y22
0.1;
−0.2
yi
0.2, i = 1, 2,
x0 = [0, 0, 0], y0 = [0.1, −0.1];
xs∗ = [−0.44490515,
 −0.30929350],  (43)
ys∗ = 4.53804920 · 10−2 , −6.58572397 · 10−2 ,
fs∗ = −1.67570237.
xr∗ = [−0.40000000, −0.24494897],
yr∗ = [0.20000000, −0.20000000].
fr∗ = −1.62282651.

Problem 18.

f (x, y) = (x1 − 1)2 + (x2 − 1)2

− (y1 − 1)2 − (y2 − 1)2 − (y3 − 1)2
+ y1 (x1 − 1) + y2 x1 x2 + y3 (x2 − 1),
x12 + y12
0.6, x22 + y22 + y32
1.5;
|y1 |
0.7, |y2 |
1.0, |y3 |
0.7;
x0 = [0, 0], y0 = [0.1, 0.1, 0.1];
(44)
xs∗ = [0.42791087, 0.42360873],
ys∗ = [0.64456423, 0.96194764, 0.62781732],
fs∗ = −0.16275239.
xr∗ = [0.33166248, 0.09999999],
yr∗ = [0.70000000, 1.00000000, 0.70000000],
fr∗ = 0.03567253.
100 ŽAKOVIĆ AND RUSTEM

Problem 19.
f (x, y) = 12 x12 + x22 − x1 x2 − 7(x1 + x2 )
− 12 y12 − y22 − y1 y2 − 7(y1 + y2 ))
+ 10−5 (x1 y1 − x2 y2 ),
4x12 + x22 − 25
0, 4y12 + y22 − 25
0;
x12 + y12
7, x22 + y22
16,
|y1 |
2.0, |y2 |
3.0;
x0 = [1.0, 1.0], y0 = [1.0, 1.0]. (45)
xs∗ = [1.87082673, 2.82843552],
ys∗ = [1.87083066, 2.82841873],
fs∗ = −6.77014363 · 10−5 .
xr∗ = [1.73205081, 2.64575131],
yr∗ = [2.00000000, 3.00000000],
fr∗ = 3.27276474.

Problem 20.
f (x, y) = (x1 − 2)2 + (x2 − 1)2 − (y1 − 2)2 − (y2 − 1)2
+ 10−5 (x1 y2 − x2 y1 ),
x1 + x2 − 2
0, x12 − x2
0,
y1 + y2 − 2
0, y12 − y2
0,
x12 + y12 − 1.5
0, x22 + y22 − 1.0
0,
|y1 |
0.9, |y2 |
0.75;
x0 = [0.0, 0.0], y0 = [0.10, 0.10]. (46)
xs∗ = [0.84089642, 0.70710678],
ys∗ = [0.84089642, 0.70710678],
fs∗ = −7.50373597 · 10−14 .
xr∗ = [0.81328828, 0.66143783],
yr∗ = [0.90000000, 0.75000000],
fr∗ = 0.25040920.

Problem 21.
f (x, y) = (x1 − 1)2 + (x2 − 3)2 − (y1 − 2)2 − (y2 − 4)2 ,
x1 − 0.5
0, x22 + y22
2;
x0 = [0.0, 0.9], y0 = [1.0, 0.9],
|y1 |
5, |y2 |
1.2;
xs∗ = [0.50000000, 0.84852814],
(47)
ys∗ = [2.00000000, 1.13137085],
fs∗ = −3.35020203.
xr∗ = [0.50000000, 0.74833148],
yr∗ = [2.00000000, 1.20000000],
fs∗ = −2.51998886.
RELAXATION PROCEDURE 101

As expected, the results differ in all five cases. However, we stress again that the
solutions presented above, although worse than the saddle point solutions ensure strict
feasibility of all constraints, for all values of uncertain parameters, which is an important
requirement in engineering design under uncertainty.

5. Applications in chemical engineering

In order to illustrate the application of the algorithms discussed in this paper, we present
the following engineering examples, detailing and discussing their solutions.

5.1. An application in optimum process design

The following example is from Grossmann and Sargent (1978).

The pipeline shown in figure 1 shall be designed in order to minimise the annual
cost C:
min max C = 196850D + 0.84632W00.86 + 0.4251W
W0 ,D η,a

994.018 5.66a
s.t. W = 381.024135 + ,
η 0.04D 4.84 (48)
W
W0 , 0.508
D
0.9144;
0.3
η
0.6, 0.02
a
0.06.

Within this problem, two different types of variables are present:

Figure 1. Pipeline with a centrifuge pump.

102 ŽAKOVIĆ AND RUSTEM

Table 4
Designing a pipeline.
C W0 D Time (sec)
1327490 885971 0.841103 0.0375

• Uncertain parameters:

∗ η – efficiency of the pump;

∗ a – the friction parameter.

• Design variables:

∗ W0 – the installed power of the pump;

∗ W – the required power of the pump;
∗ D – the diameter of the pipeline.

By introducing a dummy variable d̂, the problem can be transformed into the fol-
lowing:
min d̂
W0 ,D,d̂
s.t. max C − d̂
0,
η,a

994.018 5.66a (49)

W = 381.024135 + ,
η 0.04D 4.84
W
W0 , 0.508
D
0.9144;
0.3
η
0.6, 0.02
a
0.06.
Using the full global SIP algorithm gives the results shown in table 4. This solution
shows that, despite uncertainty in both the pump efficiency and the friction parameter, a
pipeline with a diameter of 0.84 and a pump with installed power 8.86 · 105 will incur a
cost of 1.32 · 106 .

5.2. Global structural model distinguishability

When modelling processing systems (i.e., chemical reactors, crystallisation units, fer-
mentation), it is possible to propose more than one mathematical model to describe
the system at hand. When faced with this situation, we simply wish to know whether the
mathematical structure of these models can be distinguished from one another, with
the goal of determining the best model (Asprey and Macchietto, 2000). We illustrate
these concepts through the use of a simple example, in which we wish to model the fer-
mentation of baker’s yeast in a semi-batch reactor. Two possible models can be proposed
for this (for details, see (Asprey and Macchietto, 2000)). We approximate the solution
RELAXATION PROCEDURE 103

of these models by their Fliess functional expansions, whose kth coefficient we denote
Lk (x, θ). Thus, the structural distinguishability problem can be posed as:
4 
  2
min∗ max D = Lk[1] (x, θ) − Lk[2] x, θ ∗
θ,θ x k=1
s.t. 10−3
θi
1.0, i = 1, . . . , 4,
(50)
10−3
θi∗
1.0, i = 1, . . . , 3,
1.0
x1
25.0,
10−2
x2
25.0,
where θ and θ ∗ represent vectors of parameters from the two models, and x represents
a vector of state or response variables. Again, Lki , i = 1, 2, 3, 4, k = 1, 2, represent
coefficients of the functional expansions for the solution trajectories of the two models
and are given by:

[1] θ1 x2
L1 = − θ4 x1 ,
θ2 + x2
 
θ1 x2 θ1 θ1 x2
L2[1] = − − θ4 x1 − x1 x2 − ,
θ2 + x2 θ2 + x2 (θ2 + x2 )2
θ1 x1 x2
L3[1] = − ,
θ3 (θ2 + x2 )

θ1 x1 x2 θ1 θ1 x2
L4[1] = − x1 x2 − + , (51)
θ3 (θ2 + x2 ) θ3 (θ2 + x2 ) θ3 (θ2 + x2 )2
 
L1[2] = θ1∗ x2 − θ3∗ x1 ,
 
L2[2] = − θ1∗ x2 − θ3∗ x1 − x2 θ1∗ x1 ,
θ ∗ x2 x1
L3[2] = − 1 ∗ ,
θ2
∗
θ x 2 x1
L4[2] = 2 1 ∗ .
θ2
Using the SIP algorithm, with local optimisation only and starting from several
random initial points, three solutions are obtained (see table 5).
Table 5 shows the time and number of iterations needed to obtain each solution.
Although solutions 3 and global required four iterations to converge, the global solution
Table 5
Local and global optima.
Solution D CPU Iterations
1 0.34740751 0.225 1
2 0.73329857 0.308 3
3 0.73914109 0.556 4
Global 0.73914109 0.752 4
104 ŽAKOVIĆ AND RUSTEM

used approximately 1/3 more time, as it is more expensive to obtain. In this case, the
CPU time highly depends on the number of local (candidate) solutions.

5.3. Robust optimal design of experiments

Main questions that arise in optimal design of experiments are – how to adjust time-
varying controls, initial conditions or duration of experiment to generate the maximum
amount of information?
An information matrix for experiment design is defined in the following way:


2 
2
MI (θ, φ) ≡ σ̃1rs QTr Qs , (52)
r=1 s=1

yr = yr (θ, φ, t), (53)

φ is the vector of experiment design variables and θ̂ is the vector of the estimates of
the model parameters. The matrix Qr is the matrix of sensitivity coefficients of the rth
response variable in the model computed at each of the nsp sampling points:
 
∂yr ∂yr ∂yr
··· ← sampling point t1
 ∂ θ̂1 
 ∂ θ̂2 ∂ θ̂p 
 
 ∂yr ∂yr ∂yr 
 ···  ← sampling point t2
 
Qr ≡  ∂ θ̂1 ∂ θ̂2 ∂ θ̂p  (54)
 . .. .. .. 
 .. . 
 . . 
 
 ∂yr ∂yr ∂yr 
··· ← sampling point tnsp
∂ θ̂1 ∂ θ̂2 ∂ θ̂p

The resulting robust design is the solution of the optimisation problem

 
φR = arg max min det MI (θ, φ) . (55)
φ∈ θ∈.

The above problem may be rewritten as

max /
φ∈,/∈R   (56)
s.t. /
det MI (θ, φ) ∀θ ∈ .,

where  ⊂ R29 and . ⊂ R4 represent feasible regions (upper and lower bounds on
each element of φ and θ, respectively), and thus there are no functional constraints on
the variables. In a typical situation, the dimension of  is much larger than the dimension
of ., indicating that the problem of maximising / is much more complex.
RELAXATION PROCEDURE 105

Table 6
Designing an experiment using algorithm A1.
Full global optimisation
Iteration Results
I max 39.38954
min −51.78269
at (0.05, 0.98, 0.98, 0.98)
II max 4.57472
min −6.73966
at (0.98, 0.05, 0.05, 0.01)
III max 4.48416
min −4.07338
at (0.98, 0.05, 0.98, 0.01)
IV max 4.47273
min –
at –
Time = 7622.21 s

Table 7
Designing an experiment using algorithm A2.
First violation of constraints
Iteration Results
I max 39.38954
min −35.03380
at (0.98, 0.05, 0.05, 0.01)
II max 11.362926
min −1.001543
at (0.98, 0.05, 0.05, 0.12)
III max 11.325922
min −6.343216
at (0.98, 0.05, 0.98, 0.01)
IV max 10.034939
min −1.647946
at (0.98, 0.98, 0.05, 0.33)
V max 10.033279
min −25.488726
at (0.05, 0.98, 0.98, 0.98)
VI max 4.4723
min –
at –
Time = 62318.36 s
106 ŽAKOVIĆ AND RUSTEM

In problems such as these, it seems rather obvious that the full global optimisation
algorithm should be used, as the price of computing the global optimum of a 4-variable
function should be much less than adding unnecessary constraints to the max / prob-
lem.
In tables 6 and 7 we show the solution to (55) for the model describing the semi-
batch fermentation of baker’s yeast (Asprey and Macchietto, 2000).
First the full global optimisation algorithm is used and the results are shown in
table 6.
Table 7 shows results obtained using algorithm A2.
As can be seen, the variation of the SIP which detects first violation required two
more iterations than the global SIP to converge. However, solving the max problem is
much more expensive, reflected in the CPU times needed to solve the problem with the
two different algorithms.

6. Conclusions

In this paper we extend the algorithm for solving semi-infinite programming problems
presented in Darlington et al. (1999), to include global violation of constraints. Also a
version is presented where we look for any violation of constraints, until the last iteration
when the global optimisation is performed. It does depend on the problem that we solve
which of the two versions to use, as results presented in this paper show. In general,
algorithm A2 converges in more iterations, but time is saved, as global optimisation is
performed only once.
The algorithm is used to solve minimax problems, both with coupled and decou-
pled constraints. In the latter case the results we obtain are identical to those of Rustem
and Howe (2001). In the former, more general case, the transformation from mini-
max to SIP problem obtains different results. However, the transformation does ensure
worst-case feasibility, the property of much importance in engineering design under un-
certainty, where no constraint violation can be tolerated for any values of the uncertain
parameters.
Few examples from chemical engineering are also presented. A number of differ-
ent versions of the algorithm are used to demonstrate advantages and disadvantages of
various implementations. For example, the model distinguishability problem is solved
using both local and global optimisation. Three local solutions are found. The global
solution is obtained at a price, as it needs approximately 1/3 more time. This price may
even increase more in the future, when more complex problems are considered.
We finish the paper with an example of robust optimal design. Algorithms A1 and
A2 are used, and A2 is more expensive for that particular example, as in examples 1–16
the opposite is true. However, the question of deciding which of the two versions to use
shall depend on the end user.
RELAXATION PROCEDURE 107

Appendix

Rustem–Howe algorithm B

This algorithm solves the following problem:

min max f (x, y). (57)
x∈Rn y∈Y

In the algorithm we use the max-function and the set of maximizers:

 
(x) = max f (x, y), Y(x) ≡ y(x) ∈ Y | y(x) = arg max f (x, y) . (58)
y∈Y y∈Y

Pseudo-code for solving (57) is summarised in algorithm 3.

Algorithm 3 Rustem–Howe algorithm B

1: Select x0 , y0 , H0 . Set k = 0
2: Set accuracy 0 < ε  1, line search parameter c ∈ (0, 1)
3: Set stepsize factor σ ∈ (0, 1), penalty coefficient C ∈ [0, ∞).
4: repeat
5: (xk ) = maxy∈Y f (xk , y).
6: yk+1 = arg maxy∈Y {f (xk , y) − 12 ∇x f (xk , y)2H −1 − C[(xk ) − f (xk , y)]2 }.
k
7: d̄ = −Hk−1 ∇x f (xk , yk+1 )
8: ȳ = arg maxy∈Y (xk ) ∇x f (xk , y), d̄!.
9: if ( ∇x f (xk , ȳ), d̄!  −ε) then
y 
10: λk+1 = arg minλy { y∈Y (xk ) λy ∇x f (xk , y)2H −1 |λy  0,
 y k
11: λ = 1}.
 y
y
12: gk = y∈Y (xk ) λk+1 ∇x f (xk , y)2H −1
k
13: dk = −Hk−1 gk
14: /k = − 12 gk 2H −1
k
15: else
16: dk = d̄
17: /k = f (xk , yk+1 ) + ∇x f (xk , yk+1 ), dk ! + 12 dk 2Hk
18: − C[(xk ) − f (xk , yk+1 )]2 − (xk )
19: end if
20: αk = max{α | (xk + αdk ) − (xk )
cα/k ; α = (σ )i , i = 0, 1, 2, . . .}
21: xk+1 = xk + αdk
22: update Hk → Hk+1
23: k =k+1
24: until (/k  −ε)
108 ŽAKOVIĆ AND RUSTEM

Hk is a positive definite approximation to the Hessian with respect to x of the

Lagrangian associated with (57).

Saddle Point algorithm C

Algorithm C finds the saddle point of the following problem:

min max f (x, y),
x y

s.t. (x, y) ∈ 4 = (x, y) | x ∈ Rn , y ∈ Rm , g(x, y)
0 , (59)
g : Rn+m → Rng ,
The problem is transformed into an unconstrained formulation:

ng
min max f (x, y) − µ log(si ),
x y i=1 (60)
s.t. g(x, y) + s = 0,
s > 0.
The necessary and sufficient conditions for vectors x∗ (= x∗ (µ)) and y∗ (= y∗ (µ))
to be a saddle point of f are:
   
∇x f x∗ (µ), y∗ (µ) + z∇x g x∗ (µ), y∗ (µ) = 0;
   
∇y f x∗ (µ), y∗ (µ) − z∇y g x∗ (µ), y∗ (µ) = 0;
  (61)
ZSe − µe = 0; g x∗ (µ), y∗ (µ) + s = 0;
s, z  0,
where e ∈ Rng is a vector of all ones and Z and S are diagonal matrices, defined
as S = diag(s1 , s2 , . . . , sng ), Z = diag(z1 , z2 , . . . , zng ). Let the system (61) be de-
noted Fµ (ζ ), where ζ = (x, y, s, z)t . It is convenient to characterize the problem as
minζ { 12 Fµ (ζ )22 }. Starting from an initial point ζ0 , the algorithm generates a sequence
{ζk },
ζk+1 = ζk + αk dk , (62)
where dk = (;xk , ;yk , ;sk , ;zk )t is a solution of the following system:
∇Fµ dk = −Fµ . (63)
In order to keep the new point feasible we must assure that the new iterates sk and
zk are greater then 0 (steps 9–12 in the algorithm). Once the maximum allowed step size
(say α̂k ) is determined, we then choose β ∈ (0, 12 ) and compute θ = min{0, 1, 2, . . . , }
such that:
     
Fµ (ζk + αk dk )2 − Fµ (ζk )2
ραk ∇F t Fk , dk , (64)
2 2 k

where αk = α̂k β θ .
RELAXATION PROCEDURE 109

Pseudo-code for solving (59) is presented in algorithm 4.

Algorithm 4 Saddle Point algorithm C

1: Choose x̃ 0 , ỹ 0
2: Choose (s̃0 , z̃0 ) > 0.
3: Choose µ0  0, τ, ε, η, β ∈ (0, 1), ρ ∈ (0, 1), σ .
4: Set l = 0, k = 0
5: while (Fµl (ζ̃ l )2 /(1 + ζ̃ l 2 ) > ε) do
6: ζk = ζ̃ l
7: while (Fµl (ζk )2 > ηµl ) do
8: Solve (63) to obtain dk
 
ski
9: αs,k = min − i |;sk < 0
max i
1
i
ng ;sk
 
zki
10: αz,k = min − i |;zk < 0
max i
1
i
ng ;zk
11: αk = min{αs,k , αz,k
max max max
}.
12: α̂k = min{τ αk , 1}
max

13: αk = β θ α̂k , θ = min{0, 1, 2, . . .} such that (64) is satisfied.

14: ζk+1 = ζk + αk dk
15: k =k+1
16: end while
17: µl → µl+1
18: l =l+1
19: ζ̃ l+1 = ζk
20: end while

It contains outer iteration (steps 5–20) and inner iteration (steps 7–16). At each
inner iteration a solution to problem (60) (with µ fixed) is found. Then, in step 17 µ is
decreased, and, if the convergence criteria in step 5 is not satisfied, the whole process is
repeated.

References
Asprey, S.P. and S. Macchietto. (2000). “Statistical Tools for Optimal Dynamic Model Building.” Comput.
Chem. Eng. 24, 1261–1267.
Blankenship, J.W. and J.E. Falk. (1976). “Infinitely Constrained Optimization Problems.” JOTA 19, 261–
281.
Boyd, S., L. El Ghaoui, E. Feron, and V. Balakrishnan. (1994). Linear Matrix Inequalities in Systems and
Control Theory. Philadelphia, PA: SIAM.
 110 ŽAKOVIĆ AND RUSTEM

Coope, I.D. and G.A. Watson. (1985). “A Projected Lagrangian Algorithm for Semi-Infinite Programming.”
Math. Programming 32, 337–356.
Danskin, J.M. (1967). The Theory of Max-Min. Berlin: Springer.
Darlington, J., C.C. Pantelides, B. Rustem, and B.A. Tanyi. (1999). “An Algorithm for Constrained Non-
linear Optimization under Uncertainty.” Automatica 35, 217–228.
Demyanov, V.F. and A.B. Pevnyi. (1971). “Numerical Methods for Finding Saddle Points.” USSR Comput.
Math. and Math. Phys. 12, 1099–1127.
Grossmann, I.E. and R.W.H. Sargent. (1978). “Optimum Design of Chemical Plants with Uncertain Para-
meters.” AIChE J. 24, 1021–1028.
Gustafson, S.A. (1981). “A Three-Phase Algorithm for Semi-Infinite Programming.” In A.V. Fiacco and
O. Kortanek (eds.), Semi-Infinite Programming and Applications. Lecture Notes in Economics and
Mathematical Systems, Vol. 215, Berlin: Springer, pp. 138–157.
Halemane, K.P. and I.E. Grossmann. (1983). “Optimal Process Design Under Uncertainty.” AIChE J. 29,
425–433.
Hettich, R. and K.O. Kortanek. (1993). “Semi-Infinite Programming: Theory, Methods and Applications.”
SIAM Review 35, 380–429.
John, F. (1948). “Extremum Problems with Inequalities as Subsidiary Condition.” In Studies and Essays,
Courant Anniversary Volume. New York: Wiley.
Kall, P. and S. Wallace. (1994). Stochastic Programming. New York: Wiley.
Kwak, B.M. and E.J. Haug. (1976). “Optimum Design in the Presence of Parametric Uncertainty.” JOTA
19, 527–546.
Mayne, D.Q., E. Polak, and R. Trahan. (1979). “An Outer Approximation Algorithm for Computer-Aided
Design Problems.” JOTA 28, 331–352.
Monahan, G.E. (1996). “Finding Saddle Points on Polyhedra: Solving Certain Continuous Minimax Prob-
lems.” Naval Research Logistics 43, 821–837.
NAG Library. “Subroutine E04UCF.” Available at WWW: http://www.nag.co.uk/numeric/fl/manual/html/
FLlibrarymanual.asp
Panin, V.M. (1981). “Linearization Method for Continuous Min-Max Problems.” Kibernetika 2, 75–78.
Polak, E. and D.Q. Mayne. (1976). “An Algorithm for Optimization Problems with Functional Inequality
Constraints.” IEEE Transactions on Automatic Control 21, 184–193.
Prekopa, A. (1995). Stochastic Programming. Budapest: Akademiai Kiado.
Rinnooy Kan, A. and G.T. Timmer. (1987). “Stochastic Global Optimization Methods. Part II: Multilevel
Methods.” Mathematical Programming 78, 39–57.
Rockafeller, R.T. (1972). Convex Analysis. Princeton, NJ: Princeton University Press.
Rosen, J.B. (1965). “Existence and Uniqueness of Equilibrium Points for Concave n-Person Games.”
Econometrica 33, 520–534.
Rustem, B. and M.A. Howe. (2001). Algorithms for Worst-Case Design and Applications to Risk Manage-
ment. Princeton, NJ: Princeton University Press.
Zakovic, S., C.C. Pantelides, and B. Rustem. (2000). “An Interior Point Algorithm for Computing Saddle
Points of Constrained Continuous Minimax.” Annals of Operations Research 99, 59–77.
Sasai, H. (1974). “An Interior Penalty Method for Minimax Problems with Constraints.” SIAM J. Control
12, 643–649.
Shimizu, K. and E. Aiyoshi. (1980). “Necessary Conditions for Min-Max Problems and Algorithms by a
Relaxation Procedure.” IEEE Transactions on Automatic Control 25, 62–66.
Vandenberghe, L. and S. Boyd. (1998). “Connections between Semi-Infinite and Semidefinite Program-
ming. In R. Reemtsen and J.J. Rueckmann (eds.), Semi-Infinite Programming. Dordrecht: Kluwer
Academic, pp. 277–294.

View publication stats