Numerical Algorithms for Sequential

Quadratic Optimization

Esben Lundsager Hansen s022022

Carsten Völcker s961572

Kongens Lyngby 2007

IMM-M.Sc-2007-70
Technical University of Denmark
Informatics and Mathematical Modelling
Building 321, DK-2800 Kongens Lyngby, Denmark
Phone +45 45253351, Fax +45 45882673
[email protected]
www.imm.dtu.dk

IMM-M.Sc: ISSN 0909-3192

 Abstract

This thesis investigates numerical algorithms for sequential quadratic program-

ming (SQP). SQP algorithms are used for solving nonlinear programs, i.e. math-
matical optimization problems with nonlinear constraints.

SQP solves the nonlinear constrained program by solving a sequence of associat-

ing quadratic programs (QP’s). A QP is a constrained optimization problem in
which the objective function is quadratic and the constraints are linear. The QP
is solved by use of the primal active set method or the dual active set method.
The primal active set method solves a convex QP where the Hessian matrix is
positive semi definite. The dual active set method requires the QP to be strictly
convex, which means that the Hessian matrix must be positive definite. The
active set methods solve an inequality constrained QP by solving a sequence of
corresponding equality constrained QP’s.

The equality constrained QP is solved by solving an indefinite symmetric linear

system of equations, the so-called Karush-Kuhn-Tucker (KKT) system. When
solving the KKT system, the range space procedure or the null space procedure
is used. These procedures use Cholesky and QR factorizations. The range
space procedure requires the Hessian matrix to be positive definite, while the
null space procedure only requires it to be positive semi-definite.

By use of Givens rotations, complete factorization is avoided at each iteration

of the active set methods. The constraints are divided into bounded variables
and general constraints. If a bound becomes active the bounded variable is
fixed, otherwise it is free. This is exploited for further optimization of the
factorizations.
ii Abstract

The algorithms has been implemented in Matlab and tested on strictly convex
QP’s of sizes up to 1800 variables and 7200 constraints. The testcase is the
quadruple tank process, described in appendix A.

Main Findings of this Thesis

When the number of active constraints reaches a certain amount compared to

the number of variables, the null space procedure should be used. The range
space procedure is only prefereble, when the number of active constraints is very
small compared to the number of variables.

The update procedures of the factorizations give significant improvement in

computational speed.

Whenever the Hessian matrix of the QP is positive definite the dual active
set method is prefereble. The calculation of a starting point is implicit in the
method and furthermore convergence is guaranteed.

When the Hessian matrix is positive semi definite, the primal active set can be
used. For this matter an LP solver should be implemented, which computes a
starting point and an active set that makes the reduced Hessian matrix positive
definite. This LP solver has not been implemented, as it is out of the range of
this thesis.
 Dansk Resumé

Dette Projekt omhandler numeriske algoritmer til sekventiel kvadratisk pro-

grammering (SQP). SQP benyttes til at løse ikke-lineære programmer, dvs.
matematiske optimeringsproblemer med ikke-linære begrænsninger.

SQP løser det ikke-lineært begrænsede program ved at løse en sekvens af tilhørende
kvadratiske programmer (QP’er). Et QP er et begrænset optimeringsproblem,
hvor objektfunktionen er kvadratisk og begrænsningerne er lineære. Et QP løses
ved at bruge primal aktiv set metoden eller dual aktiv set metoden. Primal aktiv
set metoden løser et konvekst QP, hvor Hessian matricen er positiv semi definit.
Dual aktiv set metoden kræver et strengt konvekst QP, dvs. at Hessian matri-
cen skal være positiv definit. Aktiv set metoderne løser et ulighedsbegrænset
QP ved at løse en sekvens af tilhørende lighedsbegænsede QP’er.

Løsningen til det lighedsbegrænsede QP findes ved at løse et indefinit sym-

metrisk lineært ligningssystem, det såkaldte Karush-Kuhn-Tucker (KKT) sys-
tem. Til at løse KKT systemet benyttes range space proceduren eller null space
proceduren, som bruger Cholesky og QR faktoriseringer. Range space proce-
duren kræver, at Hessian matricen er positiv definit. Null space proceduren
kræver kun, at den er positiv semi definit.

Ved brug af Givens rotationer ungås fuld faktorisering for hver iteration i aktiv
set metoderne. Begrænsningerne deles op i begrænsede variable og egentlige
begrænsninger beskrevet ved funktionsudtryk. Begrænsede variable betyder,
at en andel af variablene er fikserede, mens resten er frie pr. iteration. Dette
udnyttes til yderligere optimering af faktoriseringerne mellem hver iteration.

Algoritmerne er implementeret i Matlab og testet på strengt konvekse QP’er

iv Dansk Resumé

bestående af op til 1800 variable og 7200 begrænsninger. Testeksemplerne er

genereret udfra det firdobbelte tank system, som er beskrevet i appendix A.

Hovedresultater

Når antallet af aktive begrænsninger når en vis mængde i forhold til antallet
af variable, bør null space proceduren benyttes. Range space proceduren bør
kun benyttes, når antallet af aktive begrænsninger er lille i forhold til antallet
af variable.

Når fuld faktorisering undgås ved at benytte opdateringer, er der betydelige

beregningsmæssige besparelser.

Hvis Hessian matricen af et QP er positiv definit, bør dual aktiv set meto-
den benyttes. Her foregår beregningerne af startpunkt implicit i metoden, og
desuden er konvergens garanteret.

Hvis Hessian matricen er positiv semi definit, kan primal aktiv set metoden
benyttes. Men her skal der benyttes en LP-løser til at beregne et startpunkt og
et tilhørende aktivt set, som medfører at den reducerede Hessian matrix bliver
positiv definit. Denne LP-løser er ikke blevet implementeret, da den ligger
udenfor området af dette projekt.
v
vi Contents
Contents

Abstract i

Dansk Resumé iii

1 Introduction 1

1.1 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Equality Constrained Quadratic Programming 5

2.1 Range Space Procedure . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Null Space Procedure . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3 Computational Cost of the Range and the Null Space Procedures 15

2.3.1 Computational Cost of the Range Space Procedure . . . . 15

2.3.2 Computational Cost of the Null Space Procedure . . . . . 16

2.3.3 Comparing Computational Costs . . . . . . . . . . . . . . 18

viii CONTENTS

3 Updating Procedures for Matrix Factorization 21

3.1 Givens rotations and Givens reflections . . . . . . . . . . . . . . . 22

3.2 Updating the QR Factorization . . . . . . . . . . . . . . . . . . . 25

3.3 Updating the Cholesky factorization . . . . . . . . . . . . . . . . 33

4 Active Set Methods 41

4.1 Primal Active Set Method . . . . . . . . . . . . . . . . . . . . . . 42

4.1.1 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.1.2 Improving Direction and Step Length . . . . . . . . . . . 43

4.1.3 Appending and Removing a Constraint . . . . . . . . . . 49

4.2 Primal active set method by example . . . . . . . . . . . . . . . . 57

4.3 Dual active set method . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3.1 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3.2 Improving Direction and Step Length . . . . . . . . . . . 62

4.3.3 Linear Dependency . . . . . . . . . . . . . . . . . . . . . . 69

4.3.4 Starting Guess . . . . . . . . . . . . . . . . . . . . . . . . 73

4.3.5 In summary . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.3.6 Termination . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.4 Dual active set method by example . . . . . . . . . . . . . . . . . 78

5 Test and Refinements 83

5.1 Computational Cost of the Range and the Null Space Procedures
with Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

5.2 Fixed and Free Variables . . . . . . . . . . . . . . . . . . . . . . . 86

CONTENTS ix

5.3 Corresponding Constraints . . . . . . . . . . . . . . . . . . . . . . 88

5.4 Distinguishing Between Bounds and General Constraints . . . . . 93

6 Nonlinear Programming 95

6.1 Sequential Quadratic Programming . . . . . . . . . . . . . . . . . 95

6.2 SQP by example . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

7 Conclusion 103

7.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Bibliography 107

A Quadruple Tank Process 109

B QP Solver Interface 117

C Implementation 119

C.1 Equality Constrained QP’s . . . . . . . . . . . . . . . . . . . . . 119

C.2 Inequality Constrained QP’s . . . . . . . . . . . . . . . . . . . . . 120

C.3 Nonlinear Programming . . . . . . . . . . . . . . . . . . . . . . . 120

C.4 Updating the Matrix Factorizations . . . . . . . . . . . . . . . . . 120

C.5 Demos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

C.6 Auxiliary Functions . . . . . . . . . . . . . . . . . . . . . . . . . 122

D Matlab-code 123

D.1 Equality Constrained QP’s . . . . . . . . . . . . . . . . . . . . . 123

x CONTENTS

D.2 Inequality Constrained QP’s . . . . . . . . . . . . . . . . . . . . . 125

D.3 Nonlinear Programming . . . . . . . . . . . . . . . . . . . . . . . 140

D.4 Updating the Matrix Factorizations . . . . . . . . . . . . . . . . . 143

D.5 Demos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

D.6 Auxiliary Functions . . . . . . . . . . . . . . . . . . . . . . . . . 171

Chapter 1

Introduction

In optimal control there is a high demand for real-time solutions. Dynamic

systems are more or less sensitive to outer influences, and therefore require fast
and reliable adjustment of the control parameters.

A dynamic system in equilibrium can experience disturbances explicitly, e.g.

sudden changes in the environment in which the system is embedded or online
changes to the demands of the desired outcome of the system. Implicit dis-
turbances have also to be taken care of in real-time, e.g. changes of the input
needed to run the system. In all cases, fast and reliable optimal control is es-
sential in lowering the running cost of a dynamic system.

Usually the solution of a dynamic process must be kept within certain lim-
its. In order to generate a feasible solution to the process, these limits have to
be taken into account. If a process like this can be modeled as a constrained
optimization problem, model predictive control can be used in finding a feasible
solution, if it exists.

Model predictive control with nonlinear models can be performed using sequen-
tial quadratic programming (SQP). Model predictive control with linear models
may be conducted using quadratic programming (QP). A variety of different
numerical methods exist for both SQP and QP. Some of these methods com-
prise the subject of this project.
2 Introduction

The main challenge in SQP is to solve the QP, and therefore methods for solv-
ing QP’s constitute a major part of this work. In itself, QP has a variety of
applications, e.g. Portfolio Optimization by Markowitz, found in Nocedal and
Wright [14], solving constraint least squares problems and in Huber regression
Li and Swetits [1]. A QP consists of a quadratic objective function, which we
want to minimize subject to a set of linear constraints. A QP is stated as

1 T
min x Gx + g T x
x∈Rn 2
s.t. l≤x≤u
bl ≤ AT x ≤ bu ,

and this program is solved by solving a set of equality constrained QP’s

1 T
min x Gx + g T x
x∈Rn 2
T
s.t. Ā x = b̄.

The methods we describe are the primal active set method and the dual active
set method. Within these methods the Karush-Kuhn-Tucker (KKT) system1

    
G −Ā x g
T =−
−Ā 0 λ b̄

is solved using the range space procedure or the null space procedure. These
methods in themselves fulfill the demand of reliability, while the demand of ef-
ficiency is obtained by refinement of these methods.

1.1 Research Objective

We will investigate the primal and dual active set methods for solving QP’s.
Thus we will discuss the range and the null space procedures together with
different refinements for gaining efficiency and reliability. The methods and
1 This is the KKT system of the primal program, the KKT system of the dual program is

found in (4.69) at page 63.

1.2 Thesis Structure 3

procedures for solving QP’s will be implemented and tested in order to determine
the best suited combination in terms of efficiency for solving different types of
problems. The problems can be divided into two categories, those with a low
number of active constraints in relation to the number of variables, and problems
where the number of active constraints is high in relation to the number of
variables. Finally we will discuss and implement the SQP method to find out
how our QP solver performs in this setting.

1.2 Thesis Structure

The thesis is divided into five main areas: Equality constrained quadratic pro-
gramming, updating of matrix factorizations, active set methods, test and re-
finements and nonlinear programming.

Equality Constrained Quadratic Programming

In this chapter we present two methods for solving equality constrained QP’s,
namely the range space procedure and the null space procedure. The methods
are implemented and tested, and their relative benefits, and drawbacks are
investigated.

Updating of Matrix Factorizations

Both the null space and the range space procedure use matrix factorizations in
solving the equality constrained QP. Whenever the constraint matrix is changed
by either appending or removing a constraint, the matrix factorizations can be
updated using Givens rotations. By avoiding complete re-factorization, compu-
tational savings are achieved. This is the subject of this chapter and methods
for updating the QR and the Cholesky factorizations are presented.

Active Set Methods

Inequality constrained QP’s can be solved using active set methods. These
methods find a solution by solving a sequence of equality constrained QP’s,
where the difference between two consecutive iterations is a single appended or
4 Introduction

removed constraint. In this chapter we present the primal active set method
and the dual active set method.

Test and Refinements

In this chapter we test how the presented methods perform in practice, when
combined in different ways. We also implement some refinements, and their
impact on computational speed and stability are likewise tested.

Nonlinear Programming

SQP is an efficient method of nonlinear constrained optimization. The basic

idea is Newton’s method, where each step is generated as an inequality con-
strained QP. Implementation, discussion and testing of SQP are the topics of
this chapter.
Chapter 2
Equality Constrained
Quadratic Programming

In this section we present various algorithms for solving convex1 equality con-
strained QP’s. The problem to be solved is

1 T
min f (x) = x Gx + g T x (2.1a)
x∈Rn 2
s.t. AT x = b, (2.1b)

where G ∈ Rn×n is the Hessian matrix of the objective function f . The Hes-
sian matrix must be symmetric and positive semi definite2 . A ∈ Rn×m is the
constraint matrix (coefficient matrix of the constraints), where n is the number
of variables and m is the number of constraints. A has full column rank, that is
the constraints are linearly independent. The right hand side of the constraints
is b ∈ Rm and g ∈ Rn denotes the coefficients of the linear term of the objective
function.

1 The range space procedure presented in section 2.1 requires a strictly convex QP.
2 The range space procedure presented in section 2.1 requires G to be positive definite.
6 Equality Constrained Quadratic Programming

From the Lagrangian function

1 T
L(x, λ) = x Gx + g T x − λT (AT x − b), (2.2)
2

which is differentiated according to x and the Lagrange multipliers λ

∇x L(x, λ) = Gx + g − Aλ (2.3a)
T
∇λ L(x, λ) = −A x + b, (2.3b)

the problem can be formulated as the Karush-Kuhn-Tucker (KKT) system

    
G −A x g
=− . (2.4)
−AT 0 λ b

The KKT system is basically a set of linear equations, and therefore general
solvers for linear systems could be used, e.g. Gaussian elimination. In order to
solve a KKT system as fast and reliable as possible, we want to use Cholesky
and QR factorizations. But according to Gould in Nocedal and Wright [14] the
KKT matrix is indefinite, and therefore it is not possible to solve it by use of
either of the two factorizations. In this chapter, we present two procedures for
solving the KKT system by dividing it into subproblems, on which it is possible
to use these factorizations. Namely the range space procedure and the null space
procedure. We also investigate their individual benefits and drawbacks.
2.1 Range Space Procedure 7

2.1 Range Space Procedure

The range space procedure based on Nocedal and Wright [14] and Gill et al. [2]
solves the KKT system (2.4), corresponding to the convex equality constrained
QP (2.1). The Hessian matrix G ∈ Rn×n must be symmetric and positive
definite, because the procedure uses the inverted Hessian matrix G−1 . The
KKT system

    
G −A x g
=− (2.5)
−AT 0 λ b

can be interpreted as two equations

Gx − Aλ = −g (2.6a)
T
A x = b. (2.6b)

Isolating x in (2.6a) gives

x = G−1 Aλ − G−1 g, (2.7)

and substituting (2.7) into (2.6b) gives us one equation with one unknown λ

AT (G−1 Aλ − G−1 g) = b, (2.8)

which is equivalent to

AT G−1 Aλ = AT G−1 g + b. (2.9)

From the Cholesky factorization of G we get G = LLT and G−1 = (LT )−1 L−1 =
(L−1 )T L−1 . This is inserted in (2.9)

AT (L−1 )T L−1 Aλ = AT (L−1 )T L−1 g + b (2.10)

8 Equality Constrained Quadratic Programming

so

(L−1 A)T L−1 Aλ = (L−1 A)T L−1 g + b. (2.11)

From simplifying (2.11), by defining K = L−1 A and w = L−1 g, where K can

be found as the solution to LK = A, and w as the solution to Lw = g, we get

K T Kλ = K T w + b. (2.12)

By now K, w and b are known, and by computing z = K T w+b and H = K T K

we reformulate (2.12) into

Hλ = z. (2.13)

The matrix G is positive definite and the matrix A has full column rank, so H is
also positive definite. This makes it possible to Cholesky factorize H = M M T ,
and by backward and forward substitution λ is found from

M M T λ = z. (2.14)

Substituting M T λ with q gives

M q = z, (2.15)

and by forward substitution q is found. Now λ is found by backward substitution

in

M T λ = q. (2.16)

We now know λ and from (2.6a) we find x as follows

Gx = Aλ − g (2.17)
2.1 Range Space Procedure 9

gives us

LLT x = Aλ − g, (2.18)

and

LT x = L−1 Aλ − L−1 g, (2.19)

which is equivalent to

LT x = Kλ − w. (2.20)

As K, λ and w are now known, r is computed as r = Kλ−w, and by backward

substitution x is found in

LT x = r. (2.21)

The range space procedure requires G to be positive definite as G−1 is needed.

It is obvious, that the procedure is most efficient, when G−1 is easily computed.
In other words, when it is well-conditioned and even better, if G is a diagonal-
matrix or can be computed a priori. Another bottleneck of the procedure is the
factorization of the matrix AT G−1 A ∈ Rm×m . The smaller this matrix is, the
easier the factorization gets. This means, that the procedure is most effecient,
when the number of constraints is small compared to the number of variables.

Algorithm 2.1.1 summarizes how the calculations in the range space procedure
are carried out.
10 Equality Constrained Quadratic Programming

Algorithm 2.1.1: Range Space Procedure.

Note: The algorithm requires G to be positive definite and A to have full
column rank.
Cholesky factorize G = LLT
Compute K by solving LK = A
Compute w by solving Lw = g
Compute H = K T K
Compute z = K T w + b
Cholesky factorize H = M M T
Compute q by solving M q = z
Compute λ by solving M T λ = q
Compute r = Kλ − w
Compute x by solving LT x = r
2.2 Null Space Procedure 11

2.2 Null Space Procedure

The null space procedure based on Nocedal and Wright [14] and Gill et al. [3]
solves the KKT system (2.4) using the null space of A ∈ Rn×m . This procedure
does not need G ∈ Rn×n to be positive definite but only positive semi definite.
This means, that it is not restricted to strictly convex quadratic programs. The
KKT system to be solved is

    
G −A x g
=− , (2.22)
−AT 0 λ b

where A has full column rank. We compute the null space using the QR fac-
torization of A

   
R R
A=Q = (Y Z) , (2.23)
0 0

where Z ∈ Rn×(n−m) is the null space and Y ∈ Rn×m is the range space.
(Y Z) ∈ Rn×n is orthogonal and R ∈ Rm×m is upper triangular.

By defining x = Qp we write

 
py
x = Qp = (Y Z)p = (Y Z) = Y py + Zpz . (2.24)
pz

Using this formulation, we can reformulate (x λ)T in (2.22) as

 
    py
x Y Z 0
=  pz  , (2.25)
λ 0 0 I
λ

and because (Y Z) is orthogonal we also have

 
 T   py
Y Z 0 x
=  pz  . (2.26)
0 0 I λ λ
12 Equality Constrained Quadratic Programming

Now we will use (2.25) and (2.26) to express the KKT system in a more de-
tailed form, in which it becomes clear what part corresponds to the null space.
Inserting (2.25) and (2.26) in (2.22) gives

 
 T    py
Y Z 0 G −A Y Z 0  pz  =
0 0 I −AT 0 0 0 I
λ
 T  
Y Z 0 g
− , (2.27)
0 0 I b

which is equivalent to

    T 
Y T GY Y T GZ −(AT Y )T py Y g
 Z T GY Z T GZ −(AT Z)T   pz  = −  Z T g  . (2.28)
−AT Y −AT Z 0 λ b

By definition AT Z = 0, which simplifies (2.28) to

    T 
Y T GY Y T GZ −(AT Y )T py Y g
 Z T GY Z T GZ 0   pz  = −  Z T g  . (2.29)
−AT Y 0 0 λ b

This system can be solved using backward substitution, but to do this, we need
the following statement based on (2.23)

A=YR (2.30a)
T T
A = (Y R) (2.30b)
T T
A Y = (Y R) Y (2.30c)
T T T
A Y =R Y Y (2.30d)
T T
A Y =R . (2.30e)

and therefore the last block row from (2.29)

−AT Y py = −b (2.31)
2.2 Null Space Procedure 13

is equivalent to

RT py = b. (2.32)

As R is upper triangular this equation has a unique solution. When we have

computed py we can solve the middle block row in (2.29)

Z T GY py + Z T GZpz = −Z T g, (2.33)

The only unknown is pz , which we find by solving

(Z T GZ)pz = −Z T (GY py + g). (2.34)

The reduced Hessian matrix (Z T GZ) ∈ R(n−m)×(n−m) is positive definite and

therefore the solution to (2.34) is unique. We find it by use of the Cholesky
factorization (Z T GZ) = LLT . Now, having computed both py and pz , we
find λ from the first block row in (2.29)

Y T GY py + Y T GZpz − (AT Y )T λ = −Y T g, (2.35)

which is equivalent to

(AT Y )T λ = Y T G(Y py + Zpz ) + Y T g. (2.36)

Using (2.24) x = Y py + Zpz and (2.30) AT Y = RT this can be reformulated

into

Rλ = Y T (Gx + g) (2.37)

and because R is upper triangular, this equation also has a unique solution,
which is found by backward substitution.

This is the most efficient procedure, when the degree of freedom n − m is small,
i.e. when the number of constraints is large compared to the number of variables.
The reduced Hessian matrix Z T GZ ∈ R(n−m)×(n−m) grows smaller, when m
14 Equality Constrained Quadratic Programming

approaches n, and is thereby inexpensive to factorize. The most expensive part

of the computations is the QR factorization of A. While the null space Z can
be found in a number of different ways, we have chosen to use QR factorization
because it makes Y and Z orthogonal. In this way, we preserve numerical
stability, because the conditioning of the reduced Hessian matrix Z T GZ is at
least as good as the conditioning of G.

Algorithm 2.2.1 summarizes how the calculations in the null space procedure
are carried out.

Algorithm 2.2.1: Null Space Procedure.

Note: The algorithm requires G to be positive semi definite and A to have
full column rank.
 
R
QR factorize A = (Y Z)
0
Cholesky factorize Z GZ = LLT
T

Compute py by solving RT py = b
Compute g z = −Z T (GY py + g)
Compute r by solving Lr = g z
Compute pz by solving LT pz = r
Compute x = Y py + Zpz
Compute λ by solving Rλ = Y T (Gx + g)
2.3 Computational Cost of the Range and the Null Space Procedures 15

2.3 Computational Cost of the Range and the

Null Space Procedures

In this section we want to find out, how the range space and the null space
procedures perform individually. We also consider whether it is worthwhile to
shift between the two procedures dynamically.

2.3.1 Computational Cost of the Range Space Procedure

In the range space procedure there are three dominating computations in rela-
tion to time consumption. The computation of K ∈ Rn×m , computation of and
Cholesky factorization of H ∈ Rm×m .

Since L ∈ Rn×n is lower triangular, solving LK = A with respect to K is

done by simple forward substitution. The amount of work involved in forward
substitution is n2 per column, according to L. Eldén, L. Wittmeyer-Koch and
H.B. Nielsen [18]. Since K contains m columns, the total cost for computing
K is n2 m.

We define K T = K T ∈ Rm×n . Making the inner product of two vectors of

length n requires 2n operations. Since K T consists of m rows and as mentioned
above K contains m columns, then the computational workload involved in the
matrix multiplication H = K T K is 2nm2 .

The size of H is m × m, so the computational cost of the Cholesky factor-

ization is roughly 31 m3 , according to L. Eldén, L. Wittmeyer-Koch and H.B.
Nielsen [18].

Thus, we can estimate the total computational cost of the range space pro-
cedure as

1 3
3m + 2nm2 + n2 m (2.38)

and since 0 ≤ m ≤ n, the total computational workload will roughly be in the

range

0 ≤ 31 m3 + 2nm2 + n2 m ≤ 10 3
3 n . (2.39)
16 Equality Constrained Quadratic Programming

Here we also see, that the range space procedure gets slower, as the number of
constraints compared to the number of variables increases.

Figure 2.1 shows the theoretical computational speed of the range space proce-
dure. As stated above, it is obvious that the method gets slower as the number
of constraints increases in comparison to the number of variables.

x 10
9 RSP, theoretical computational cost, n = 1000
4
L−1A
3.5 KTK
chol(H)
3 total
RSP

2.5
cost [flops]

1.5

0.5

0
0 100 200 300 400 500 600 700 800 900 1000
m

Figure 2.1: Theoretical computational speed for the range procedure.

2.3.2 Computational Cost of the Null Space Procedure

The time consumption of the null space procedure is dominated by two compu-
tations. The QR factorization of the constraint matrix A and computation of
the reduced Hessian matrix Z T GZ ∈ R(n−m)×(n−m) .

With A ∈ Rn×m as our point of departure, the computational work of the QR

factorization is in the region 2m2 (n − 13 m), see L. Eldén, L. Wittmeyer-Koch
and H.B. Nielsen [18]. The QR factorization of A is

   
R
R
A=Q = Y Z , (2.40)
0 0

where Y ∈ Rn×m , Z ∈ Rn×(n−m) , R ∈ Rm×m and 0 ∈ R(n−m)×m .

We now want to find the amount of work involved in computing the reduced
Hessian matrix Z T GZ. We define Z T = Z T ∈ R(n−m)×n . The computational
workload of making the inner product of two vectors in Rn is 2n. Since Z T
2.3 Computational Cost of the Range and the Null Space Procedures 17

contains n − m rows and G consists of n columns, the computational cost of

the matrix product Z T G is 2n(n − m)n. Because (Z T G) ∈ R(n−m)×n and Z
consists of n − m columns, the amount of work involved in the matrix product
(Z T G)Z is 2n(n − m)(n − m). Therefore the computational cost of making the
reduced Hessian matrix is

2n(n − m)n + 2n(n − m)(n − m) = 2n(n − m)(2n − m). (2.41)

So the total computational cost of the null space procedure is roughly

2m2 (n − 31 m) + 2n(n − m)(2n − m) (2.42)

and since 0 ≤ m ≤ n, the total computational workload is estimated to be in

the range of

4 3
3n ≤ 2m2 (n − 13 m) + 2n(n − m)(2n − m) ≤ 4n3 . (2.43)

Therefore the null space procedure accelerates, as the number of constraints

compared to the number of variables increases. Figure 2.2 illustrates this.

x 10
9 NSP, theoretical computational cost, n = 1000
4
qr(A)
ZTGZ
3.5
total
NSP
3

2.5
cost [flops]

1.5

0.5

0
0 100 200 300 400 500 600 700 800 900 1000
m

Figure 2.2: Theoretical computational costs for the null space procedure.
18 Equality Constrained Quadratic Programming

2.3.3 Comparing Computational Costs

To take advantage of the individual differences in computational speeds, we

want to find out at what ratio between the number of constraints related to
the number of variables, the null space procedure gets faster than the range
space procedure. This is done by comparing the computational costs of both
procedures, hereby finding the point, at which they run equally fast. With
respect to m we solve the polynomial

1 3
3m + 2nm2 + n2 m − 2m2 (n − 13 m) − 2n(n − m)(2n − m) =
m3 − 2nm2 + 7n2 m − 4n3 = 0, (2.44)

where by we find the relation to be m ≃ 0.65n.

In figure 2.3 we have n = 1000 and 0 ≤ m ≤ n, so the ratio, i.e. the point
at which one should shift from using the range space to using the null space
procedure is of course to be found at m ≃ 650.

x 10
9 RSP vs NSP, theoretical computational costs, n = 1000
4
totalRSP
3.5 totalNSP

2.5
cost [flops]

1.5

0.5

0
0 100 200 300 400 500 600 700 800 900 1000
m

Figure 2.3: Total estimated theoretical computational costs for the range space
and the null space procedures.

We made some testruns of the the two procedures by setting up a KKT system
consisting of identity matrices. The theoretical computational costs are based
on full matrices, and we know, that Matlab treats identity matrices like full
matrices. So by means of this simple KKT system we are able to compare the
theoretical behavior of the respective procedures with the real behavior of our
implementation. The test setup consists of n = 1000 variables and 0 ≤ m ≤ n
2.3 Computational Cost of the Range and the Null Space Procedures 19

constraints, as illustrated in figures 2.4 and 2.5. The black curves represent
all computations carried out by the two procedures. It is clear, that they run
parallel to the magenta representing the dominating computations. This verifies
our comparison between the theoretical computational workloads with our test
setup.

In figure 2.4 we test the range space procedure, and the behavior is just as
expected, when compared to the theory represented in figure 2.1.

RSP, real computational cost, n = 1000

6 −1
L A
KTK

5 chol(H)
totalRSP
RSP
4
cost [s]

0
0 100 200 300 400 500 600 700 800 900 1000
m

Figure 2.4: Real computational cost for the range space procedure.

In figure 2.5 the null space procedure is tested. The computation of the re-
duced Hessian matrix Z T GZ behaves as expected. The behavior of the QR
factorization of A however is not as expected, compared to figure 2.2. When
0 ≤ m . 100, the computational time is too small to be properly measured. The
QR factorization still behaves some what unexpectedly, when 100 . m ≤ 1000.
Hence the advantage of shifting from the range space to the null space proce-
dure decreases. In other words, the ratio between the number of constraints
related to the number of variables, where the null space procedure gets faster
than the range space procedure, is larger than expected. Therefore using the
null space procedure might actually prove to be a disadvantage. This is clearly
in figure 2.6, where the dominating computations for the range space and the
null space procedures are compared to each other. From this plot it is clear, that
the ratio indicating when to shift procedure is much bigger in practice than in
theory. The cause of this could be an inappropriate implementation of the QR
factorization in Matlab, architecture of the processing unit, memory access
etc.. Perhaps implementation of the QR factorization in a low level language
could prove different.

It must be mentioned at this point, that the difference in unit on the abscissa
20 Equality Constrained Quadratic Programming

in all figures in this section does not influence the shape of the curves between
theory and testruns, because the only difference between them is the constant
ratio time/flop.

NSP, real computational cost, n = 1000

6 qr(A)
ZTGZ
total
5 NSP
NSP

4
cost [s]

0
0 100 200 300 400 500 600 700 800 900 1000
m

Figure 2.5: Real computational cost for the null space procedure.

RSP vs NSP, real computational costs, n = 1000

6 totalRSP
totalNSP

4
cost [s]

0
0 100 200 300 400 500 600 700 800 900 1000
m

Figure 2.6: Real computational costs for the range space and the null space
procedures.
Chapter 3

Updating Procedures for

Matrix Factorization

Matrix factorization is used, when solving an equality constrained QP. The

factorization of a square matrix of size n × n has computational complexity
O(n3 ).

As we will describe in chapter 4, the solution of an inequality constrained QP

is found by solving a sequence of equality constrained QP’s. The difference
between two consecutive equality constrained QP’s in this sequence is one single
appended or removed constraint. This is the procedure of the active set methods.

Because of this property, the factorization of the matrices can be done more
efficiently than complete refactorization. This is done by an updating procedure
where the current factorization of the matrices in the sequence, is partly used to
factorize the matrices for the next iteration. The computational complexity of
this updating procedure is O(n2 ) and therefore a factor n faster than a complete
factorization. This is important in particular for large-scale problems. The
updating procedure discussed in the following is based on Givens rotations and
Givens reflections.
22 Updating Procedures for Matrix Factorization

3.1 Givens rotations and Givens reflections

Givens rotations and reflections are methods for introducing zeros in a vector.
This is achieved by rotating or reflecting the coordinate system according to an
angle or a line. This angle or line is defined so that one of the coordinates of
a given point p becomes zero. As both methods serve the same purpose, we
have chosen to use only Givens rotations which we will explain in the following.
This theory is based on the work of Golub and Van Loan [4] and Wilkinson
[5]. A Givens rotation is graphically illustrated in figure 3.1. As illustrated we
can rotate the coordinate system so the coordinates of p in the rotated system
actually become (x′ , 0)T .

y p

y′ x′ θ
x
Figure 3.1: A Givens rotation rotates the coordinate system according to an
angle θ.

The Givens rotation matrix Q̂ ∈ R2×2 is defined as

 
c s x y
Q̂ = , c= p = cos(θ), s= p = sin(θ), (3.1)
−s c x2 + y 2 x2 + y 2

and (x, y)T , x 6= 0 ∧ y 6= 0, is the vector p in which we want to introduce a

zero
3.1 Givens rotations and Givens reflections 23

 y

√ x √  
x2 +y 2 x2 +y 2 x
Q̂p =  
− √ 2y 2 √ 2x 2 y
x +y x +y
 2

√x+y 2
 x2 +y 2 
= −xy+yx
√ 
 x2 +y 2 

 p 
x2 + y 2
= . (3.2)
0

If we want to introduce zeros in a vector v ∈ Rn , the corresponding rotation

matrix is constructed by the identity matrix I ∈ Rn×n , c and s. The matrix
introduces one zero, modifies one element m and leaves the rest of the vector
untouched

 
  
1 0 0 0 0x x
 ..   ..   .. 
 0
 . 0 0 0  .  
   . 

 0 0 1 0 0  x 
  = x  . (3.3)
   
 0 0 0 c s   x   m 
0 0 0 −s c x 0

Any Givens operation introduces only one zero at a time, but if we want to
introduce more zeros, a sequence of Givens operations Q̃ ∈ Rn×n can be con-
structed

Q̃ = Q̂1,2 Q̂2,3 . . . Q̂n−2,n−1 Q̂n−1,n , (3.4)

which yields

   
x γ

 x  
  0 

Q̃v = Q̃  .. = .. , γ = ±||v||2 . (3.5)
 .   . 
x 0
24 Updating Procedures for Matrix Factorization

For example when v ∈ R4 the process is as follows

       
x x x m
 x   x   m   0 
  Q̂3,4   Q̂   Q̂  , (3.6)
 x − −→  m  −−2,3
→ 0  −−1,2
→ 0 
x 0 0 0

where m is the modified element.

3.2 Updating the QR Factorization 25

3.2 Updating the QR Factorization

When a constraint is appended to, or removed from, the active set, updating
the factorization is done using Givens rotations. This section is based on the
work of Dennis and Schnabel [6], Gill et al. [7] and Golub and Van Loan [4] and
describes how the updating procedure is carried out.

Appending a Constraint

Before the new column is appended, we take a closer look at the constraint
matrix A ∈ Rn×m and its QR-factorization Q ∈ Rn×n and R ∈ Rm×m . The
constraint matrix A has full column rank, and can be written as

A = a1 . . . am , (3.7)

where ai is the ith column of the constraint matrix. The QR-factorization of A

is

 
R
A=Q . (3.8)
0

As Q is orthogonal we have Q−1 = QT , so

 
R
QT A = . (3.9)
0

Inserting (3.7) in (3.9) gives

QT A = QT a1 . . . am . (3.10)

Expression (3.9) can be written as

 
  x(1,1) ... x(1,m)
R  ... ... 
QT A = = . (3.11)
0  x(m,m) 
0
26 Updating Procedures for Matrix Factorization

Now we append the new column ā ∈ Rn to the constraint matrix A ∈ Rn×m ,

which becomes Ā ∈ Rn×m+1 . To optimize the efficiency of the updating proce-
dure, the new column is appended at index m + 1. The new constraint matrix
Ā is

Ā = (a1 . . . am ) ā . (3.12)

Replacing A in (3.10) with Ā gives

QT Ā = QT (a1 . . . am ) ā , (3.13)

which is equivalent to

QT Ā = QT (a1 . . . am ) QT ā . (3.14)

Thus from (3.10), (3.11) and (3.14) we have

   
T R v v
Q Ā = , = QT ā, (3.15)
0 w w

where v ∈ Rm and w ∈ Rn−m . This can be expressed as

 
x(1,1) ... x(1,m)
 ... ... v 
QT Ā =  . (3.16)
 x(m,m) 
0 w

Unless only the first element is different from zero in vector w, the triangular
structure is violated by appending ā. By using Givens rotations, zeros are intro-
duced in the vector (v, w)T with a view to making the matrix upper triangular
again. As a Givens operation only introduces one zero at a time, a sequence
Q̃ ∈ Rn×n of n − m + 1 Givens rotations is used

Q̃ = Q̂(m+1,m+2) Q̂(m+2,m+3) . . . Q̂(n−1,n) , (3.17)

3.2 Updating the QR Factorization 27

where Q̂(i+2,i+3) defines the Givens rotation matrix that introduces one zero
at index i + 3 and modifies the element at index i + 2. It is clear from (3.16)
that the smallest amount of Givens rotations are needed, when ā is appended
at index m + 1. This sequence is constructed so that

 
v
  
 γ 

v  0 
Q̃ = . (3.18)
w  .. 
 . 
0

Now that the sequence Q̃ has been constructed, when we multiply it with (3.16)
we get

 
T R v
Q̃Q Ā = Q̃
0 w
 
R v
= 0 γ 
0 0
 
x(1,1) . . . x(1,m)

 ... ... v 

= x(m,m)


 0 γ 
0 0
 
R̄
= . (3.19)
0

This indicates, that the triangular shape is regained and the Givens operations
only affects the elements in w.

The QR-factorization of the new constraint matrix is

 
R̄
Ā = Q̄ . (3.20)
0

Now that R̄ has been found, we only need to find Q̄ to complete the updating
28 Updating Procedures for Matrix Factorization

procedure. From (3.19) we have

 
T R̄
Q̃Q Ā = , (3.21)
0

and because both Q̃ and QT are orthogonal, this can be reformulated as

T
Ā = QQ̃ R̄. (3.22)

From this expression it is seen that Q̄ is

T
Q̄ = QQ̃ . (3.23)

The updating procedure, when appending one constraint to the constraint ma-
trix is summarized in algorithm 3.2.1.

Algorithm 3.2.1: Updating the QR-Factorization, when appending a col-

umn.
Note: Having A ∈ Rn×m and its QR factorization, where Q ∈ Rn×n and
R ∈ Rm×m . Appending a column ā to matrix A at index m + 1 gives a new
matrix Ā. The new factorization is Ā = Q̄R̄.
Compute Ā
 = (A,
 ā)
v
Compute = QT ā, where v ∈ Rm and w ∈ Rn−m .
w
 
  v
v
Compute the Givens rotation matrix Q̃ such that: Q̃ =  γ ,
w
0
where γ ∈ R.
T
Compute R̄ = Q̃Q Ā
T
Compute Q̄ = QQ̃

Removing a Constraint

To begin with we take a close look at the situation before the column is removed.
A ∈ Rn×m and its QR-factorization Q ∈ Rn×n and R ∈ Rm×m have the
3.2 Updating the QR Factorization 29

following relationship

A = (a1 . . . ai−1 ) ai (ai+1 . . . am ) , (3.24)

where (a1 . . . ai−1 ) are the first i−1 columns, ai is the ith column and (ai+1 . . . am )
are the last m − i columns. The QR-factorization of A is

 
R
A=Q , (3.25)
0

and as Q is orthogonal we have

 
R
QT A = . (3.26)
0

From (3.24) and (3.26) we thus have

QT A = QT (a1 . . . ai−1 ) ai (ai+1 . . . am ) , (3.27)

which is equivalent to

QT A = QT (a1 . . . ai−1 ) QT ai QT (ai+1 . . . am ) . (3.28)

Using expression (3.26) and (3.28) gives

30 Updating Procedures for Matrix Factorization

 
R11 R12 R13
 0 R22 R23 
QT A =   (3.29)
 0 0 R33 
0 0 0
 
x(1,1) . . . x(1,i−1) x(1,i) x(1,i+1) ... x(1,m)

 . . . ... ... ... ... ... 


 x(i−1,i−1) ... ... ... ... 

 0 x(i,i) ... ... ... 
= .
 x(i+1,i+1) ... ... 


 0 0 ... 

 x(m,m) 
0 0 0

Removing the column of index i changes the constraint matrix A ∈ Rn×m to

Ā = (a1 . . . ai−1 ) (ai+1 . . . am ) , (3.30)

where Ā ∈ Rn×(m−1) . Replacing A with Ā in (3.27) gives

QT Ā = QT (a1 . . . ai−1 ) (ai+1 . . . am ) , (3.31)

which is equivalent to

QT Ā = QT (a1 . . . ai−1 ) QT (ai+1 . . . am ) . (3.32)

Together expression (3.28), (3.29) and (3.32) indicate that

 
x ... x x ... x


  ... ... ... ... ... 

R11 R13 
 x x ... x 

T
 0 R23   0 x ... x 
Q Ā =  = . (3.33)
 0 R33  
 x ... x 

0 0 
 0 ... ... 

 x 
0 0
3.2 Updating the QR Factorization 31

The triangular structure is obviously violated and in order to regain it, Givens
rotations are used. It is only the upper Hessenberg matrix (R23 , R33 )T , that
needs to be made triangular. This is done using a sequence of m − i Givens
rotation matrices Q̃ ∈ Rn×n

 
  x ... ... x
x ... x  ... ... ... ... 
 ... ... ...   
     x ... ... x 
R13  x ... x   
   m ... ... m 
 R23  
 = Q̃  x ... x  
=

.
Q̃  m ... m (3.34)
 R33   x ... x   
   ... ... 
0  ... ...   
   m 
 x   
 0 
0
0

This means, that the triangular matrix R̄ is found from the product of Q̃ and
QT Ā, so that we get

 
x ... x x ... ... x

 ... ... ... ... ... ... 


 x x ... ... x 

   0 m ... ... m 
R̄ T  
= Q̃Q Ā =  m ... m . (3.35)
0  

 0 ... ... 


 m 

 
0 0

Now that we have found the upper triangular matrix R̄ of the new factorization
Ā = Q̄R̄, we only need to find the orthogonal matrix Q̄. As Q̃ and QT are
orthogonal (3.35) can be reformulated as

T
Ā = QQ̃ R̄, (3.36)

which means that

T
Q̄ = QQ̃ . (3.37)
32 Updating Procedures for Matrix Factorization

The updating procedure, when removing a constraint from the constraint matrix
is summarized in algorithm 3.2.2.

Algorithm 3.2.2: Updating the QR-Factorization, when removing a column.

Note: Having A ∈ Rn×m and its QR factorization, where Q ∈ Rn×n and
R ∈ Rm×m . Removing a column c from matrix A gives a new matrix Ā. The
new factorization is Ā = Q̄R̄.
Compute Ā by removing c from A
Compute P = QT Ā
Compute the Givens rotation matrix Q̃ such that: Q̃P is upper triangular
Compute R̄ = Q̃P
T
Compute Q̄ = QQ̃
3.3 Updating the Cholesky factorization 33

3.3 Updating the Cholesky factorization

The matrix AT G−1 A = H ∈ Rm×m , derived through (2.9) on page 7 and

(2.13) on page 8, is both symmetric and positive definite. Therefore it has the
Cholesky factorization H = LLT , where L ∈ Rm×m is lower triangular. This
section is based on the work of Dennis and Schnabel [6], Gill et al. [7] and
Golub and Van Loan [4], and presents the updating procedure of the Cholesky
factorization to be employed, when appending or removing a constraint from
constraint matrix A.

Appending a Constraint

When a constraint is appended to constraint matrix A ∈ Rn×m at column m+1,

the matrix H ∈ Rm×m becomes

 
H q
H̄ = , (3.38)
qT r

where H̄ ∈ R(m+1)×(m+1) , q ∈ Rm and r ∈ R. The new Cholesky factorization

is

 
T L̃ 0
H̄ = L̄L̄ , L̄ = , (3.39)
sT t

where L̄ ∈ R(m+1)×(m+1) , L̃ ∈ Rm×m , s ∈ Rm and t ∈ R. Together (3.38) and

(3.39) give

 
H q
H̄ =
qT r
  !
T
L̃ 0 L̃ s
=
sT t 0 t
T
!
L̃L̃ L̃s
= T . (3.40)
s L̃T
s s + t2
T
34 Updating Procedures for Matrix Factorization

From this expression L̄ can be found via L̃, s and t. Furthermore from (3.40)
and the fact, that H = LLT , we have

T
H = L̃L̃ = LLT , (3.41)

which means that

L̃ = L. (3.42)

From (3.40) and (3.42) we know that s can be found from the expression

q = L̃s = Ls, (3.43)

and from (3.40) we also have

r = sT s + t2 . (3.44)

On this basis t can be found as

p
t= r − sT s. (3.45)

Now L̃, s and t have been isolated, and the new Cholesky factorization has
been shown to be easily found from (3.39). Algorithm 3.3.1 summarizes how the
updating procedure of the Cholesky factorization is carried out, when appending
3.3 Updating the Cholesky factorization 35

a column to constraint matrix.

Algorithm 3.3.1: Updating the Cholesky factorization when appending a

column.
Note: The constraint matrix is A ∈ Rn×m and the corresponding matrix
AT G−1 A = H ∈ Rm×m has the Cholesky factorization LLT , where L ∈
Rm×m . Appending a column c to matrix A at index m + 1 changes H into
T
H̄ ∈ R(m+1)×(m+1) . The new Cholesky factorization is H̄ = L̄L̄ .
Let p be the last column of H̄
Let q be p except the last element
Let r be the last element of p
Solve for s in q = Ls
2
Solve for t in r= sT s + t
L 0
Compute L̄ =
sT t

Removing a Constraint

Before removing a constraint, i.e. the ith column, from the constraint matrix
A ∈ Rn×m , the matrix H ∈ Rm×m can be formulated as

 
H 11 a H 12
H =  aT c bT  , (3.46)
H T12 b H 22

where H 11 ∈ R(i−1)×(i−1) , H 22 ∈ R(m−i)×(m−i) , H 12 ∈ R(i−1)×(m−i) , a ∈

R(i−1) , b ∈ R(m−i) and c ∈ R. The matrix H is Cholesky factorized as follows

 
L11
H = LLT , L =  dT e , (3.47)
L12 f L22

where L ∈ Rm×m , L11 ∈ R(i−1)×(i−1) and L22 ∈ R(m−i)×(m−i) are lower tri-
angular and non-singular matrices with positive diagonal-entries. Also having
L12 ∈ R(m−i)×(i−1) . The vectors d and f have dimensions R(i−1) and R(m−i)
respectively and e ∈ R. The column (aT c bT )T and the row (aT c bT ) in (3.46)
are removed, when the constraint at column i is removed from A. This gives us
36 Updating Procedures for Matrix Factorization

H̄ ∈ R(m−1)×(m−1) , which is both symmetric and positive definite

 
H 11 H 12
H̄ = . (3.48)
H T12 H 22

This matrix has the following Cholesky factorization

 
T L̄11
H̄ = L̄L̄ , L̄ = , (3.49)
L̄12 L̄22

which is equivalent to

  !
T T
L̄11 L̄11 L̄12
H̄ = T
L̄12 L̄22 L̄22
!
T T
L̄11 L̄11 L̄11 L̄12
= T T T , (3.50)
L̄12 L̄11 L̄12 L̄12 + L̄22 L̄22

where H̄ ∈ R(m−1)×(m−1) , L̄11 ∈ R(i−1)×(i−1) and L̄22 ∈ R(m−i)×(m−i) are

lower triangular, non-singular matrices with positive diagonal entries. Matrix
L̄12 is of dimension R(m−i)×(i−1) .

From (3.46) and (3.47) we then have

 
H 11 a H 12
 aT T 
c b =H
H T12 b H 22
= LLT
  
L11 LT11 d LT12
=  dT e  e fT  , (3.51)
L12 f L22 LT22

which gives
3.3 Updating the Cholesky factorization 37

 
H 11 a H 12
 aT c bT  =
H T12 b H 22
 
L11 LT11 L11 d L11 LT12
 dT LT11 dT d + e2 dT LT12 + ef T . (3.52)
T
L12 LT11 T T
L12 d + f e L12 L12 + f f + L22 L22

From (3.48) and (3.50) we know that

  !
T T
H 11 H 12 L̄11 L̄11 L̄11 L̄12
= . (3.53)
H T12 H 22 T
L̄12 L̄11
T T
L̄12 L̄12 + L̄22 L̄22

Expressions (3.52) and (3.53) give

T
H 11 = L11 LT11 = L̄11 L̄11 , (3.54)

and

T
H 12 = L11 LT12 = L̄11 L̄12 , (3.55)

which means that

L̄11 = L11 and L̄12 = L12 . (3.56)

From (3.52) and (3.53) we also get

T T
H 22 = L12 LT12 + f f T + L22 LT22 = L̄12 L̄12 + L̄22 L̄22 , (3.57)

and together with (3.56) this gives

T
H 22 = L12 LT12 + f f T + L22 LT22 = L12 LT12 + L̄22 L̄22 , (3.58)
38 Updating Procedures for Matrix Factorization

which is equivalent to

T
H 22 = f f T + L22 LT22 = L̄22 L̄22 . (3.59)

From this expression we get

T
L̄22 L̄22 = (f L22 )(f L22 )T . (3.60)

From (3.47) we know that (f L22 ) is not triangular. Therefore we now construct
a sequence of Givens rotations Q̃ ∈ R(m−i+1)×(m−i+1) so that

 
x x
 .. .. 
 . . x 
(f L22 )Q̃ =  .. .. ..
 Q̃
 .. 
 . . . . 
x x x ... x
 
x
 .. 
 . x 
= .. ..

 .. 
 . . . 
x x ... x 0
= (L̃ 0), (3.61)

where L̃ is lower triangular. As the Givens rotation matrix Q̃ is orthogonal, we

T
have that Q̃Q̃ = I, and therefore we can reformulate (3.60) as

T T
L̄22 L̄22 = (f L22 )Q̃Q̃ (f L22 )T , (3.62)

which is equivalent to

T
L̄22 L̄22 = ((f L22 )Q̃)((f L22 )Q̃)T , (3.63)

and according to (3.61) this renders

T T
L̄22 L̄22 = (L̃ 0)(L̃ 0)T = L̃L̃ . (3.64)
3.3 Updating the Cholesky factorization 39

Finally we now know that

L̄22 = L̃, (3.65)

which means that L̄22 may be constructed as

(L̄22 0) = (L̃ 0) = (f L22 )Q̃. (3.66)

Hence we now have everything for constructing the new Cholesky factorization:

 
T L̄11
H̄ = L̄L̄ , L̄ = . (3.67)
L̄12 L̄22

Algorithm 3.3.2 summarizes the updating procedure of the Cholesky factoriza-

tion, when a column is removed from the constraint matrix.

Algorithm 3.3.2: Updating the Cholesky factorization when removing a col-

umn.
Note: Having the constraint matrix A ∈ Rn×m and the corresponding ma-
trix AT G−1 A = H ∈ Rm×m with the Cholesky factorization LLT , where
L ∈ Rm×m . Removing column c from matrix A at index i changes H into
T
H̄ ∈ R(m−1)×(m−1) . The new Cholesky factorization is H̄ = L̄L̄ .
 
L11
Let L =  d T
e  , where (dT e) is the row at index i and
L12 f L22
(e f T )T is the column at index i.
Let L̄11 = L11 .
Let L̄12 = L12 .
Let L̂ = (f L22 )
Compute the Givens rotation matrix Q̃ such that L̂Q̃=(L̄22 0), where L̄22 is
triangular.  
L̄11 0
Compute L̄=
L̄12 L̄22
40 Updating Procedures for Matrix Factorization
Chapter 4
Active Set Methods

In this chapter we investigate how to solve an inequality constrained convex QP

of type

1 T
min f (x) = x Gx + g T x (4.1a)
x∈Rn 2
s.t. ci (x) = ai x − bi ≥ 0, i ∈ I. (4.1b)

The solution of this problem x∗ is also the same as to the equality constrained
convex QP

1 T
min f (x) = x Gx + g T x (4.2a)
x∈Rn 2
s.t. ci (x) = ai x − bi = 0, i ∈ A(x∗ ). (4.2b)

In other words, this means that in order to find the optimal point we need to
find the active set A(x∗ ) of (4.1). As we shall see in the following, this is done
by solving a sequence of equality constrained convex QP’s. We will investigate
two methods for solving (4.1), namely the primal active set method (section 4.1)
and the dual active set method (section 4.3).
42 Active Set Methods

4.1 Primal Active Set Method

The primal active set method discussed in this section is based on the work of
Gill and Murray [8] and Gill et al. [9]. The algorithm solves a convex QP with
inequality constraints (4.1).

4.1.1 Survey

The inequality constrained QP is written on the form

1 T
min x Gx + g T x (4.3a)
x∈Rn 2
s.t. aTi x ≥ bi i∈I (4.3b)

where G ∈ Rn×n is symmetric and positive definite.

The objective function of the QP is given as

1 T
f (x) = x Gx + g T x (4.4)
2

and the feasible region is

Ω = {x ∈ Rn : aTi x ≥ bi , i ∈ I} (4.5)

The idea of the primal active set method is to compute a feasible sequence
{xk ∈ Ω}, where k = N0 , with decreasing value of the objective function,
f (xk+1 ) < f (xk ). For each step in the sequence we solve an equality constraint
QP

1 T
min x Gxk + g T xk (4.6a)
xk ∈Rn 2 k
s.t. aTi xk = bi i ∈ A(xk ) (4.6b)
4.1 Primal Active Set Method 43

where A(xk ) = {i ∈ I : aTi xk = bi } is the current active set. Because the

vectors ai are linearly independent for i ∈ A(xk ), the strictly convex equality
constrained QP can be solved by solving the corresponding KKT system using
the range space or the null space procedure.

The sequence of equality constrained QP’s is generated, so that the sequence

{xk } converges to the optimal point x∗ , where the following KKT conditions

X
Gx∗ + g − ai µ∗i = 0 (4.7a)
i∈I

aTi x∗ = bi i ∈ Wk (4.7b)
aTi x∗ ≥ bi i ∈ I\Wk (4.7c)
µ∗i ≥ 0 i ∈ Wk (4.7d)
µ∗i = 0 i ∈ I\Wk (4.7e)

are satisfied.

4.1.2 Improving Direction and Step Length

For every feasible point xk ∈ Ω, we have a corresponding working set Wk , which

is a subset of the active set A(xk ), Wk ⊂ A(xk ). Wk is selected so that the
vectors ai , i ∈ Wk , are linearly independent, which corresponds to full column
rank of Ak = [ai ]i∈Wk .

If no constraints are active, aTi x0 > bi for i ∈ I, then the corresponding working
set is empty, W0 = ∅. The objective function (4.4) is convex, and therefore if
minx∈R f (x) ∈ / Ω, then one or more constraints will be violated, when xk seeks
the minimum of the objective function. This explains why the working set is
never empty, once a constraint has become active, i.e. aTi xk = bi , i ∈ Wk 6= ∅,
k ∈ N.

Improving Direction

The feasible sequence {xk ∈ Ω} with decreasing value, f (xk+1 ) < f (xk ), is
generated by following the improving direction p ∈ Rn such that xk+1 = xk + p.
This leads us to
44 Active Set Methods

f (xk+1 ) = f (xk + p)
1
= (xk + p)T G(xk + p) + g T (xk + p)
2
1
= (xTk G + pT G)(xk + p) + g T xk + g T p
2
1
= (xTk Gxk + xTk Gp + pT Gxk + pT Gp) + g T xk + g T p
2
1 1
= xTk Gxk + g T xk + (xTk G + g T )p + pT Gp
2 2
T 1 T
= f (xk ) + (Gxk + g) p + p Gp
2
= f (xk ) + φ(p). (4.8)

and in order to satisfy f (xk+1 ) < f (xk ), the improving direction p must be
computed so that φ(p) < 0 and x̄ = (xk + p) ∈ Ω. Instead of computing
the new optimal point x̄ directly, computational savings are achieved by only
computing p. When x̄ = xk + p, the constraint aTi x̄ = bi becomes

aTi x̄ = aTi (xk + p) = aTi xk + aTi p = bi + aTi p, (4.9)

so

aTi p = 0 (4.10)

and the objective function becomes

f (x̄) = f (xk + p) = f (xk ) + φ(p). (4.11)

So for the subspaces Mk = {x ∈ Rn : aTi x = bi , i ∈ Wk } and Sk = {p ∈ Rn :

aTi p = 0, i ∈ Wk }, we get

min f (x̄) = min f (xk + p) = f (xk ) + min φ(p) (4.12)

x̄∈Mk p∈Sk p∈Sk

and hereby we have the following relations

4.1 Primal Active Set Method 45

x̄∗ = xk + p∗ (4.13)
f (x̄∗ ) = f (xk ) + φ(p∗ ). (4.14)

For these relations x̄∗ and f (x̄∗ ) respectively are the optimal solution and the
optimal value of

1 T
min f (x̄) = x̄ Gx̄ + g T x̄ (4.15a)
x̄∈Rn 2
s.t. aTi x̄ = bi i ∈ Wk (4.15b)

and p∗ and φ(p∗ ) respectively are the optimal solution and the optimal value of

1 T
min φ(p) = p Gp + (Gxk + g)T p (4.16a)
p∈Rn 2
s.t. aTi p = 0 i ∈ Wk . (4.16b)

The right hand side of the constraints in (4.16) is zero,which is why it is easier
to find the improving direction p than to solve (4.15). This means that the
improving direction is found by solving (4.16).

Step Length

If we take the full step p, we cannot be sure, that xk+1 = xk + p is feasible.

In this section we will therefore find the step length α ∈ R, which ensures
feasibility.

If the optimal solution is p∗ = 0, then φ(p∗ ) = 0. And because (4.4) is strictly

convex, then φ(p∗ ) < φ(0) = 0, if p∗ 6= 0, so

φ(p∗ ) = 0, p∗ = 0 (4.17)
φ(p∗ ) < 0, p∗ ∈ {p ∈ Rn : aTi p = 0, i ∈ Wk }\{0}. (4.18)
46 Active Set Methods

The relation between f (xk + αp∗ ) and φ(αp∗ ) is

f (xk+1 ) = f (xk + αp)

1
= (xk + αp)T G(xk + αp) + g T (xk + αp)
2
1 T
= (xk G + αpT G)(xk + αp) + g T xk + g T αp
2
1 T
= (xk Gxk + xTk Gαp + αpT Gxk + αpT Gαp)
2
+ g T xk + g T αp
1 1
= xTk Gxk + g T xk + (xTk G + g T )αp + αpT Gαp
2 2
T 1 T
= f (xk ) + (Gxk + g) αp + αp Gαp
2
= f (xk ) + φ(αp). (4.19)

For p∗ 6= 0 we have

f (xk + αp) = f (xk + αxk − αxk + αp)

= f ((1 − α)xk + α(xk + p))
≤ (1 − α)f (xk ) + αf (xk + p)
< (1 − α)f (xk ) + αf (xk )
= f (xk ) + αf (xk ) − αf (xk )
= f (xk ) (4.20)

and because of the convexity of the objective function f , this is a fact for all
α ∈]0; 1].

We now know, that if an α ∈]0; 1] exists, then f (xk + αp∗ ) < f (xk ). On this
basis we want to find a point on the line segment p∗ = xk+1 − xk , whereby
the largest possible reduction of the objective function is achieved, and at the
same time the constraints not in the current working set, i.e. i ∈ I\Wk , remain
satisfied. In other words, looking from point xk in the improving direction p∗ ,
we would like to find an α, so that the point xk + αp∗ remains feasible. In this
way the greatest reduction of the objective function is obtained by choosing the
largest possible α without leaving the feasible region.
4.1 Primal Active Set Method 47

As we want to retain feasibility, we only need to consider the potentially violated

constraints. This means the constraints not in the current working set satisfy

aTi (xk + αp∗ ) ≥ bi , i ∈ I\Wk . (4.21)

Since xk ∈ Ω, we have

αaTi p∗ ≥ bi − aTi xk ≤ 0, i ∈ I\Wk , (4.22)

and whenever aTi p∗ ≥ 0, this relation is satisfied for all α ≥ 0. As bi −aTi xk ≤ 0,

the relation can still be satisfied for aTi p∗ < 0, if we consider an upper bound
0 ≤ α ≤ ᾱi , where

bi − aTi xk
ᾱi = ≥ 0, aTi p∗ < 0, i ∈ I\Wk . (4.23)
aTi p∗

Whenever aTi xk = bi , and aTi p∗ < 0 for i ∈ I\Wk , we have ᾱi = 0. So

x̄ = xk + αp∗ will remain feasible, x̄ ∈ Ω, whenever 0 ≤ α ≤ mini∈I\Wk ᾱi .
In other words, the upper bound of α will be chosen in a way, that the nearest
constraint not in the current working set will become active.

From the Lagrangian function of (4.16), we know by definition, that p∗ satisfies

Gp∗ + (Gxk + g) − Aµ∗ = 0 (4.24a)

T ∗
A p = 0, (4.24b)

and by transposing and multiplying with p∗ we get

(Gxk + g)T p∗ = (Aµ∗ − Gp∗ )T p∗

∗
= µ∗T AT p −p∗T Gp∗
| {z }
=0
∗T
= −p Gp∗ . (4.25)
48 Active Set Methods

From (4.19) and (4.25) we define the line search function h(α) as

h(α) = f (xk + αp)

1
= f (xk ) + α(Gxk + g)T p + α2 pT Gp
2
1
= f (xk ) − αp∗T Gp∗ + α2 p∗T Gp∗
2
1
= p∗T Gp∗ α2 − p∗T Gp∗ α + f (xk ). (4.26)
2

If p∗ 6= 0 is the solution of (4.16), we have p∗T Gp∗ > 0, as G is positive

definite. So the line search function is a parabola with upward legs. The first
order derivative is

dh
(α) = p∗T Gp∗ α − p∗T Gp∗ (4.27a)
dα
= (α − 1)p∗T Gp∗ , (4.27b)

dh
which tells us, that the line search function has its minimum at dα (1) = 0.
Therefore the largest possible reduction in the line search function (4.26) is
achieved by selecting α ∈ [0; 1] as large as possible. So the optimal solution of

1 ∗T
min h(α) = p Gp∗ α2 − p∗T Gp∗ α + f (xk ) (4.28a)
α∈R 2
s.t. aTi (xk + αp∗ ) ≥ bi i∈I (4.28b)

is

 
∗
α = min 1, min ᾱi
i∈I\Wk :aT
i p <0
∗

 
bi − aTi xk
= min 1, min ≥ 0. (4.29)
i∈I\Wk :aT
i p <0
∗ aTi p∗

The largest possible reduction in the objective function along the improving
direction p∗ is obtained by the new point xk+1 = xk + α∗ p∗ .
4.1 Primal Active Set Method 49

4.1.3 Appending and Removing a Constraint

The largest possible reduction of the objective function in the affine space Mk =
{x ∈ Rn : aTi x = bi , i ∈ Wk } is obtained at point xk+1 = xk + p∗ , i.e. by
selecting α∗ = 1 and Wk+1 = Wk . This point satisfies f (xk+1 ) < f (xk ), and
since Wk+1 = Wk , this point will also be the optimal solution in the affine space
Mk+1 = Mk , thus a new iterate will give p∗ = 0. So, in order to minimize the
objective function further, we must update the working set for each iteration.
This is done either by appending or removing a constraint from the current
working set Wk .

Appending a Constraint

If the point xk+1 = xk + p∗ ∈ / Ω = {x ∈ Rn : aTi x ≥ bi , i ∈ I}, then the

point is not feasible with respect to one or more constraints not in the current
working set Wk . Therefore, by choosing the point xk+1 = xk + α∗ p∗ ∈ Mk ∩ Ω,
where α∗ ∈ [0; 1[, feasibility is sustained and the largest possible reduction of
the objective function is achieved. In other words, we have a blocking constraint
with index j ∈ I\Wk , such that aTj (xk + α∗ p∗ ) = bj . So, by appending con-
straint j to the current working set, we get a new working set Wk+1 = Wk ∪ {j}
corresponding to xk+1 , which is then a feasible point by construction.

The set of blocking constraints is defined as

bi − aTi xk
J = arg min (4.30)
i∈I\Wk :aT
i p <0
∗ aTi p∗

The blocking constraint to be appended, is the most violated constraint. In

other words, it is the violated constraint, found closest to the current point xk .
As mentioned, the working set is updated as Wk+1 = Wk ∪ {j}, which means,
that we append the vector aj , where j ∈ J , to the current working set. The
constraints in the current working set, i.e. the vectors ai for which i ∈ Wk ,
satisfies

aTi p∗ = 0, i ∈ Wk . (4.31)

If vector aj , where j ∈ J , is linearly dependent of the constraints in the current

50 Active Set Methods

working set, i.e. aj ∈ span{ai }i∈Wk , then we have

X
∃γi ∈ R : aj = γi ai , (4.32)
i∈Wk

hence aj must satisfy

X
aTj p∗ = γi (aTi p∗ ) = 0, j ∈ I\Wk . (4.33)
| {z }
i∈Wk
=0

But since we choose j ∈ I\Wk , such that aTi p∗ < 0, we have aj ∈

/ span{ai }i∈Wk .
So we are guaranteed, that the blocking constraint j is linearly independent of
the constraints in the current working set, i.e. (A aj ) maintains full column
rank.

Removing a Constraint

We now have to decide whether xk is a global minimizer of the inequality con-

strained QP (4.3).

From the optimality conditions

X
Gx∗ + g − ai µ∗i = 0 (4.34a)
i∈I

aTi x∗ ≥ bi i∈I (4.34b)

µ∗i ≥ 0 i∈I (4.34c)
µ∗i (aTi x∗ − bi ) = 0 i∈I (4.34d)
4.1 Primal Active Set Method 51

it is seen that xk is the global minimizer x∗ if and only if the pair xk , µ satisfies

X X X
Gx + g − ai µi = Gx + g − ai µi − ai µi = 0 (4.35a)
|{z}
i∈I i∈Wk i∈I\Wk =0
| {z }
=0
aTi xk ≥ bi i∈I (4.35b)
µi ≥ 0 i ∈ I. (4.35c)

We must remark, that

µi (aTi xk − bi ) = 0, i ∈ Wk (4.36a)
| {z }
=0
µi (aTi xk − bi ) = 0, i ∈ I\Wk (4.36b)
|{z}
=0

from which we have

µi (aTi xk − bi ) = 0, i ∈ I. (4.37)

So we see, that xk is the unique global minimizer of (4.3), if the computed

Lagrange multipliers µi for i ∈ Wk are non-negative. The remaining Lagrangian
multipliers for i ∈ I\Wk are then selected according to the optimality conditions
(4.7), so

x∗ = xk (4.38)
(
∗ µi , i ∈ Wk
µi = (4.39)
0, i ∈ I\Wk .

But if there exists an index j ∈ Wk such that µj < 0, then the point xk cannot
be the global minimizer of (4.3). So we have to relax, in other words leave, the
constraint aTj xk = bj and move in a direction p such that aTj (xk + αp) > bj .
From sensitivity theory in Nocedal and Wright [14] we know, that a decrease in
function value f is obtained by choosing any constraint for which the Lagrange
52 Active Set Methods

multiplier is negative. The largest rate of decrease is obtained by selecting

j ∈ Wk corresponding to the most negative Lagrange multiplier.

So if an index j ∈ Wk exists, where µj < 0, we will find an improving direction

p∗ , which is a solution to

1 T
min φ(p) = p Gp + (Gxk + g)T p (4.40a)
p∈Rn 2
s.t. aTi p = 0 i ∈ Wk \{j}. (4.40b)

As p∗ is the global minimizer of (4.40), there exists multipliers µ∗ so that

X
Gp∗ + Gxk + g − ai µ∗i = 0, (4.41)
i∈Wk \{j}

and if we let xk and µ̂ satisfy

X
Gxk + g − ai µ̂i = 0 (4.42a)
i∈Wk

aTi xk = bi i ∈ Wk , (4.42b)

and we subtract (4.42a) from (4.41) we get

X
Gp∗ − ai (µ∗i − µ̂i ) + aj µ̂j = 0, (4.43)
i∈Wk \{j}

which is equivalent to

X µ∗i − µ̂i Gp∗

aj = ai − . (4.44)
µ̂j µ̂j
i∈Wk \{j}

Since ai is linearly independent for i ∈ Wk and thereby also for i ∈ Wk \{j},

4.1 Primal Active Set Method 53

then aj cannot be a linear combination of ai , which means

X µ∗i − µ̂i
aj 6= ai (4.45)
µ̂j
i∈Wk \{j}

implying that p∗ 6= 0. Now we will shortly summarize, what have been stated in
this section so far. If the optimal solution has not been found at iteration k some
negative Lagrange multipliers exist. The constraint j ∈ Wk , which correspond
to the most negative Lagrange multiplier µj is removed from the working set.
The new improving direction is computed by solving (4.40) and it is guaranteed
to be non-zero p∗ 6= 0. This statement is important in the following derivations.

By taking a new step after removing constraint j, we must now guarantee, that
the relaxed constraint is not violated again, in other words that the remaining
constraints in the current working set are still satisfied and that we actually get
a decrease in function value f .

Taking the dot-product of aj and p∗ , by means of multiplying (4.44) with p∗T ,

we get

X µ∗i − µ̂i ∗T p∗T Gp∗

p∗T aj = p ai − (4.46)
µ̂j µ̂j
i∈Wk \{j}

which by transposing becomes

X µ∗i − µ̂i T ∗ p∗T Gp∗ p∗T Gp∗

aTj p∗ = ai p − =− . (4.47)
µ̂j | {z } µ̂j µ̂j
i∈Wk \{j} =0

Since p∗T Gp∗ > 0, p∗ 6= 0, and µ̂j < 0, it follows that

aTj p∗ > 0. (4.48)

Bearing in mind that xk ∈ Mk = {x ∈ Rn : aTi xk = bi , i ∈ Wk } , we see that

aTj (xk + αp∗ ) = aTj xk +α aTj p∗ > bj , ∀α ∈]0, 1] (4.49)

| {z } | {z }
=bj >0
54 Active Set Methods

and

aTi (xk + αp∗ ) = aTi xk +α aTi p∗ = bi , ∀α ∈]0, 1], i ∈ Wk \{j}. (4.50)

| {z } | {z }
=bi =0

As expected, we see that the relaxed constraint j and the constraints in the new
active set are still satisfied.

As (4.40) is strictly convex, we know, that φ(p∗ ) < 0 for the improving direction
p∗ , and the feasible non-optimal solution p = 0 has the value φ(0) = 0. If we
also keep in mind that

f (xk + αp∗ ) = f (xk ) + φ(αp∗ ) (4.51)

and when α = 1, then we get

f (xk + p∗ ) = f (xk ) + φ(p∗ ) < f (xk ). (4.52)

From this relation, the convexity of f and because α ∈]0; 1], we know that

f (xk + αp∗ ) = f (xk + αxk − αxk + αp∗ )

= f ((1 − α)xk + α(xk + p∗ ))
≤ (1 − α)f (xk ) + αf (xk + p∗ )
< (1 − α)f (xk ) + αf (xk )
= f (xk ) + αf (xk ) − αf (xk )
= f (xk ). (4.53)

So in fact we actually get a decrease in function value f , by taking the new step
having relaxed constraint j.

In this section we have found that if µj < 0, then the current point xk cannot
be a global minimizer. So to proceed we have to remove constraint j from the
current working set, Wk+1 = Wk \{j}, and update the current point by taking
a zero step, xk+1 = xk .
4.1 Primal Active Set Method 55

A constraint removed from the working set cannot be appended to the working
set in the iteration immediately after taking the zero step. This is because a
blocking constraint is characterized by aTj p∗ < 0, while from (4.48) we know,
that aTj p∗ > 0. Still, the possibility of cycling can be a problem for the primal
active set method, for example in cases like

−j −l +j +l −j
. . . −→ {i, j, l} −−→ {i, l} −→ {i} −−→ {i, j} −→ {i, j, l} −−→ {i, l} −→ . . .
(4.54)

This and similar cases are not considered, so if any cycling occurs, it is stopped
by setting a maximum number of iterations for the method.

The procedure of the primal active set method is stated in algorithm 4.1.1.
56 Active Set Methods

Algorithm 4.1.1: Primal Active Set Algorithm for Convex Inequality Con-
strained QP’s.
Input: Feasible point x0 , W = A0 = {i : aTi x0 = bi }.
while NOT STOP do /* find improving direction p∗ */
∗
Find the improving direction p by solving the equality constrained QP:
1
min φ(p) = pT Gp + (Gx + g)T p
p∈Rn 2
s.t. aTi p = 0, i∈W

if ||p∗ || = 0 then /* compute Lagrange multipliers µi */

Compute the Lagrange multipliers µi , i ∈ W by solving:
X
ai µi = Gx + g
i∈W

µi ← 0, i ∈ I\W
if µi ≥ 0 ∀i ∈ W then
STOP, the optimal solution x∗ has been found!
else /* remove constraint j */
x←x
W ← W\{j}, j ∈ W : µj < 0
else /* compute step length α */
 
bi − aTi x
α = min 1, min
i∈I\W:aT
i p <0
∗ aTi p∗
bi − aTi x
J = arg min
i∈I\W:aT
i p <0
∗ aTi p∗

if α < 1 then /* append constraint j */

x ← x + αp∗
W ← W ∪ {j}, j ∈ J
else
x ← x + p∗
W ←W
4.2 Primal active set method by example 57

4.2 Primal active set method by example

We will now demonstrate how the primal active set method finds the optimum
in the following example:

   
1 1 0 0
min f (x) = xT Gx + g T x, G = , g=
x∈Rn 2 0 1 0
s.t. c1 = −x1 + x2 − 1 ≥ 0
1
c2 = − x1 − x2 + 2 ≥ 0
2
c3 = −x2 + 2.5 ≥ 0
c4 = −3x1 + x2 + 3 ≥ 0.

At every iteration k the path (x1 . . . xk ) is plotted together with the constraints,
where active constraints are indicated in red. The 4 constraints and their column
index in A are labeled on the constraint in the plot. The feasible area is in the
top right corner where the plot is lightest. The start position is chosen to be
x = [4.0, 4.0]T which is feasible and the active set is empty W = ∅. For every
iteration we have plotted the situation when we enter the while-loop at x, see
algorithm 4.1.1 .

Iteration 1
The situation is illustrated in figure 4.1. On entering the while-loop the work-
ing set is empty and therefore the improving direction is found to be p =
[−4.0, −4.0]T . As figure 4.1 suggests, the first constraint to be violated taking
this step is c3 which is at step length α = 0.375. The step x̄ = x + αp is taken
and the constraint c3 is appended to the working set.
58 Active Set Methods

Figure 4.1: Iteration 1, W = ∅, x = [4.0, 4.0]T .

Iteration 2
The situation is illustrated in figure 4.2. Now the working set is W = [3] which
means that the new improving direction p is found by minimizing f (x) subject
to c3 (x + p) = 0. The improving direction is found to be p = [−2.5, 0.0]T and
µ = [2.5]. The first constraint to be violated in this direction is c4 which is at
step length α = 0.267. The step x̄ = x + αp is taken and the constraint c4 is
appended to the working set.

Figure 4.2: Iteration 2, W = [3], x = [2.5, 2.5]T .

4.2 Primal active set method by example 59

Iteration 3
The situation is illustrated in figure 4.3. Here the working set is W = [3, 4]
and the new improving direction is found to be p = [0, 0]T and µ = [3.1, 0.6]T .
Because p = 0 and no negative Lagrange Multipliers exist, position x is optimal.
Therefore the method terminates with x∗ = [1.8, 2.5]T .

Figure 4.3: Iteration 3, W = [3, 4], x∗ = [1.8, 2.5]T .

An interactive demo application QP demo.m is found in appendix D.5.

60 Active Set Methods

4.3 Dual active set method

In the foregoing, we have described the primal active set method which solves
an inequality constrained convex QP, by keeping track of a working set W. In
this section we will examine the dual active set method, which requires the QP
to be strictly convex. The dual active set method uses the dual set of W, which
we will call WD . The method benefits from always having an easily calculated
feasible starting point and the method does not have the possibility of cycling.
The theory is based on earlier works by Goldfarb and Idnani [10], Schmid and
Biegler [11] and Schittkowski [12].

4.3.1 Survey

The inequality constrained strictly convex QP that we want to solve is as follows

1
min f (x) = xT Gx + g T x (4.55a)
x∈Rn 2
s.t. ci (x) = aTi x − bi ≥ 0, i ∈ I. (4.55b)

The corresponding Lagrangian function is

1 T X
L(x, µ) = x Gx + g T x − µi (aTi x − bi ). (4.56)
2
i∈I

The dual program of (4.55) is

1 X
max L(x, µ) = xT Gx + g T x − µi (aTi x − bi ) (4.57a)
x∈R ,µ∈R
n m 2
i∈I
X
s.t. Gx + g − ai µi = 0 (4.57b)
i∈I
µi ≥ 0 i ∈ I. (4.57c)

The necessary and sufficient conditions for optimality of the dual program is
4.3 Dual active set method 61

X
Gx∗ + g − ai µ∗i = 0 (4.58a)
i∈I

ci (x∗ ) = T ∗
ai x − bi = 0 i ∈ A(x∗ ) (4.58b)
∗
ci (x ) = aTi x∗ − bi > 0 i ∈ I\A(x ) ∗
(4.58c)
∗
µi ≥ 0 i ∈ A(x ) (4.58d)
µi = 0 i ∈ I\A(x∗ ). (4.58e)

These conditions are exactly the same as the optimality conditions of the primal
program (4.55), and this corresponds to the fact that the optimal value L(x∗ , µ∗ )
of the dual program is equivalent to the optimal value f (x∗ ) of the primal
program. This is why the solution of the primal program can be found by
solving the dual program (4.57).

The method maintains dual feasibility at any iteration {xk , µk } by satisfying

(4.57b) and (4.57c). This is done by keeping track of a working set W. The
constraints in the working set satisfy

X
Gxk + g − ai µki = 0 (4.59a)
i∈W

ci (xk ) = T k
ai x − bi =0 i∈W (4.59b)
µki ≥0 i ∈ W. (4.59c)

The constraints in the complementary set WD = I\W, i.e. the active set of the
dual program, satisfy

X
Gxk + g − ai µki = 0 (4.60a)
i∈WD

µki = 0 i ∈ WD , (4.60b)

and from (4.58), (4.59) and (4.60) it is clear that an optimum has been found
xk = x∗ if

ci (xk ) = aTi xk − bi ≥ 0 i ∈ WD . (4.61)

62 Active Set Methods

If this is not the case some violated constraint r ∈ WD exists, i.e. cr (xk ) < 0.
The following relationship explains why {xk , µk } cannot be an optimum in this
case

∂L k k
(x , µ ) = −cr (xk ) > 0. (4.62)
∂µr

This means that (4.57a) can be increased by increasing the Lagrangian multiplier
µr . In fact, this explains the key idea of the dual active set method. The idea is
to choose a constraint cr from the dual active working set WD which is violated
cr (xk ) < 0 and make it satisfied cr (xk ) ≥ 0 by increasing the Lagrangian
multiplier µr . This procedure continues iteratively until no constraints from
WD are violated. At this point the optimum has been found and the method
terminates.

4.3.2 Improving Direction and Step Length

If optimality has not been found at iteration k it indicates that a constraint cr

is violated, which means that cr (xk ) < 0. In this section we will investigate how
to find both an improving direction and a step length which satisfy the violated
constraint cr .

Improving Direction

The Lagrangian multiplier µr of the violated constraint cr from WD should be

changed from zero to some value that will optimize (4.57a) and satisfy (4.57b)
and (4.57c). After this operation the new position is

x̄ = x + s (4.63a)
µ̄i = µi + ui i∈W (4.63b)
µ̄r = µr + t (4.63c)
µ̄i = µi = 0 i ∈ WD \r. (4.63d)

From (4.57b) and (4.59b) we know that x̄ and µ̄ should satisfy

4.3 Dual active set method 63

X
Gx̄ + g − ai µ̄i = 0 (4.64a)
i∈I

ci (x̄) = aTi x̄ − bi = 0 i ∈ W. (4.64b)

As µi 6= 0 for i ∈ W, µ̄r 6= 0 and r yet not in W, this can be written as

      
G −A x̄ g ar
+ − µ̄r = 0, (4.65)
−AT 0 µ̄ b 0

where A = [ai ]i∈W has full column rank, G is symmetric and positive definite,
µ̄ = [µ̄i ]Ti∈W , ar is the constraint from WD we are looking at and µr is the
corresponding Lagrangian multiplier. Using (4.63) this can be formulated as

      
G −A x g ar
+ − µr +
−AT 0 µ b 0
    
G −A s ar
− t = 0. (4.66)
−AT 0 u 0

From (4.64) we have

      
G −A x g ar
+ − µr = 0 (4.67)
−AT 0 µ b 0

and therefore (4.66) is simplified as follows

    
G −A s ar
− t = 0, (4.68)
−AT 0 u 0

which is equivalent to

        
G −A p ar s p
= , = t. (4.69)
−AT 0 v 0 u v

The new improving direction (p, v)T is found by solving (4.69), using a solver
for equality constrained QP’s, e.g. the range space or the null space procedure.
64 Active Set Methods

Step Length

Having the improving direction we now would like to find the step length t
(4.69). This step length should be chosen in a way, that makes (4.57c) satisfied.
From (4.63), (4.66) and (4.69) we have the following statements about the new
step

x̄ = x + s = x + tp (4.70a)
µ̄i = µi + ui = µi + tvi i∈W (4.70b)
µ̄r = µr + t (4.70c)
µ̄i = µi = 0 i ∈ WD \r. (4.70d)

To make sure that µ̄r ≥ 0 (4.57c), we must require, that t ≥ 0. When vi ≥ 0

we have µ̄r ≥ 0 and (4.57c) is satisfied for any value of t ≥ 0. When vi < 0 we
must require t to be some positive value less than −µ vi , which makes µ̄i ≥ 0 as
i

µi ≥ 0 and vi < 0. This means that t should be chosen as

−µi
t ∈ [0, tmax ], tmax = min(∞, min ) ≥ 0. (4.71)
i:vi <0 vi

Now we know what values of t we can choose, in order to retain dual feasibility
when taking the new step. To find out what exact value of t in the interval
(4.71) we should choose to make the step optimal we need to examine what
happens to the primal objective function (4.55a), the dual objective function
(4.57a), and the constraint cr as we take the step.

The relation between cr (x) and cr (x̄)

Now we shall examine how cr (x̄) is related to cr (x). For this reason, we need
to state the following properties. From (4.69) we have

ar = Gp − Av (4.72a)
T
A p = 0. (4.72b)

Multiplying (4.72a) with p gives

aTr p = (Gp − Av)T p = pT Gp − v T AT p = pT Gp (4.73)

4.3 Dual active set method 65

and because G is positive definite, we have

aTr p = pT Gp ≥ 0 (4.74a)
aTr p T
= p Gp = 0 ⇔ p=0 (4.74b)
aTr p T
= p Gp > 0 ⇔ p 6= 0. (4.74c)

As x̄ = x + tp we get

cr (x̄) = cr (x + tp) = aTr (x + tp) − br = aTr x − br + taTr p (4.75)

and because cr (x) = aTr x − br this is equivalent to

cr (x̄) = cr (x) + taTr p. (4.76)

From (4.71) and (4.74a) we know that taTr p ≥ 0 and therefore

cr (x̄) ≥ cr (x). (4.77)

This means that the constraint cr is increasing (if t > 0) as we move from x to
x̄ and this is exactly what we want as it is negative and violated at x.

The relation between f (x) and f (x̄)

In addition to the foregoing we will now investigate what happens to the primal
objective function (4.55a) as we move from x to x̄. Inserting x̄ = x + tp in
(4.55a) gives

1
f (x̄) = f (x + tp) = (x + tp)T G(x + tp) + g T (x + tp), (4.78)
2

which may be reformulated as

1 T 1
f (x̄) = x Gx + g T x + t2 pT Gp + t(Gx + g)T p, (4.79)
2 2
66 Active Set Methods

and using (4.55a) this leads to

1
f (x̄) = f (x) + t2 pT Gp + t(Gx + g)T p. (4.80)
2

From (4.67) we have the relation

Gx − Aµ + g − ar µr = 0, (4.81)

which is equivalent to

Gx + g = Aµ + ar µr (4.82)

and when multiplied with p this gives

(Gx + g)T p = (Aµ + ar µr )T p = µT AT p + µr aTr p. (4.83)

Furthermore using the fact that AT p = 0, this is equivalent to

(Gx + g)T p = µr aTr p. (4.84)

Inserting this in (4.80) gives

1
f (x̄) = f (x) + t2 pT Gp + tµr aTr p. (4.85)
2

Using pT Gp = aTr p from (4.74a) we get

1 1
f (x̄) = f (x) + t2 aTr p + tµr aTr p = f (x) + t(µr + t)aTr p. (4.86)
2 2

As t ≥ 0, aTr p ≥ 0 and µr ≥ 0 the primal objective function does not decrease

when we move from x to x̄.
4.3 Dual active set method 67

The relation between L(x, µ) and L(x̄, µ̄)

We will now investigate what happens to the Lagrangian function (4.57a) as we

move from (x, µ) to (x̄, µ̄). After taking a new step we have

X
L(x̄, µ̄) = f (x̄) − µi ci (x̄), (4.87)
i∈I

and because µi = 0 for i ∈ WD and ci (x̄) = 0 for i ∈ W this is equivalent to

L(x̄, µ̄) = f (x̄) − µ̄r cr (x̄). (4.88)

By replacing f (x̄) with (4.86), µ̄r with µr + t and cr (x̄) with (4.76), we then
have

1
L(x̄, µ̄) = f (x) + t(µr + t)aTr p − (µr + t)(cr (x) + taTr p), (4.89)
2

which we reformulate as

1
L(x̄, µ̄) = f (x)+µr taTr p+ t2 aTr p−µr cr (x)−µr taTr p−tcr (x)−t2 aTr p (4.90)
2

and finally this gives

1
L(x̄, µ̄) = f (x) − µr cr (x) − t2 aTr p − tcr (x). (4.91)
2

The Lagrangian L(x, µ) before taking the new step is

X
L(x, µ) = f (x) − µi ci (x) (4.92)
i∈I

and as in the case above we have the precondition µi = 0 for i ∈ WD and

68 Active Set Methods

ci (x) = 0 for i ∈ W and therefore (4.92) is equivalent to

L(x, µ) = f (x) − µr cr (x), (4.93)

and inserting this in (4.91) gives us

1
L(x̄, µ̄) = L(x, µ) − t2 aTr p − tcr (x). (4.94)
2

Now we want to know what values of t make L(x̄, µ̄) ≥ L(x, µ), i.e what values
of t that satisfy

1
L(x̄, µ̄) − L(x, µ) = − t2 aTr p − tcr (x) ≥ 0. (4.95)
2

This inequality is satisfied when

−cr (x)
t ∈ [0, 2 ]. (4.96)
aTr p

When t is in this interval, the Lagrangian function increases as we move from

(x, µ) to (x̄, µ̄). To find the value of t that gives the greatest increment we must
differentiate (4.94) with respect to t

dL
= −taTr p − cr (x). (4.97)
dt

The greatest increment is at t∗ where

−cr (x)
−t∗ aTr p − cr (x) = 0 ⇔ t∗ = . (4.98)
aTr p

At this point we would like to stop up and present a short summary of what
has been revealed throughout the latest sections. If optimality has not been
4.3 Dual active set method 69

found at iteration k, some violated constraint cr (xk ) ≤ 0 must exist. The new
improving direction is found by solving the equality constrained QP

    
G −A p ar
= (4.99)
−AT 0 v 0

where A = [ai ]i∈W has full column rank, G is symmetric and positive definite
and ar is the violated constraint. The optimal step length t, which ensures
feasibility is found from statements (4.71) and (4.98)

−µi −cr (x)

t = min( min , ). (4.100)
i:vi <0 vi aTr p

Both the dual objective function (4.57) and the violated constraint increase as
we take the step.

4.3.3 Linear Dependency

The KKT system (4.69) can only be solved if G is positive definite, and A
has full column rank. If the constraints in A = [ai ]i∈W and ar are linearly
dependent, it is not possible to add constraint r to the working set W, as
A = [ai ]i∈W∪r in this case would not have full column rank. This problem is
solved by removing constraint j from W, which makes the constraints in the
new working set W̄ = W\{j} and ar linearly independent. This particular case
will be investigated in the following. The linear dependency of A = [ai ]i∈W and
ar can be written as

m
X
ar = γi ai = Aγ. (4.101)
i=1

When multiplied with p we get

aTr p = γ T AT p, (4.102)
70 Active Set Methods

and as AT p = 0 (4.72b) we then have

aTr p = 0 ⇔ ar ∈ spanA = [ai ]i∈W . (4.103)

Now we will investigate what to do when

cr (x) < 0 ∧ ar ∈ spanA = [ai ]i∈W . (4.104)

When ar and the constraints in the working set are linearly dependent and
(4.69) is solved

    
G −A p ar
= (4.105)
−AT 0 v 0

we know from (4.74b) and (4.103) that p = 0. If v contains any negative values,
t can be calculated using (4.71)

−µj −µj
t = min ≥ 0, j = arg min . (4.106)
j:vj <0 vj j:vj <0 vj

When we move from x to x̄ we then have

µ̄j = µj + tvj = 0 (4.107)

and this is why we need to remove constraint cj from W and we call the new set
W̄ = W\{j}. Now we will see that ar is linearly independent of the vectors ai
for i ∈ W̄. This proof is done by contradiction. Lets assume that ar is linearly
dependent of the vectors ai for i ∈ W̄, i.e. ar ∈ spanA = [ai ]i∈W̄ .

As ar ∈ spanA = [ai ]i∈W and hence p = 0 we can therefore write

X
ar = Gp − Av = −Av = A(−v) = ai (−vi ). (4.108)
i∈W
4.3 Dual active set method 71

At the same time because W = W̄ ∪ {j}, we have

X
ar = ai (−vi ) + aj (−vj ) (4.109)
i∈W̄

and isolation of aj gives

1 1 X
aj = ar + ai vi . (4.110)
−vj −vj
i∈W̄

Since we assumed ar ∈ spanA = [ai ]i∈W̄ , using (4.101) ar can be formulated

as

X
ar = ai γi (4.111)
i∈W̄

and inserting this equation in (4.110) gives us

1 X 1 X
aj = ai γi + ai vi , (4.112)
−vj −vj
i∈W̄ i∈W̄

which is equivalent to

X γi + vi
aj = ai . (4.113)
−vj
i∈W̄

As we have

X γi + vi
aj = βi ai , βi = (4.114)
−vj
i∈W̄

clearly aj is linearly dependent on the vectors ai for i ∈ W̄ and this is a

contradiction to the fact that A = [ai ]i∈W has full column rank, i.e. the vectors
ai for i ∈ W̄ ∪ j are linearly independent. This means that the assumption
72 Active Set Methods

ar ∈ spanA = [ai ]i∈W̄ cannot be true, and therefore we must conclude that
ar ∈
/ spanA = [ai ]i∈W̄ .

Furthermore from (4.94) we know that

1
L(x̄, µ̄) = L(x, µ) − t2 aTr p − tcr (x) (4.115)
2

and because of linear dependency aTr p = 0, this is equivalent to

L(x̄, µ̄) = L(x, µ) − tcr (x) (4.116)

and as tcr (x) ≤ 0 we know that L(x̄, µ̄) ≥ L(x, µ) when ar ∈ spanA = [ai ]i∈W .

Now we shall see what happens when no negative elements exist in v from
(4.105). From (4.71) we know that t can be chosen as any non negative value,
and therefore (4.116) becomes

lim L(x̄, µ̄) = ∞. (4.117)

t→∞

In this case the dual program is unbounded which means that the primal pro-
gram (4.55) is infeasible. Proof of this is to be found in Jørgensen [13]. This
means that no solution exists.

In short, what has been stated in this section, is that when (4.69) is solved and
aTr p = 0 we know that A = [ai ]i∈W and ar are linearly dependent and p = 0.
If there are no negative elements in v the problem is infeasible and no solution
exist. If some negative Lagrangian multipliers exist we should find constraint
cj from W where

−µj −µj
j = arg min , t = min ≥0 (4.118)
j:vj <0 vj j:vj <0 vj

and the following step is taken

µ̄i = µi + tvi , i ∈ W (4.119a)

µ̄r = µr + t. (4.119b)
4.3 Dual active set method 73

As p = 0, we know that x̄ = x and therefore this step is not mentioned in

(4.119). Again, when µ̄j = 0 it means that constraint cj belongs to the dual
active set WD and is therefore removed from W. The constraints in the new
working set W̄ = W\{j} and ar are linearly independent, and as a result a new
improving direction and step length may be calculated.

4.3.4 Starting Guess

One of the forces of the dual active set method is that a feasible starting point
is easily calculated. Starting out with all constraints in the dual active set WD
and therefore W being empty

µi = 0, i ∈ WD = I, W =∅ (4.120)

and if we start in

x = −G−1 g (4.121)

(4.57b) and (4.57c) are satisfied

X
Gx + g − ai µi = Gx + g = 0 (4.122a)
i∈I
µi ≥ 0 i ∈ I. (4.122b)

The Lagrangian function is

1 T X
L(x, µ) = x Gx + g T x − µi (aTi x − bi )
2
i∈I
1
= xT Gx + g T x
2
= f (x), (4.123)

which means that the starting point is at the minimum of the objective function
of the primal program (4.55a) without taking notice of the constrains (4.55b).
74 Active Set Methods

Because the inverse Hessian matrix is used we must require the QP to be strictly
convex.

4.3.5 In summary

Now we will summarize what has been discussed in this section and show how
the dual active set method works. At iteration k we have (x, µ, r, W, WD ) where
cr (x) < 0. Using the null space or the range space procedure the new improving
direction is calculated by solving

    
G −A p ar
= , A = [ai ]i∈W . (4.124)
−AT 0 v 0

If A = [ai ]i∈W and ar are linearly dependent and no elements from v are
negative the problem is infeasible and the method is terminated. Using (4.103)
and (4.117) this is the case when

aTr p = 0 ∧ vi ≥ 0, i ∈ W. (4.125)

If on the other hand A = [ai ]i∈W and ar are linearly dependent and some
elements from v are negative, cj is removed from W, step length t is calculated
according to (4.106) and a new step is taken
−µj −µj
t = min ≥ 0, j = arg min (4.126a)
j:vj <0 vj j:vj <0 vj
µ̄i = µi + tvi i∈W (4.126b)
µ̄r = µr + t (4.126c)
W = W\{j}. (4.126d)

If A = [ai ]i∈W and ar are linearly independent and some elements from v are
negative, we calculate two step lengths t1 and t2 according to (4.71) and (4.98)
−µj −µj
t1 = min(∞, min ), j = arg min (4.127a)
j:vj <0 vj j:vj <0 vj

−cr (x)
t2 = , (4.127b)
aTr p

where t1 can be regarded as the step length in dual space because it assures
that (4.57c) is satisfied whenever 0 ≤ t ≤ t1 . Constraint (4.55b) is satisfied for
4.3 Dual active set method 75

cr when t ≥ t2 and therefore t2 can be regarded as the step length in primal

space. Therefore we will call t1 tD and t2 tP .

If tD < tP then tD is used as the step length and (4.57c) remain satisfied when
we take the step. After taking the step, cj is removed from W because µ̄j = 0

x̄ = x + tD p (4.128a)
µ̄i = µi + tD vi , i ∈ W (4.128b)
µ̄r = µr + tD (4.128c)
W = W\{j}. (4.128d)

If tP ≤ tD then tP is used as the step length and (4.55b) get satisfied for cr .
After taking the step we have that cr (x̄) = 0 and therefore r is appended to W

x̄ = x + tP p (4.129a)
µ̄i = µi + tP vi , i ∈ W (4.129b)
µ̄r = µr + tP (4.129c)
W = W ∪ {r}. (4.129d)

If A = [ai ]i∈W and ar are linearly independent and no elements from v are
negative we have found the optimum and the program is terminated.

The procedure of the dual active set method is stated in algorithm 4.3.1.
76 Active Set Methods

Algorithm 4.3.1: Dual Active Set Algorithm for Strictly Convex Inequality
Constrained QP’s. Note: WD = I\W.
Compute x0 = −G−1 g, set µi = 0, i ∈ WD and W = ∅.
while NOT STOP do
if ci (x) ≥ 0 ∀i ∈ WD then
STOP, the optimal solution x∗ has been found!
Select r ∈ WD : cr (x) < 0.
while cr (x) < 0 do /* find improving direction p */
Find the improving direction p by solving the equality constrained
QP:     
G −A p ar
= , A = [ai ]i∈W
−AT 0 v 0
if aTr p = 0 then
if vi ≥ 0 ∀i ∈ W then
STOP, the problem is infeasible!
else /* compute step length t, remove constraint j */
−µi −µi
t = min , J = arg min
i∈W:vi <0 vi i∈W:vi <0 vi
x←x
µi ← µi + tvi , i ∈ W
µr ← µr + t
W ← W\{j}, j ∈ J
else /* compute step length tD and tP */
 
−µi −µi
tD = min ∞, min , J = arg min
i∈W:vi <0 vi i∈W:vi <0 vi
−cr (x)
tP =
aTr p

if tP ≤ tD then /* append constraint r */

x ← x + tP p
µi ← µi + tP vi , i ∈ W
µr ← µr + tP
W ← W ∪ {r}
else /* remove constraint j */
x ← x + tD p
µi ← µi + tD vi , i ∈ W
µr ← µr + tD
W ← W\{j}, j ∈ J
4.3 Dual active set method 77

4.3.6 Termination

The dual active set method does not have the ability to cycle as it terminates
in a finite number of steps. This is one of the main forces of the method, and
therefore we will now investigate this property.

As the algorithm (4.3.1) suggests, the method mainly consists of two while-loops
which we call outer-loop and inner-loop. In the outer-loop we test if optimality
has been found. If this is not the case we choose some violated constraint r,
cr (x) < 0 and move to the inner-loop.

At every iteration of the inner-loop we calculate a new improving direction and a

corresponding step length: t = min(tD , tP ), where tD is the step length in dual
space and tP is the step length in primal space. The step length in primal space
r (x )
is always positive, tP > 0 as tP = −c T
aTr p , where cr (x) < 0 and ar p > 0. From
r (x )
(4.95) and (4.96) we know that L(x̄, µ̄) > L(x, µ) whenever 0 < t < −2c aTr p .
This means that the dual objective function L increases when a step in primal
space is taken. A step in primal space also means that we leave the inner-loop
as constraint cr is satisfied cr (x̄) = cr (x) + taTr p = 0.

A step in dual space is taken whenever tD < tP and in this case we have
cr (x̄) = cr (x) + tD aTr p < cr (x) + tP aTr p = 0. This means that we will never
leave the inner-loop after a step in dual space as cr (x̄) < 0. A constraint cj
is removed from the working set W when we take a step in dual space, which
means that |W| is the maximum number of steps in dual space that can be
taken in succession. After a sequence of 0 ≤ s ≤ |W| steps in dual space, a step
in primal space will cause us to leave the inner-loop. This step in primal space
guarantees that L is strictly larger when we leave the inner-loop than when we
entered it.

As the constraints in the working set W are linearly independent at any time,
the corresponding solution (x, µ) is unique. Also as L(xq+1 , µq+1 ) > L(xq , µq )
(where q defines the q ′ th iteration of the outer loop) we know that the combina-
tion of constraints in W is unique for any iteration q. And because the number
of different ways the working set can be chosen from I is finite and bounded by
2|I| , we know that the method will terminate in a finite number of iterations.
78 Active Set Methods

4.4 Dual active set method by example

In the following example we will demonstrate how the dual active set method
finds the optimum

   
1 1 0 0
min f (x) = xT Gx + g T x, G = , g=
x∈Rn 2 0 1 0
s.t. c1 = −x1 + x2 − 1 ≥ 0
1
c2 = − x1 − x2 + 2 ≥ 0
2
c3 = −x2 + 2.5 ≥ 0
c4 = −3x1 + x2 + 3 ≥ 0.

At every iteration k we plot the path (x1 . . . xk ) together with the constraints,
where active constraints are indicated with red. The 4 constraints and their
column-index in A are labeled on the constraint in the plots. The primal fea-
sible area is in the top right corner where the plot is lightest. We use the least
negative value of c(xk ) every time cr is chosen, even though any negative con-
straint could be used. For every iteration we have plotted the situation when
we enter the while loop.

Iteration 1
This situation is illustrated in figure 4.4. The starting point is at x = G−1 g =
(0, 0)T , µ = 0 and W = ∅. On entering the while loop we have c(x) =
[1.0, −2.0, −2.5, −3.0]T and therefore r = 2 is chosen because the second element
is least negative. The working set is empty and therefore the new improving
direction is found to p = [0.5, 1.0]T and u = []. As the step length in primal
space is tP = 1.6 and the step length in dual space is tD = ∞, tP is used, and
therefore r is appended to the working set. The step is taken as seen in figure
4.5.
4.4 Dual active set method by example 79

Figure 4.4: Iteration 1, W = ∅, x = [0, 0]T , µ = [0, 0, 0, 0]T .

Iteration 2
This situation is illustrated in figure 4.5. On entering the while loop we have
W = [2], and because cr (x) = 0 we should choose a new r. As c(x) =
[0.2, 0, −0.9, −2.2]T , r = 3 is chosen. The new improving direction is found
to be p = [−0.4, 0.2]T and u = [−0.8]. As tD = 2.0 and tP = 4.5 a step in dual
space is taken and c2 is removed from W.

Figure 4.5: Iteration 2, W = [2], x = [0.8, 1.6]T , µ = [0, 1.6, 0, 0]T .

Iteration 3
This situation is illustrated in figure 4.6. Because cr (x) 6= 0 we keep r = 3.
The working set is empty W = ∅. The improving direction is found to be
p = [0.0, 1.0]T and u = []. A step in primal space is taken as tD = ∞ and
80 Active Set Methods

tP = 0.5 and r is appended to W.

Figure 4.6: Iteration 3, W = [], x = [0, 2]T , µ = [0, 0, 2, 0]T .

Iteration 4
This situation is illustrated in figure 4.7. Now cr (x) = 0 and therefore a new r
should be chosen. As c(x) = [−1.5, 0.5, 0, −5.5]T , we choose r = 1. The working
set is W = [3], and the new improving direction is p = [1.0, 0.0]T and u = [1].
We use tP as tD = ∞ and tP = 1.5 and r is appended to W after taking the
step.

Figure 4.7: Iteration 4, W = [3], x = [0, 2.5]T , µ = [0, 0, 2.5, 0]T .

Iteration 5
This situation is illustrated in figure 4.8. As cr (x) = 0 we must choose a new
r and as c(x) = [0, 1.25, 0, −1.0]T , r = 4 is chosen. At this point W = [3, 1].
4.4 Dual active set method by example 81

The new improving direction is p = [0.0, 0.0]T and u = [−2, −3] and therefore
aTr p = 0. This means that ar is linearly dependent of the constraints in W and
therefore 1 is removed from W.

Figure 4.8: Iteration 5, W = [3, 1], x = [1.5, 2.5]T , µ = [1.5, 0, 4, 0]T .

Iteration 6
This situation is illustrated in figure 4.9. On entering the while loop we have
c3 in the working set W = [3]. And r remains 4 because cr (x) 6= 0. The new
improving direction is p = [3.0, 0.0]T and u = [1]. The step lengths are tD = ∞
and tP = 0.11 and therefore a step in primal space is taken and r is appended
to W.

Figure 4.9: Iteration 6, W = [3], x = [1.5, 2.5]T , µ = [0, 0, 3, 0.5]T .

82 Active Set Methods

Iteration 7
This situation is illustrated in figure 4.10. Now the working set is W = [3, 4]
and as cr (x) = 0, a new r must be chosen. But c(x) = [0.33, 1.42, 0, 0]T (no
negative elements) and therefore the global optimal solution has been found
and the algorithm is terminated. The optimal solution is x∗ = [1.83, 2.50]T and
µ∗ = [0, 0, 3.11, 0.61]T .

Figure 4.10: Iteration 7, W = [3, 4], x∗ = [1.8, 2.5]T , µ∗ = [0, 0, 3.11, 0.61]T .

An interactive demo application QP demo.m is found in appendix D.5.

Chapter 5

Test and Refinements

When solving an inequality constrained convex QP, we use either the primal
active set method or the dual active set method. In both methods we solve a
sequence of KKT systems, where each KKT system correspond to an equality
constrained QP. To solve the KKT system we use one of four methods: The
range space procedure, the null space procedure, or one of the two with fac-
torization update instead of complete factorizations. We will test these four
methods for computational speed to find out how they perform compared to
each other.
Usually the constraints in an inequality constrained QP are divided into bounded
variables and general constraints. This division can be used to further optimiza-
tion of the factorization updates, as we will discuss later in this chapter. As a
test case we will use the quadruple tank problem which is described in appendix
A.

5.1 Computational Cost of the Range and the

Null Space Procedures with Update

The active set methods solve a sequence of KKT systems by use of the range
space procedure, the null space procedure or one of the two with factorization
84 Test and Refinements

update. Now we will compare the performance of these methods by solving the
quadruple tank problem. By discretizing with N = 300 we define an inequality
constrained strictly convex QP with n = 1800 variables and |I| = 7200 con-
straints. We have chosen to use the dual active set method because it does
not need a precalculated starting point. Different parts of the process are il-
lustrated in figure 5.1. Figure 5.1(a) shows the computational time for solving
the KKT system for each iteration and figure 5.1(c) shows the number of active
constraints |W | for each iteration. The size of the active set grows rapidly in
the first third of the iterations after which this upward movement fades out a
little. This explains why the computational time for the null space procedure
decreases fast to begin with and then fades out, as it is proportional to the
size of the null space (n − m). Likewise, the computational time for the range
space procedure grows proportional to the dimension of the range space (m).
The null space procedure with factorization update is much faster than the null
space procedure with complete factorization even though some disturbance is
observed in the beginning. This disturbance is probably due to the fact that the
testruns are carried out on shared servers. The range space procedure improves
slightly whenever factorization update is used. When solving this particular
problem, it is clear from figures 5.1(a) and 5.1(c), that range space procedure
with factorization update should be used until approximately 800 constraints
800
are active, corresponding to 1800 n =≃ 0.45n, after which the null space proce-
dure with factorization update should be used. In theory the total number of
iterations should be exactly the same for all four methods, however they differ
a little due to numerical instability as seen in figure 5.1(c), where the curves
are not completely aligned. The number of active constraints at the optimal
solution is |W | = 1658.
5.1 Computational Cost of the Range and the Null Space Procedures with
Update 85

Test of Speed: N =300, n =1800 Test of Speed: N =300, n =1800

70 70
null space null space
null space update null space update
60 range space 60
range space update
range space
range space update
50 50

40 40
time [s]

time [s]
30 30

20 20

10 10

0 0
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 0 200 400 600 800 1000 1200 1400 1600 1800
iteration |W| (appending)

(a) Computational time for solving the (b) Computational time for solving the
KKT system plotted for each iteration. KKT system each time a constraint is ap-
pended to the active set W .

Number of elements in W at each iteration , max(|W|) =1658 Test of Speed: N =300, n =1800
1800 70
null space
null space update
1600
60 range space
range space update
1400 null space
null space update 50
1200 range space
range space update
40
time [s]

1000
|W|

800 30

600
20
400

10
200

0 0
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 0 200 400 600 800 1000 1200 1400 1600 1800
iteration |W| (removing)

(c) Number of active constraints plotted at (d) Computational time for solving the
each iteration. KKT system each time a constraint is re-
moved from the active set W .

Figure 5.1: The process of solving the quadruple tank problem with N=300,
(1800 variables and 7200 constraints). Around 8800 iterations are needed (de-
pending on the method) and the number of active constraints at the solution is
|W | = 1658.
86 Test and Refinements

5.2 Fixed and Free Variables

So far, we have only considered constraints defined like aTi x ≥ bi . Since some
of the constraints are likely to be bounds on variables xi ≥ bi , we divide all
constraints into bounds and general constraints. The structure of our QP solver
is then defined as follows

1 T
min x Gx + g T x (5.1a)
x∈Rn 2
s.t. li ≤ xi ≤ ui i ∈ Ib = 1, 2, ..., n (5.1b)
(bl )i ≤ aTi x ≤ (bu )i i ∈ Igc = 1, 2, ..., mgc (5.1c)

where Ib is the set of bounds and Igc is the set of general constraints. This
means that we have upper and lower limits on every bound and on every general
constraint, so that the total number of constraints is |I| = 2n + 2mgc . We call
the active constraint matrix C ∈ Rn×m and it contains both active bounds
and active general constraints. Whenever a bound is active we say that the
corresponding variable xi is fixed. By use of a permutation matrix P ∈ Rn×n
we organize x and C in the following manner

   
x̃ C̃
= P x, = PC (5.2)
x̂ Ĉ

where x̃ ∈ Rñ , x̂ ∈ Rn̂ , C̃ ∈ Rñ×m and Ĉ ∈ Rn̂×m , ñ is the number of free

variables and n̂ is the number of fixed variables (n̂ = n − ñ). Now we reorganize
the active constraint matrix P C

 
0 Ã
P C = P (B A) = (5.3)
I Â

where B ∈ Rn×n̂ contains the bounds and A ∈ Rn×(m−n̂) contains the general
constraints. So we have à ∈ Rñ×(m−n̂) and  ∈ Rn̂×(m−n̂) and I ∈ Rn̂×n̂ as
the identity matrix.

The QT factorization (which is used in the null space procedure) of (5.3) is

5.2 Fixed and Free Variables 87

defined as

   
Q̃ 0 0 T̃
Q= , T = (5.4)
0 I I Â

where Q ∈ Rn×n , Q̃ ∈ Rñ×ñ , I ∈ Rn̂×n̂ , T ∈ Rn×m and T̃ ∈ Rñ×(m−n̂) , Gill et

al. [9]. This is a modified QT factorization, as only T̃ in T is lower triangular.

The part of the QT factorization which corresponds to the free variables consists
of Q̃ and T̃ . From (5.4) it is clear that this is the only part that needs to be
updated whenever a constraint is appended to or removed from the active set.
The details of how these updates are carried out can be found in Gill et al.
[9] but the basic idea is similar to the one described in chapter 3. The QT
structure is obtained using givens rotations on specific parts of the modified QT
factorization after appending or removing a constraint.

We have implemented these updates. To find out how performance may be

improved we have plotted the computational speed when solving the quadruple
tank problem with N = 200, defining 1200 variables and 4800 constraints. We
tested both the null space procedure with the factorization update as described
in chapter 3 and the null space procedure with factorization update based on
fixed and free variables. From figure 5.2 it is clear that the recent update has
made a great improvement in computational time. But of course the improve-
ment is dependent on the number of active bounds in the specific problem.
88 Test and Refinements

Test of Speed: N =200 n =1200 m =4800

5 1200
null space update 1
4.5 null space update 2
1000
4

number of constraints
3.5 800
time [s]

3
600
2.5

2 400

1.5
200 total number of active constraints
number of active bounds
1
number of active general constraints

0.5 0
0 500 1000 1500 2000 2500 3000 3500 0 500 1000 1500 2000 2500 3000 3500
iteration iteration

(a) Computational time for solving the (b) The number of active bounds and active
KKT system plotted for each iteration. Null general constraints and the sum of the two
space update 2 is the new update based on plotted at each iteration.
fixed and free variables.

Figure 5.2: Computational time and the corresponding number of active bounds
and active general constraints plotted for each iteration when solving the
quadruple tank problem with N=200, (1200 variables and 4800 constraints).
The problem is solved using both the null space update and the null space
update based on fixed and free variables.

5.3 Corresponding Constraints

In our implementation we only consider inequality constraints, and they are

organized as shown in (5.1), where bounds and general constraints are connected
in pairs. So all constraints, by means all bounds and all general constraints
together, indexed i, are organized in I as follows

i ∈ I ={1, 2, ..., n, (5.5)

| {z }
x≥l

n + 1, n + 2, ..., 2n, (5.6)

| {z }
−x≥−u

2n + 1, 2n + 2, ..., 2n + mgc , (5.7)

| {z }
aT x≥bl
2n + mgc + 1, 2n + mgc + 2, ..., 2n + 2mgc } (5.8)
| {z }
−aT x≥−bu

and the corresponding pairs, indexed p, are then organized in P in the following
manner
5.3 Corresponding Constraints 89

p ∈ P ={n + 1, n + 2, ..., 2n, (5.9)

| {z }
−x≥−u

1, 2, ..., n, (5.10)
| {z }
x≥l

2n + mgc + 1, 2n + mgc + 2, ..., 2n + 2mgc , (5.11)

| {z }
−aT x≥−bu

2n + 1, 2n + 2, ..., 2n + mgc }. (5.12)

| {z }
aT x≥bl
(5.13)

Unbounded variables and unbounded general constraints, where the upper and/or
lower limits are ±∞ respectively, are never violated. So they are not considered,
when I and P are initialized. E.g. if l2 = −∞, then i = 2 will not exist in I,
and p2 = n + 2 will not exist in P.

In practice, the primal and the dual active set methods are implemented using
two sets, the active set given as the working set Wk and the inactive set I\Wk .
When a constraint j ∈ I\Wk becomes active, it is appended to the active set

Wk+1 = Wk ∪ {j} (5.14)

and because two corresponding inequality constraints cannot be active at the

same time, it is removed together with its corresponding pair pj ∈ P from the
inactive set as follows

I\Wk+1 = {I\Wk }\{j, pj }. (5.15)

When it becomes inactive it is removed from the active set

Wk+1 = Wk \{j} (5.16)

and appended to the inactive set together with its corresponding pair

I\Wk+1 = {I\Wk } ∪ {j, pj }. (5.17)

90 Test and Refinements

So by using corresponding pairs, we have two constraints less to examine feasi-

bility for, every time a constraint is found to be active.

Besides the gain of computational speed, the stability of the dual active set
method is also increased. Equality constraints are given as two inequalities
with the same value as upper and lower limits. So because of numerical in-
stabilities, the method tends to append corresponding constraints to the active
set, when it is close to the solution. If this is the case, the constraint matrix A
becomes linearly depended, and the dual active set method terminates because
of infeasibility. But by removing the corresponding pairs from the inactive set,
this problem will never occur. The primal active set method will always find the
solution before the possibility of two corresponding constraints becomes active
simultaneously, so for this method we gain computational speed only.

The quadruple tank problem is now solved, see figure 5.3, without removing the
corresponding pairs from the inactive set - so only the active constraints are
removed.

In figure 5.3(a) and 5.3(b) we see the indices of the constraints of the active
set Wk and the inactive set I\Wk respectively. Not surprisingly it is seen,
that the constraints currently in the active set are missing in the inactive set.
Also in figure 5.3(c) and 5.3(d) we see, that the relation between the number of
constraints in the active set and the number of constraints in the inactive set
as expected satisfy

|Wk | + |I\Wk | = |I|. (5.18)

5.3 Corresponding Constraints 91

W: i = [1;60], i = [61;120], i = [121;180], i = [181;240] I\W: i = [1;60], i = [61;120], i = [121;180], i = [181;240]

l u b b l u b b
l u l u
240 240

220 220

200 200

180 180

160 160

140 140

120 120
i

i
100 100

80 80

60 60

40 40

20 20

50 100 150 200 250 50 100 150 200 250

iteration iteration

(a) Indices of active bounds (blue) and ac- (b) Indices of inactive bounds (blue) and
tive general constraints (red) per iteration. inactive general constraints (red) per iter-
ation.

I \ W: max no bounds = 9, max no gen constr = 58, max total = 67 I\W: min no bounds = 111, min no gen constr = 62, min total = 173
240 240
no bounds no bounds
220 220
no gen constr no gen constr
200 total 200 total

180 180

160 160

140 140
no

no

120 120

100 100

80 80

60 60

40 40

20 20

50 100 150 200 250 50 100 150 200 250

iteration iteration

(c) Number of active bounds and general (d) Number of inactive bounds and general
constraints per iteration. constraints per iteration.

Figure 5.3: The process of solving the quadruple tank problem using the primal
active set method with N = 10, so n = 60 and |I| = 240, without removing the
corresponding pairs from the inactive set I\Wk .
92 Test and Refinements

The quadruple tank problem is now solved again, see figure 5.4, but this time
we remove the corresponding pairs from the inactive set as well.

W: i = [1;60], i = [61;120], i = [121;180], i = [181;240] I\W: i = [1;60], i = [61;120], i = [121;180], i = [181;240]

l u b b l u b b
l u l u
240 240

220 220

200 200

180 180

160 160

140 140

120 120
i

i
100 100

80 80

60 60

40 40

20 20

50 100 150 200 250 50 100 150 200 250

iteration iteration

(a) Indices of active bounds (blue) and ac- (b) Indices of inactive bounds (blue) and
tive general constraints (red) per iteration. inactive general constraints (red) per iter-
ation.

I \ W: max no bounds = 9, max no gen constr = 58, max total = 67 I\W: min no bounds = 102, min no gen constr = 4, min total = 106
240 240
no bounds no bounds
220 220
no gen constr no gen constr
200 total 200 total

180 180

160 160

140 140
no

no

120 120

100 100

80 80

60 60

40 40

20 20

50 100 150 200 250 50 100 150 200 250

iteration iteration

(c) Number of active bounds and general (d) Number of inactive bounds and general
constraints per iteration. constraints per iteration.

Figure 5.4: The process of solving the quadruple tank problem using the primal
active set method with N = 10, so n = 60 and |I| = 240, showing the effect of
removing the corresponding pairs from the inactive set I\Wk .

In figure 5.4(a) the indices of the constraints in the active set Wk are the same
as before the removal of the corresponding pairs. And in figure 5.4(b) we now
see, that all active constraints and their corresponding pairs are removed from
the inactive set I\Wk , and the set is seen to be much more sparse. The new
relation between the number of constraints in the active set and the number of
constraints in the inactive set is seen in figure 5.4(c) and 5.4(d). And the indices
of the constraints in the inactive set, when we also remove the corresponding
pairs, are found to be {I\Wk }\{pi }, i ∈ Wk . So now we have the new relation
5.4 Distinguishing Between Bounds and General Constraints 93

described as follows

2|Wk | + |{I\Wk }\{pi}| = |I|, i ∈ Wk . (5.19)

The size of {I\Wk }\{pi }, i ∈ Wk is found by combining (5.18) and (5.19) as

follows

2|Wk | + |{I\Wk }\{pi }| = |Wk | + |I\Wk |, i ∈ Wk (5.20)

which leads to

|{I\Wk }\{pi }| = |I\Wk | − |Wk |, i ∈ Wk . (5.21)

So we see, that the inactive set overall is reduced twice the size of the active set
by also removing all corresponding constraints pi , i ∈ Wk , from the inactive set.
This is also seen by comparing figure 5.4(c) and 5.3(c).

5.4 Distinguishing Between Bounds and Gen-

eral Constraints

In both the primal and the dual active set methods some computations involving
the constraints are made, e.g. checking the feasibility of the constraints. All
constraints in I are divided into bounds and general constraints, and via the
indices i ∈ I it is easy to distinguish, if a constraint is a bound or a general
constraint. This can be exploited to gain some computational speed, since
computations regarding a bound only involve the fixed variable, and therefore
it is very cheap to carry out.
94 Test and Refinements
Chapter 6
Nonlinear Programming

In this chapter we will investigate how nonlinear convex programs with nonlinear
constraints can be solved by solving a sequence of QP’s. The nonlinear program
is solved using Newton’s method and the calculation of a Newton step can
be formulated as a QP and found using a QP solver. As Newton’s method
solves a nonlinear program by a sequence of Newton steps, this method is called
sequential quadratic programming (SQP).

6.1 Sequential Quadratic Programming

Each step of Newton’s method is found by solving a QP. The theory is based
on the work of Nocedal and Wright [14] and Jørgensen [15]. To begin with, we
will focus on solving the equality constrained nonlinear program

min f (x) (6.1a)

x∈Rn
s.t. h(x) = 0 (6.1b)

where x ∈ Rn and h(x) ∈ Rm . This is done using the corresponding Lagrangian

96 Nonlinear Programming

function

L(x, y) = f (x) − y T h(x). (6.2)

The optimum is found by solving the corresponding KKT system

∇x L(x, y) = ∇f (x) − ∇h(x)y = 0 (6.3a)

∇y L(x, y) = −h(x) = 0. (6.3b)

The KKT system is written as a system of nonlinear equations as follows

 
F1 (x, y)
F (x, y) =
F2 (x, y)
 
∇x L(x, y)
=
∇y L(x, y)
 
∇f (x) − ∇h(x)y
= = 0. (6.4)
−h(x)

Newton’s method is used to solve this system. Newton’s method approximates

the root of a given function g(x) by taking successive steps in the direction of
∇g(x). A Newton step is calculated like this

g(xk ) + J(xk )∆x = 0, J(xk ) = ∇g(xk )T . (6.5)

As we want to solve (6.4) using Newton’s method, we need the gradient of

F (x, y) which is given by

 
F1 (x, y)
∇F (x, y) = ∇
F2 (x, y)
 ∂F1 ∂F2 
∂ x1 ∂ x1
= ∂F1 ∂F2
∂y2 ∂y2
 2 
∇xx L(x, y) −∇h(x)
= , (6.6)
−∇h(x)T 0
6.1 Sequential Quadratic Programming 97

where ∇2xx L(x, y) is the Hessian of L(x, y)

m
X
∇2xx L(x, y) = ∇2 f (x) − yi ∇2 hi (x). (6.7)
i=1

Because ∇F (x, y) is symmetric we know that J(x, y) = ∇F (x, y)T = ∇F (x, y),
and therefore Newton’s method (6.5) gives

    
∇2xx L(x, y) −∇h(x) ∆x ∇f (x) − ∇h(x)y
=− . (6.8)
−∇h(x)T 0 ∆y −h(x)

This system is the KKT system of the following QP

1
min ∆xT (∇2xx L(x, y))∆x + (∇x L(x, y))T ∆x (6.9a)
∆x∈Rn 2
s.t. ∇h(x)T ∆x = −h(x). (6.9b)

This is clearly a QP and the optimum (∆xT , ∆y T ) from (6.8) is found by using
a QP-solver, e.g. the one implemented in this thesis, see appendix B.

The system (6.8) can be expressed in a simpler form, by replacing ∆y with µ−y

    
∇2xx L(x, y) −∇h(x) ∆x ∇f (x) − ∇h(x)y
=− (6.10)
−∇h(x)T 0 µ−y −h(x)

which is equivalent to

    
∇2xx L(x, y) −∇h(x) ∆x ∇h(x)y
+ =
−∇h(x)T 0 µ 0
   
∇f (x) ∇h(x)y
− + . (6.11)
−h(x) 0
98 Nonlinear Programming

This means that (6.8) can be reformulated as

    
∇2xx L(x, y) −∇h(x) ∆x ∇f (x)
=− , (6.12)
−∇h(x)T 0 µ −h(x)

and the corresponding QP is

1
min ∆xT ∇2xx L(x, y)∆x + ∇f (x)T ∆x (6.13a)
∆x 2
s.t. ∇h(x)T ∆x = −h(x). (6.13b)

As Newton’s method approximates numerically, a sequence of Newton iterations

is thus necessary to find an acceptable solution. At every iteration the improving
direction is found as the solution of the QP (6.13), and therefore the process
is called sequential quadratic programming. Whenever ∇2xx L(x, y) is positive
definite and ∇h(x) has full column rank, the solution to (6.13) can be found
using either the range space procedure or the null space procedure. Also, if the
program (6.1) is extended to include inequalities

min f (x) (6.14a)

x∈Rn
s.t. h(x) ≥ 0 (6.14b)

then the program, that defines the Newton step is an inequality constrained QP
of the form

1
min ∆xT ∇2xx L(x, y)∆x + ∇f (x)T ∆x (6.15a)
∆x 2
s.t. ∇h(x)T ∆x ≥ −h(x). (6.15b)

When ∇2xx L(x, y) is positive definite and ∇h(x)T has full column rank the
solution to this program can be found using either the primal active set method
or the dual active set method.
6.2 SQP by example 99

6.2 SQP by example

In this section our SQP implementation will be tested and each Newton step
will be illustrated graphically. The nonlinear program that we want to solve is

min f (x) = x41 + x42

x∈Rn
s.t. x2 ≥ x21 − x1 + 1
x2 ≥ x21 − 4x1 + 6
x2 ≤ −x21 + 3x1 + 2.

The procedure is to minimize the corresponding Lagrangian function

L(x, y) = f (x) − y T h(x) (6.17)

where y are the Lagrangian multipliers and h(x) are the function values of the
constraints. This is done by using Newton’s method to find the solution of

 
F1 (x, y)
F (x, y) =
F2 (x, y)
 
∇x L(x, y)
=
∇y L(x, y)
 
∇f (x) − ∇h(x)y
= = 0. (6.18)
−h(x)

A Newton step is defined by the following QP

    
∇2xx L(x, y) −∇h(x) ∆x ∇f (x) − ∇h(x)y
=− . (6.19)
−∇h(x)T 0 ∆y −h(x)

In this example we have calculated the analytical Hessian matrix ∇2xx L(x, y) of
(6.17) to make the relation (6.19) exact, even though a BFGS update by Powell
[16] has been implemented. This is done for illustrational purpose alone, as we
want to plot each Newton step in F11 (x, y) and F12 (x, y) from (6.18) in relation
100 Nonlinear Programming

to the improving direction. The analytical Hessian matrix can be very expensive
to evaluate, and therefore the BFGS approximation is usually preferred. When
the improving direction [∆x, ∆y]T has been found by solving (6.19), the step
size α is calculated in a line search function implemented according to the one
suggested by Powell [16].

Iteration 1
We start at position (x0 , y 0 ) = ([−4, −4]T , [0, 0, 0]T ) with the corresponding
Lagrangian function value L(x0 , y 0 ) = 512. The first Newton step leads us
to (x1 , y 1 ) = ([−0.6253, −2.4966]T , [0, 32.6621, 0]T ) with L(x1 , y 1 ) = 410.9783,
and the path is illustrated in figure 6.1(a). In figures 6.1(b) and 6.1(c) F11
and F12 are plotted in relation to the step size α, where the red line illustrates
the step taken. We have plotted F11 and F12 for α ∈ [−1, 3] even though the
line search function returns α ∈ [0, 1]. In figures 6.1(b) and 6.1(c), α = 0 is the
position before the step is taken and α ∈ [0, 1] where the red line ends illustrates
the position after taking the step. It is clear from the figures, that a full step
(α = 1) is taken and that F11 and F12 increase from −256 to −172.4725, and
−256 to −94.9038, respectively.

F(x ) = −256 F(x ) = −256

1 old 2 old
F(x ) = −172.4725 F(x2)new = −94.9038
1 new
2000 0

1500
−200
1000
F2
F1

500 −400

0
−600
−500

−1000 −800
−1 0 1 2 3 −1 0 1 2 3
α α

(a) The path. (b) Newton step in F11 . (c) Newton step in F12 .

Figure 6.1: The Newton step at iteration 1. L(x0 , y 0 ) = 512 and L(x1 , y 1 ) =
410.9783.

Iteration 2
Having taken the second step the position is (x2 , y 2 ) = ([1.3197, −1.3201]T , [0, 25.7498, 0]T )
as seen in figure 6.2(a). The Lagrangian function value is L(x2 , y 2 ) = 103.4791.
The step size is α = 1, F11 increases from −172.4725 to −25.8427 and F12
increases from −94.9038 to −34.9515 as seen in figures 6.2(b) and 6.2(c).
6.2 SQP by example 101

F(x ) = −172.4725 F(x ) = −94.9038

1 old 2 old
F(x ) = −25.8427 F(x2)new = −34.9515
1 new
1000 0

−50
500
−100

F2
F1
0
−150

−500
−200

−1000 −250
−1 0 1 2 3 −1 0 1 2 3
α α

(a) The path. (b) Newton step in F11 . (c) Newton step in F12 .

Figure 6.2: The Newton step at iteration 2. L(x1 , y 1 ) = 410.9783 and

L(x2 , y 2 ) = 103.4791.

Iteration 3
After the third step, the position is (x3 , y 3 ) = ([1.6667, 1.9907]T , [15.7893, 44.2432, 0]T ),
see figure 6.3(a). The Lagrangian function value is L(x3 , y 3 ) = 30.6487. Again
the step size is α = 1, F11 increases from −25.8427 to 25.8647 and F12 increases
from −34.9515 to −28.4761 as seen in figure 6.3(b) and 6.3(c).

F(x ) = −25.8427 F(x ) = −34.9515

1 old 2 old
F(x ) = 25.8647 F(x2)new = −28.4761
1 new
300 2500

2000
200
1500
F2
F1

100 1000

500
0
0

−100 −500
−1 0 1 2 3 −1 0 1 2 3
α α

(a) The path. (b) Newton step in F11 . (c) Newton step in F12 .

Figure 6.3: The Newton step at iteration 3. L(x2 , y 2 ) = 103.4791 and

L(x3 , y 3 ) = 30.6487.

Iteration 4
The fourth step takes us to (x4 , y 4 ) = ([1.6667, 2.1111]T , [2.1120, 35.1698, 0]T ),
see figure 6.4(a). The Lagrangian function value is L(x4 , y 4 ) = 27.5790. The
step size is α = 1, F11 decreases from 25.8647 to −3.55271e−15 and F12 increases
from −28.4761 to 0.3533 as seen in figures 6.4(b) and 6.4(c). Even though
refinements can be made by taking more steps we stop the algorithm at the
optimal position (x∗ , y ∗ ) = (x4 , y 4 ) = ([1.6667, 2.1111]T , [2.1120, 35.1698, 0]T )
where the optimal value is f (x∗ ) = 27.5790.
102 Nonlinear Programming

F(x ) = 25.8647 F(x ) = −28.4761

1 old 2 old
F(x ) = −3.5527e−15 F(x2)new = 0.35325
1 new
60 100

40
50
20

F2
F1
0 0

−20
−50
−40

−60 −100
−1 0 1 2 3 −1 0 1 2 3
α α

(a) The path. (b) Newton step in F11 . (c) Newton step in F12 .

Figure 6.4: The Newton step at iteration 4. L(x3 , y 3 ) = 30.6487 and

L(x4 , y 4 ) = 27.5790.

An interactive demo application SQP demo.m is found in appendix D.5.

Chapter 7

Conclusion

In this thesis we have investigated the active set methods, together with the
range and null space procedures which are used in solving QP’s. We have also
focused on refining the methods and procedures in order to gain efficiency and
reliability. Now we will summarize the most important observations found in
the thesis.

The primal active set method is the most intuitive method. However, it has
two major disadvantages. Firstly, it requires a feasible starting point, which is
not trivial to find. Secondly and most crucially, is the possibility of cycling.
The dual active set method does not suffer from these drawbacks. The method
easily computes the starting point itself, and furthermore convergence is guar-
anteed. On the other hand, the primal active set method has the advantage of
only requiring the Hessian matrix to be positive semi definite.

The range space and the null space procedures are equally good. But, where
the range space procedure is fast, the null space procedure is slow and vice
versa. Thus, in practice the choice of method is problem specific. For problems
consisting of a small number of active constraints in relation to the number of
variables, the range space procedure is preferable. And for problems with a large
number active constraints compared to the number of variables, the null space
procedure is to be preferred. If the nature of the problem potentially allows a
large number of constraints in comparison to the number of variables, then, to
104 Conclusion

gain advantage of both procedures, it is necessary to shift dynamically between

them. This can easily be done by comparing the number of active constraints
against the number of variables e.g. for each iteration However, this requires
a theoretically predefined relation pointing at when to shift between the range
space and the null space procedures. This relation can as mentioned be found in
theory, but in practice it also relies on the way standard Matlab functions are
implemented, the architecture of the processing unit, memory access etc., and
therefore finding this relation in practice is more complicated than first assumed.

By using Givens rotations, the factorizations used to solve the KKT system can
be updated instead of completely recomputed. And as the active set methods
solve a sequence of KKT systems, the total computational savings are signifi-
cant. The null space procedure in particular has become more efficient. These
updates have been further refined by distinguishing bounds, i.e. fixed variables,
from general constraints. The greater fraction of active bounds compared to
active general constraints, the smaller the KKT system gets and vice versa.
Therefore, this particular update is of the utmost importance, when the QP
contains potentially many active bounds.

The SQP method is useful in solving nonlinear constrained programs. It is

founded in Newton steps. The SQP solver is based on a sequence of Newton
steps, where each single step is solved as a QP. So a fast and reliable QP solver
is essential in the SQP method. The QP solver which has been developed in the
thesis, see appendix B, has proved successful in fulfilling this task.

7.1 Future Work

Dynamic Shift Implementation of dynamic shift between the range space and
null space procedures would be interesting, because computational speed
could be gained this way.

Low Level Language Our QP solver has been implemented in Matlab, and
standard Matlab functions such as chol and qr have been used. In
future works, implementation in Fortran or C++ would be preferable.
This would make the performance tests of the different methods more
reliable. Implementation in any low level programming language may
be expected to improve general performance significantly. Furthermore,
any theoretically computed performances may also be expected to hold in
practice.

Precomputed Active Set The dual active set method requires the Hessian
matrix G of the objective function to be positive definite, as it computes
7.1 Future Work 105

the starting point x0 by use of the inverse Hessian matrix: x0 = −G−1 g.

The primal active set method using the null space procedure only requires
the reduced Hessian matrix Z T GZ to be positive definite. In many prob-
lems it is possible to find an active set which makes the reduced Hessian
matrix positive definite even if the Hessian matrix is positive semi definite.
In future works the LP solver which finds the starting point to the primal
active set method should be designed so that it also finds the active set
which makes the reduced Hessian matrix positive definite. This extension
would give the primal active set method an advantage compared to the
dual active set method.
106 Conclusion
Bibliography

[1] Li, W. and Swetits, J. J.

The Linear l1 Estimator and the Huber M-Estimator, SIAM Journal on
Optimization, (1998).

[2] Gill, P E., Gould, N. I. M., Murray, W., Saunders, M. A., Wright, M. H.
A Weighted Gram-Schmidt Method for Convex Quadratic Pro- gramming.
Mathematical Programming, 30, (1984).

[3] Gill, P. E. and Murray, W.

Numerically Stable Methods for Quadratic Programming. Mathematical Pro-
gramming, 14, (1978).

[4] Golub, G. H. and Van Loan, C. F.

Matrix Computations, (1996).

[5] Wilkinson, J. H.
The Algebraic Eigenvalue Problem, (1965).

[6] Dennis, J. E. and Schnabel, R. B.

Numerical Methods for Unconstrained Optimization and Nonlinear Equa-
tions., (1996).

[7] Gill, P. E., Golub, G. H., Murray, W. and Saunders, M. A.

Methods for Modifying Matrix Factorizations. Mathematics of Computation,
28., (1974).

[8] Gill, P. E. and Murray, W. Numerically Stable Methods for Quadratic Pro-
gramming. Mathematical Programming, 14.,(1978).
108 BIBLIOGRAPHY

[9] Gill, P. E., Murray, W., Saunders, M. E. and Wright, M. H. Procedures

for Optimization Problems with a Mixture of Bounds and General Linear
Constraints. ACM Transactions on Mathematical Software, 10.,(1984).
[10] Goldfarb, D. and Idnani, A.
A numerically stable dual method for solving strictly convex quadratic pro-
grams, (1983).
[11] Schmid, C. and Biegler, L.
Quadratic programming methods for reduced hessian SQP, (1994).
[12] Schittkowski, K.
QL: A Fortran Code for Convex Quadratic Programming - Users Guide.
Technical report, Department of Mathematics, University of Bayreuth,
(2003).
[13] John Bagterp Jørgensen.
Quadratic Programming, (2005).
[14] Nocedal, J. and Wright, S. J.
Numerical Optimization, Springer Series in Operations Research, Second
Edition, (2006).
[15] John Bagterp Jørgensen.
Lecture notes from course 02611 Optimization Algorithms and Data-Fitting,
IMM, DTU, DK-2800 Lyngby, (november 2005).
[16] Powell, M. J. D.
A Fast Algorithm for Nonlinearly Constrained Optimization Calculations. In
G. A. Watson, editor, Numerical Analysis, (1977).

[17] John Bagterp Jørgensen.

Lecture notes from course: Model Predictive Control, IMM, DTU, DK-2800
Lyngby, (february 2007).
[18] L. Eldén, L. Wittmeyer-Koch and H.B. Nielsen:
Introduction to Numerical Computation, published by Studentlitter-
atur (2002).
Appendix A

Quadruple Tank Process

The quadruple tank process Jørgensen [17] is a system of four tanks, which are
connected through pipes as illustrated in figure A.1. Water from a main-tank
is transported around the system and the flow is controlled by the pumps F1
and F2 . The optimization problem is to stabilize the water level in tank 1 and
2 at some level, called set points illustrated as a red line. Values γ1 and γ2 of
the two valves control how much water is pumped directly into tank 1 and 2
respectively. The valves are constant, and essential for the ease with which the
process is controlled.

The dynamics of the quadruple tank process are described in the following
differential equations

dh1 γ1 a3 p a1 p
= F1 + 2gh3 − 2gh1 (A.1a)
dt A1 A1 A1
dh2 γ2 a4 p a2 p
= F2 + 2gh4 − 2gh2 (A.1b)
dt A2 A2 A2
dh3 1 − γ2 a3 p
= F2 − 2gh3 (A.1c)
dt A3 A3
dh4 1 − γ1 a4 p
= F1 − 2gh4 (A.1d)
dt A4 A4
110 Quadruple Tank Process

Figure A.1: Quadruple Tank Process.

where Ai is the cross sectional area, ai is the area of outlet pipe, hi is the water
level of tank no. i, γ1 and γ2 are flow distribution constants of the two valves,
g is acceleration of gravity and F1 and F2 are the two rate of flows. As the QP
solver requires the constraints to be linear we need to linearize the equations in
(A.1). Of course this linearization causes the model to be a much more coarse
approximation, but as the purpose is to build a convex QP for testing, this is
of no importance. The linearizations are
 111

dh1 γ1 a3 a1
= F1 + 2gh3 − 2gh1 (A.2a)
dt A1 A1 A1
dh2 γ2 a4 a2
= F2 + 2gh4 − 2gh2 (A.2b)
dt A2 A2 A2
dh3 1 − γ2 a3
= F2 − 2gh3 (A.2c)
dt A3 A3
dh4 1 − γ1 a4
= F1 − 2gh4 . (A.2d)
dt A4 A4

This system of equations is defined as the function

d
x(t) = f (x(t), u(t)), x = [h1 h2 h3 h4 ]T , u = [F1 F2 ]T (A.3)
dt

which is discretized using Euler

d x(tk+1 ) − x(tk ) xk+1 − xk

x(t) ≃ = = f (xk , uk ) (A.4a)
dt tk+1 − tk ∆t
xk+1 = xk + ∆tf (xk , uk ) (A.4b)
F (xk , uk , xk+1 ) = xk + ∆tf (xk , uk ) − xk+1 = 0. (A.4c)

The function F (xk , uk , xk+1 ) = 0 defines four equality constraints, one for
each height in x. As the time period is discretized into N time steps, this
gives 4N equality constraints. Because each equality constraint is defined as
two inequality constraints of identical value, as lower and upper bound, (A.4)
defines 8N inequality constraints, called general constraints.

To make the simulation realistic we define bounds on each variable

umin ≤ uk ≤ umax (A.5a)

xmin ≤ xk ≤ xmax (A.5b)

which gives 2N (|u| + |x|) = 12N inequality constraints, called bounds. We have
also defined restrictions on how much the rate of flows can change between two
time steps
112 Quadruple Tank Process

∆umin ≤ uk − uk−1 ≤ ∆umax (A.6)

in addition this gives 2N |u| = 4N inequality constraints, also general con-

straints.

The objective function which we want to minimize is

Z
1
min ((h1 (t) − r1 )2 + (h2 (t) − r2 )2 )dt (A.7)
2

where r1 and r2 are the set points. The exact details of how the system is set
up as a QP can be found in either Jørgensen [17] or our Matlab implemen-
tation quad tank demo.m. The quadruple tank process defines an inequality
constrained convex QP of (|u| + |x|)N = 6N variables and (8 + 12 + 4)N = 24N
inequality constraints consisting of 12N bounds and 12N general constraints.

Quadruple Tank Process by example

Now we will set up a test example of the quadruple tank problem. For this we
use the following settings

t = [0, 360]
N = 100
umin = [0, 0]T
umax = [500, 500]T
∆umin = [−50, −50]T
∆umax = [50, 50]T
u0 = [0, 0]T
γ1 = 0.45
γ2 = 0.40
r1 = 30
r2 = 30
xmin = [0, 0, 0, 0]T
xmax = [40, 40, 40, 40]T
x0 = [0, 0, 0, 0]T .
 113

This defines an inequality constrained convex QP with 6 ∗ 100 = 600 variables

and 24 ∗ 100 = 2400 constraints. The solution to the problem is found by
using our QP solver, see appendix B. The solution is illustrated in figure A.2,
where everything is seen to be as expected. We have also written a program
quad tank plot.m for visualizing the solution of the quadruple tank problem as
an animation. In figure A.3 to A.8 we have illustrated the solution x∗k and u∗k
for k ∈ {1, 3, 6, 10, 15, 20, 25, 30, 40, 60, 80, 100} using quad tank plot.m.

40 40

30 30
h [cm]

h [cm]
20 20

2
1

10 10

0 0
0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 400
t [min] t [min]

40 40

30 30
h3 [cm]

h [cm]

20 20
4

10 10

0 0
0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 400
t [min] t [min]

600 600

400 400
F1 [cm /s]

F [cm3/s]
3

200 200
2

0 0

−200 −200
−50 0 50 100 150 200 250 300 350 400 −50 0 50 100 150 200 250 300 350 400
t [min] t [min]

100 100

50 50
∆ F1 [cm /s ]

∆ F [cm3/s2]
3 2

0 0
2

−50 −50

−100 −100
−50 0 50 100 150 200 250 300 350 400 −50 0 50 100 150 200 250 300 350 400
t [min] t [min]

Figure A.2: The solution of the quadruple tank problem found by using our QP
solver. It is seen that the water levels in tank 1 and 2 are stabilized around the
setpoints. The water levels in tank 3 and 4, the two flows F1 and F2 and the
difference in flow between time steps ∆F1 and ∆F2 are also plotted.

An interactive demo application quad tank demo.m is found in appendix D.5.

114 Quadruple Tank Process

(a) k = 1 (b) k = 3

Figure A.3: discretization k = 1 and k = 3.

(a) k = 6 (b) k = 10

Figure A.4: discretization k = 6 and k = 10.

(a) k = 15 (b) k = 20

Figure A.5: discretization k = 15 and k = 20.

 115

(a) k = 25 (b) k = 30

Figure A.6: discretization k = 25 and k = 30.

(a) k = 40 (b) k = 60

Figure A.7: discretization k = 40 and k = 60.

(a) k = 80 (b) k = 100

Figure A.8: discretization k = 60 and k = 100.

116 Quadruple Tank Process
Appendix B

QP Solver Interface

Our QP solver is implemented in Matlab as QP solver.m, and it is founded

on the following structure

1
minn f (x) = xT Gx + g T x (B.1a)
x∈R 2
s.t. li ≤ xi ≤ ui i = 1, 2, ..., n (B.1b)
(bl )i ≤ aTi x ≤ (bu )i i = 1, 2, ..., m (B.1c)

where f is the objective function. The number of bounds is 2n and the number
of general constraints is 2m. This means, that we have upper and lower limits
on every bound and on every general constraint. The Matlab interface of the
QP solver is constructed as follows

x = QP solver(G, g, l, u, A, bl, bu, x).

The input parameters of the QP solver are described in table B.1.

118 QP Solver Interface

G The Hessian matrix G ∈ Rn×n of the objective function.

g The linear term g ∈ Rn of the objective function.
l The lower limits l ∈ Rn of bounds.
u The upper limits u ∈ Rn of bounds.
A The constraint matrix A = [aTi ]i=1,2,...,m , so A ∈ Rm×n .
bl The lower limits bl ∈ Rm of general constraints.
bu The upper limits bu ∈ Rm of general constraints.
x A feasible starting point x ∈ Rn used in the primal active set method. If x is
not given or empty, then the dual active set method is called within the
QP solver.

Table B.1: The input parameters of the QP and the LP solver.

It is possible to define equalities, by means the lower and upper limits are equal,
as li = ui and (bl )i = (bu )i respectively. If any of the limits are unbounded,
they must be defined as −∞ for lower limits and ∞ for upper limits. If the QP
solver is called with a starting point x, then the primal active set method is
called within the QP solver. The feasibility of the starting point is checked by
the QP solver before the primal active set method is called.

It is possible to find a feasible starting point with our LP solver, which we

implemented in Matlab as LP solver.m. The LP solver is based on (B.1)
and the Matlab function linprog. The Matlab interface of the LP solver is
constructed as follows

x = LP solver(l, u, A, bl, bu).

The input parameters of the LP solver are described in table B.1.

It must be mentioned, that both the QP and LP solver has some additional
input and output parameters. These parameters are e.g. used for tuning and
performance analysis of the solvers. For a further description of these parameters
we refer to the respective Matlab help files.
Appendix C

Implementation

The algorithms discussed in the thesis have been implemented in Matlab ver-
sion 7.3.0.298 (R2006b), August 03, 2006. In the following, we have listed
the implemented functions in sections.

C.1 Equality Constrained QP’s

The null space procedure solves an equality constrained QP by using the null
space of the constraint matrix, and is implemented in

null space.m .

The range space procedure solves the same problem by using the range space of
the constraint matrix, and is implemented in

range space.m .
120 Implementation

C.2 Inequality Constrained QP’s

An inequality constrained convex QP can be solved by use of the primal active

set method which is implemented in

primal active set method.m

or the dual active set method

dual active set method.m .

The active set methods have been integrated in a QP solver that sets up the
QP with a set of bounds and a set of general constraints. An interface has been
provided that offers different options to the user.

QP solver.m .

If the user want to use the primal active set method in the QP solver, a feasible
starting point must be calculated. This can be done by using the LP solver

LP solver.m .

C.3 Nonlinear Programming

SQP solves a nonlinear program by solving a sequence of inequality constrained

QP’s. The SQP solver is implemented in

SQP solver.m .

C.4 Updating the Matrix Factorizations

By updating the matrix factorizations, the efficiency of the null space procedure
and the range space procedure can be increased significantly. The following
C.4 Updating the Matrix Factorizations 121

implementations are used for updating the matrix factorizations used in the
null space procedure and the range space procedure. All the updates are based
on Givens rotations which are computed in

givens rotation matrix.m .

Update of matrix factorizations used in range space procedure are implemented

in the following files

range space update.m

qr fact update app col.m

qr fact update rem col.m .

And for the null space procedure the matrix updates are implemented in

null space update.m

null space update fact app col.m

null space update fact rem col.m .

Further optimization of the matrix factorization is done by using updates based

on fixed and free variables. These updates have only been implemented for the
null space procedure and are found in

null space updateFRFX.m

null space update fact app general FRFX.m

null space update fact rem general FRFX.m

null space update fact app bound FRFX.m

null space update fact rem bound FRFX.m

122 Implementation

C.5 Demos

For demonstrating the methods and procedures, we have implemented different

demonstration functions. The QP solver is demonstrated in

QP demo.m

which uses the plot function

active set plot.m .

Among other options the user can choose between the primal active set method
and the dual active set method.

The QP solver is also demonstrated on the quadruple tank process in

quad tank demo.m

which uses the plot functions

quad tank animate.m

quad tank plot.m .

Besides having the possibility of adjusting the valves, pumps and the set points
individually, the user can vary the size of the QP by the input N .

The SQP solver is demonstrated on a small two dimensional nonlinear program

and the path is visualized at each iteration. The implementation is found in

SQP demo.m .

C.6 Auxiliary Functions

add2mat.m .
line search algorithm.m .
Appendix D

Matlab-code

D.1 Equality Constrained QP’s

null space.m

1 function [ x , u ] = n u l l s p a c e (G, A, g , b )
2
3 % NULL SPACE s o l v e s t h e e q u a l i t y c o n s t r a i n e d c o n v e x QP:
4 % min 1/2 x ’ Gx+g ’ x (G i s r e q u i r e d t o be p o s t i v e s e m i
definite )
5 % s.t. A’ x = b (A i s r e q u i r e d t o hav e f u l l col um n
rank )
6 % w here t h e number o f v a r i a b l e s i s n and t h e number o f c o n s t r a i n t s i s m.
7 % The n u l l s p a c e o f t h e OP i s u s e d t o f i n d t h e s o l u t i o n .
8 %
9 % Call
10 % [ x , u ] = n u l l s p a c e (G, A, g , b )
11 %
12 % Input parameters
13 % G : i s t h e H e s s i a n m a t r i x ( nxn ) o f t h e QP.
14 % A : i s t h e c o n s t r a i n t m a t r i x ( nxm) : e v e r y col um n c o n t a i n s
a from t h e
15 % equality : a ’ x = b .
16 % g : i s t h e g r a d i e n t ( nx1 ) o f t h e QP .
17 % b : i s t h e r i g h t hand s i d e o f t h e c o n s t r a i n t s .
18 %
19 % Output p a r a m e t e r s
20 % x : the s o l u t i o n
21 % mu : the l ag r ang i an m u l t i p l i e r s
22 %
23 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
24 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
25 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
26 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
27 % D ate : 08. february 2007.
28
29 [ n ,m] = s i z e (A) ;
30 i f ( m˜=0 ) % f o r situations w here A i s empty
124 Matlab-code

31 [ Q, R ] = q r (A) ;
32 Q1 = Q ( : , 1 :m) ;
33 Q2 = Q ( : ,m+ 1:n ) ;
34 R = R ( 1 : m, : ) ;
35 py = R’ \ b ;
36 Q2t = Q2 ’ ;
37 g z = Q2t ∗ (G∗ (Q1∗py ) + g ) ;
38 Gz = Q2t ∗G∗Q2 ;
39 L = c h o l ( Gz ) ’ ;
40 pz = L\−g z ;
41 pz = L ’ \ pz ;
42 x = Q1∗py + Q2∗ pz ;
43 u = R\ (Q1 ’ ∗ (G∗x + g ) ) ;
44 else
45 x = −G\ g ;
46 u = [];
47 end

range space.m

1 f u n c t i o n [ x mu ] = r a n g e s p a c e ( L , A, g , b )
2 % RANGE SPACE s o l v e s t h e e q u a l i t y c o n s t r a i n e d c o n v e x QP:
3 % min 1/2 x ’ Gx+g ’ x (G i s r e q u i r e d t o be p o s t i v e
definite )
4 % s.t. A’ x = b (A i s r e q u i r e d t o hav e f u l l col um n
rank )
5 % w here t h e number o f v a r i a b l e s i s n and t h e number o f c o n s t r a i n t s i s m.
6 % The r a n g e s p a c e o f t h e OP i s u s e d t o f i n d t h e s o l u t i o n .
7 %
8 % Call
9 % [ x , u ] = r a n g e s p a c e ( L , A, g , b )
10 % Input parameters
11 % L : i s th e c h o l e s k y f a c t o r i z a t i o n o f th e H es s i an matrix (
nxn ) o f t h e QP.
12 % A : i s t h e c o n s t r a i n t m a t r i x ( nxm) : e v e r y col um n c o n t a i n s
a from t h e
13 % equality : a ’ x = b .
14 % g : i s t h e g r a d i e n t ( nx1 ) o f t h e QP .
15 % b : i s t h e r i g h t hand s i d e o f t h e c o n s t r a i n t s .
16 % Output p a r a m e t e r s
17 % x : the s o l u t i o n
18 % mu : the l a gr an gi an m u l t i p l i e r s
19
20 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
21 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
22 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
23 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
24 % D ate : 08. february 2007.
25 % Reference : −−−−−−−−−−−−−−−−−−−−−
26
27 Lt = L ’ ;
28 K = Lt \A ;
29 H = K’ ∗K ;
30 w = Lt \ g ;
31 z = b+K’ ∗w ;
32 M = c h o l (H) ;
33 mu = M’ \ z ;
34 mu = M\mu ;
35 y = K∗mu−w ;
36 x = L\y ;
D.2 Inequality Constrained QP’s 125

D.2 Inequality Constrained QP’s

primal active set method.m

1 f u n c t i o n [ x , mu, i n f o , p e r f ] = p r i m a l a c t i v e s e t m e t h o d (G, g , A , b , x , w non , pbc , o p t s ,

trace )
2
3 % PRIMAL ACTIVE SET METHOD S o l v i n g an i n e q u a l i t y c o n s t r a i n e d QP o f t h e
4 % form :
5 % min f ( x ) = 0 . 5 ∗ x ’ ∗G∗x + g ∗x
6 % s . t . A∗x >= b ,
7 % by s o l v i n g a s e q u e n c e o f e q u a l i t y c o n s t r a i n e d QP’ s u s i n g t h e p r i m a l
8 % a c t i v e s e t method . The method u s e s t h e r a n g e s p a c e p r o c e d u r e o r t h e n u l l
9 % s p a c e p r o c e d u r e t o s o l v e t h e KKT s y s t e m . Both t h e r a n g e s p a c e and t h e
10 % n u l l s p ace p r o c e d u r e s has been p rov i d ed with f a c t o r i z a t i o n u p d ates .
11 %
12 % Call
13 % x = p r i m a l a c t i v e s e t m e t h o d (G, g , A, b , w non , pbc )
14 % x = p r i m a l a c t i v e s e t m e t h o d (G, g , A, b , w non , pbc , o p t s )
15 % [ x , mu, i n f o , p e r f ] = p r i m a l a c t i v e s e t m e t h o d ( . . . )
16 %
17 % Input parameters
18 % G : The H e s s i a n m a t r i x o f t h e o b j e c t i v e f u n c t i o n , s i z e nxn .
19 % g : The l i n e a r term o f t h e o b j e c t i v e f u n c t i o n , s i z e nx1 .
20 % A : The c o n s t r a i n t m a t r i x h o l d i n g t h e c o n s t r a i n t s , s i z e nxm .
21 % b : The r i g h t −hand s i d e o f t h e c o n s t r a i n t s , s i z e mx1 .
22 % x : S t a r t i n g p o i n t , s i z e nx1 .
23 % w non : L i s t o f i n a c t i v e c o n s t r a i n t s , p o i n t i n g on c o n s t r a i n t s i n A .
24 % pbc : L i s t o f c o r r e s p o n d i n g c o n s t r a i n t s , p o i n t i n g on c o n s t r a i n t s i n
25 % A . Can be empty .
26 % op ts : V ector with 3 el em en ts :
27 % o p t s ( 1 ) = T o l e r a n c e u s e d t o s t a b i l i z e t h e m ethods n u m e r i c a l l y .
28 % I f | v a l u e | <= o p t s ( 1 ) , t h e n v a l u e i s r e g a r d e d a s z e r o .
29 % o p t s ( 2 ) = maximum no . o f i t e r a t i o n s t e p s .
30 % opts ( 3 ) = 1 : Using n u l l space procedure .
31 % 2 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
32 % update .
33 % 3 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
34 % u p d a t e b a s e d on f i x e d and f r e e v a r i a b l e s . Can o n l y
35 % be c a l l e d , i f t h e i n e q u a l i t y c o n s t r a i n e d QP i s
36 % s e t u p on t h e form s e e n i n Q P s o l v e r .
37 % I f o p t s i s n o t g i v e n o r empty , t h e d e f a u l t o p t s = [ 1 e−8 1 0 0 0 3 ] .
38 %
39 % Output parameters
40 % x : The o p t i m a l s o l u t i o n .
41 % mu : The L a g r a n g e m u l t i p l i e r s a t t h e o p t i m a l s o l u t i o n .
42 % info : Performace i n form ati on , v e c t o r with 3 el em en ts :
43 % inf o (1) = f i n a l v al ues o f the o b j e c t i v e fu n ct i o n .
44 % inf o (2) = no . o f i t e r a t i o n s t e p s .
45 % inf o (3) = 1 : F e a s i b l e s o l u t i o n found .
46 % 2 : No . o f i t e r a t i o n s t e p s e x c e e d e d .
47 % perf : Performace , s t r u c t h o l d i n g :
48 % perf . x : V a l u e s o f x , s i z e i s nx ( i t +1) .
49 % perf . f : Values o f the o b j e c t i v e f u n c t i o n , s i z e i s 1 x ( i t +1) .
50 % p e r f . mu : V a l u e s o f mu, s i z e i s nx ( i t +1) .
51 % perf . c : V a l u e s o f c ( x ) , s i z e i s mx( i t +1) .
52 % p e r f .Wa : A c t i v e s e t , s i z e i s mx( i t +1) .
53 % p e r f . Wi : I n a c t i v e s e t , s i z e i s mx( i t +1) .
54 %
55 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 .
56 % Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
57 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n .
58 % M. Sc . , IMM, DTU, DK−2800 Lyngby .
59 % Supervisor : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r .
60 % Per Grove Thomsen , P r o f e s s o r .
61 % D ate : 0 7 . J une 2 0 0 7 .
62
63 % the s i z e of the constraint matrix , w here the constraints are given columnwise
...
64 [ n ,m] = s i z e (A) ;
65
66 nb = 2∗ n ; % number o f bounds
67 ngc = m − nb ; % number o f g e n e r a l constraints
68
69 % initialize ...
70 z = z e r o s (m, 1 ) ;
71 x0 = x ;
72 f 0 = o b j e c t i v e (G, g , x0 ) ;
73 mu0 = z ;
74 c 0 = c o n s t r a i n t s (A, b , x0 ) ;
75 w act0 = z ;
76 w non0 = ( 1 : 1 :m) ’ ;
126 Matlab-code

77
78
79 % i n i t i a l i z e options . . .
80 t o l = opts ( 1) ;
81 it max = opts (2 ) ;
82 method = o p t s ( 3 ) ;
83 % i n i t i a l i z e containers . . .
84 %t r a c e = ( n a r g o u t > 3 ) ;
85 perf = [ ] ;
86 i f trace
87 X = repm at ( z e r o s ( n , 1 ) , 1 , i t m a x ) ;
88 F = repm at ( 0 , 1 , i t m a x ) ;
89 Mu = repm at ( z , 1 , i t m a x ) ;
90 C = repm at ( z , 1 , i t m a x ) ;
91 W act = repm at ( z , 1 , i t m a x ) ;
92 W non = repm at ( z , 1 , i t m a x ) ;
93 end
94
95 % i n i t i a l i z e counters . . .
96 i t = 0;
97 Q = [ ] ; T = [ ] ; L = [ ] ; rem = []; % both for null space update and for
null space update FXFR
98
99 if method == 3 % n u l l space w i t h FXFR−u p d a t e
100 nb1 = n ∗ 2 ; % number o f bounds
101 else
102 nb1 = 0 ;
103 end
104
105 nab = 0 ; % number of active bouns
106 P = eye ( n ) ;
107
108 w act = [];
109 Q old = []; T old = []; L old = []; rem = [];
110
111 % iterate . . .
112 stop = 0 ;
113 w hi l e ˜ stop
114 it = i t + 1;
115 i f i t >= i t m a x
116 s t o p = 2 ; % maximum no iterations exceeded
117 end
118
119 % c a l l range / n u l l space procedure . . .
120 mu = z ;
121
122 if method == 1
123 [ p , mu ] = n u l l s p a c e (G, A ( : , w a c t ) ,G∗x+g , z e r o s ( l e n g t h ( w a c t ) , 1 ) ) ;
124 end
125
126 if method == 2
127 [ p , mu , Q, T , L ] = n u l l s p a c e u p d a t e (G, A ( : , w a c t ) ,G∗x+g , z e r o s ( l e n g t h ( w a c t
) , 1 ) ,Q, T , L , rem ) ;
128 end
129
130 if method == 3
131 Cr = G∗x+g ;
132 A = A ( : , w a c t ( nab + 1: end ) ) ;
133 % a j u s t C ( : , r ) t o make i t c o r r e s p o n d t o t h e f a c t o r i z a t i o n s o f
the
134 % F i x e d v a r i a b l e s ( w h e n e v e r −1 a p p e a r s a t v a r i a b l e i C ( : , r ) i
s houl d change s i g n )
135 i f nab % some bounds a r e i n t h e a c t i v e s e t
136 u i d x = f i n d ( w a c t > nb1 /2 & w a c t< nb1 +1) ;
137 v a r = n−nab+u i d x ;
138 Cr ( v a r ) = −Cr ( v a r ) ;
139 A ( v ar , : ) = −A ( v ar , : ) ;
140 end
141 [ p , mu , Q, T , L ] = n u l l s p a c e u p d a t e F R F X (Q, T , L , G, A , Cr , z e r o s ( l e n g t h ( w a c t )
, 1 ) , nab , rem−nab ) ;
142 end
143
144 mu( w a c t ) = mu ;
145
146 if norm ( p ) < t o l
147 i f mu > − t o l
148 s top = 1 ; % s o l u t i o n found
149 else
150 % com pute i n d e x j o f bound / c o n s t r a i n t t o be rem ov ed . . .
151 [ dummy , rem ] = min ( mu ) ;
152 [ w a c t w non A P x nab G g ] = r e m o v e c o n s t r a i n t ( rem , A , w a c t , w non , x
, P , nb1 , nab , n , G, g , pbc , b ) ;
153 end
154 else
155 % com pute s t e p l e n g t h and i n d e x j o f bound / c o n s t r a i n t t o be appended . . .
156 [ a l p h a , j ] = s t e p l e n g t h (A, b , x , p , w non , nb , n , t o l ) ;
157 i f alpha < 1
D.2 Inequality Constrained QP’s 127

158 % make c o n s t r a i n e d s t e p . . .
159 x = x + a l p h a ∗p ;
160 [ w a c t w non A P x nab G g Q] = a p p e n d c o n s t r a i n t ( j , A , w a c t , w non , x
, P , nb1 , nab , n , G, g , Q, pbc , b ) ; % r i s i n d e x o f A
161 else
162 % make f u l l step . . .
163 x = x + p;
164 end
165 end
166 % c o l l e c t i n g output i n c o n t a i n e r s . . .
167 i f trace
168 i f nb1 % method 3 i s u s e d
169 X ( : , i t ) = P’ ∗ x ;
170 else
171 X( : , i t ) = x ;
172 end
173 F ( i t ) = o b j e c t i v e (G, g , x ) ;
174 Mu ( : , i t ) = mu ;
175 C ( : , i t ) = c o n s t r a i n t s (A, b , x ) ;
176 W act ( w a c t , i t ) = w a c t ;
177 W non ( w non , i t ) = w non ;
178 end
179 end
180
181 if nb1 % method 3 is used
182 x = P’ ∗ x ;
183 end
184
185 % building info . . .
186 i n f o = [ o b j e c t i v e (G, g , x ) it stop ] ;
187
188 % building perf . . .
189 i f trace
190 X = X ( : , 1 : i t ) ; X = [ x0 X ] ;
191 F = F(1: i t ) ; F = [ f0 F ] ;
192 Mu = Mu ( : , 1 : i t ) ; Mu = [ mu0 Mu ] ;
193 C = C ( : , 1 : i t ) ; C = [ c0 C ] ;
194 W act = W act ( : , 1 : i t ) ; W act = [ w a c t 0 W act ] ;
195 W non = W non ( : , 1 : i t ) ; W non = [ w non0 W non ] ;
196 p e r f = s t r u c t ( ’ x ’ , {X} , ’ f ’ , { F} , ’mu ’ , {Mu} , ’ c ’ , {C} , ’Wa ’ , { W act } , ’ Wi ’ , { W non } ) ;
197 end
198
199 f u n c t i o n [ a l p h a , j ] = s t e p l e n g t h (A , b , x , p , w non , nb , n , t o l )
200 alpha = 1 ; j = [ ] ;
201 f o r app = w non
202 i f app > nb
203 fv = 1 : 1 : n ; % general constraint
204 else
205 f v = mod ( app −1 ,n ) + 1; % i n d e x o f f i x e d v a r i a b e l
206 end
207 ap = A( f v , app ) ’ ∗ p ( f v ) ;
208 i f ap < − t o l
209 temp = ( b ( app ) − A( f v , app ) ’ ∗ x ( f v ) ) / ap ;
210 i f − t o l < temp & temp < a l p h a
211 a l p h a = temp ; % s m a l l e s t s t e p l e n g t h
212 j = app ; % i n d e x j o f bound t o be appended
213 end
214 end
215 end
216
217 % f u n c t i o n [ w a c t , w non ] = a p p e n d c o n s t r a i n t ( b , w a c t , w non , j , pbc )
218 % w a c t = [ w a c t j ] ; % append c o n s t r a i n t j t o a c t i v e s e t
219 % w non = w non ( f i n d ( w non ˜= j ) ) ; % rem ov e c o n s t r a i n t j from n o n a c t i v e s e t
220 % i f ˜ i s i n f ( b ( pbc ( j ) ) )
221 % w non = w non ( f i n d ( w non ˜= pbc ( j ) ) ) ; % rem ov e c o n s t r a i n t pbc ( j ) from
n o n a c t i v e s e t , i f n o t unbounded
222 % end
223
224 % f u n c t i o n [ w a c t , w non ] = r e m o v e c o n s t r a i n t ( b , w a c t , w non , j , pbc )
225 % w a c t = w a c t ( f i n d ( w a c t ˜= j ) ) ; % rem ov e c o n s t r a i n t j from a c t i v e s e t
226 % w non = [ w non j ] ; % append c o n s t r a i n t j t o n o n a c t i v e s e t
227 % i f ˜ i s i n f ( b ( pbc ( j ) ) )
228 % w non = [ w non pbc ( j ) ] ; % append c o n s t r a i n t pbc ( j ) t o n o n a c t i v e s e t , if
n o t unbounded
229 % end
230
231 f u n c t i o n [ w a c t w non C P x nab G g ] = r e m o v e c o n s t r a i n t ( wi , C , w a c t , w non , x , P ,
nb , nab , n , G, g , pbc , b ) % w i i s i n d e x o f w a c t
232 j = w a c t ( wi ) ;
233
234 if j < nb+1 % j is a bound and we hav e to
r e o r g a n i z e the v a r i a b l e s
235 v a r 1 = n−nab + 1;
236 v a r 2 = n−nab+w i ;
237
238 temp = C( v ar1 , : ) ;
239 C( v ar1 , : ) = C( v ar2 , : ) ;
128 Matlab-code

240 C( v ar2 , : ) = temp ;

241
242 temp = x ( v a r 1 ) ;
243 x ( var1 ) = x ( var2 ) ;
244 x ( v a r 2 ) = temp ;
245
246 temp = P ( v ar1 , : ) ;
247 P ( v ar1 , : ) = P ( v ar2 , : ) ;
248 P ( v ar2 , : ) = temp ;
249
250 temp = G( v ar1 , v a r 1 ) ;
251 G( v ar1 , v a r 1 ) = G( v ar2 , v a r 2 ) ;
252 G( v ar2 , v a r 2 ) = temp ;
253
254 temp = g ( v a r 1 ) ;
255 g ( var1 ) = g ( var2 ) ;
256 g ( v a r 2 ) = temp ;
257 nab = nab − 1 ;
258
259 temp = w a c t ( w i ) ;
260 w a c t ( wi ) = w a c t ( 1 ) ;
261 w a c t ( 1 ) = temp ;
262 j = w act (1 ) ;
263 end
264 w a c t = w a c t ( f i n d ( w a c t ˜= j ) ) ; % bound / general constraint j is
rem ov ed from a c t i v e s e t
265 w non = [ w non j ] ; % bound / general constraint j
appended t o n o n a c t i v e s e t
266
267 if ˜ i s i n f ( b ( pbc ( j ) ) )
268 w non = [ w non pbc ( j ) ] ; % append bound / c o n s t r a i n t pbc ( j )
to nonacti v e set , i f not unbounded
269 end
270
271 f u n c t i o n [ w a c t w non C P x nab G g Q ] = a p p e n d c o n s t r a i n t ( j , C , w a c t , w non , x , P ,
nb , nab , n , G, g , Q, pbc , b ) % j i s i n d e x o f C
272
273 if j < nb+1 % j is a bound and we hav e to
r e o r g a n i z e the v a r i a b l e s
274 v a r 1 = f i n d ( a b s (C ( : , j ) ) ==1) ;
275 v a r 2 = n−nab ;
276
277 temp = C( v ar1 , : ) ;
278 C( v ar1 , : ) = C( v ar2 , : ) ;
279 C( v ar2 , : ) = temp ;
280
281 temp = Q( v ar1 , : ) ;
282 Q( v ar1 , : ) = Q( v ar2 , : ) ;
283 Q( v ar2 , : ) = temp ;
284
285 temp = x ( v a r 1 ) ;
286 x ( var1 ) = x ( var2 ) ;
287 x ( v a r 2 ) = temp ;
288
289 temp = P ( v ar1 , : ) ;
290 P ( v ar1 , : ) = P ( v ar2 , : ) ;
291 P ( v ar2 , : ) = temp ;
292
293 temp = G( v ar1 , v a r 1 ) ;
294 G( v ar1 , v a r 1 ) = G( v ar2 , v a r 2 ) ;
295 G( v ar2 , v a r 2 ) = temp ;
296
297 temp = g ( v a r 1 ) ;
298 g ( var1 ) = g ( var2 ) ;
299 g ( v a r 2 ) = temp ;
300 nab = nab + 1 ;
301 w act = [ j w act ] ; % j ( is a bound ) is appended to
active set
302 else
303 w act = [ w act j ] ; % j ( i s a general c onstr aint )
i s appended t o a c t i v e s e t
304 end
305 w non = w non ( f i n d ( w non ˜= j ) ) ; % bound / g e n e r a l c o n s t r a i n t j
i s rem ov ed fom n o n a c t i v e s e t
306 i f ˜ i s i n f ( b ( pbc ( j ) ) )
307 w non = w non ( f i n d ( w non ˜= pbc ( j ) ) ) ; % rem ov e bound / c o n s t r a i n t pbc ( j
) from n o n a c t i v e s e t , i f n o t unbounded
308 end
309
310 f u n c t i o n f = o b j e c t i v e (G, g , x )
311 f = 0 . 5 ∗ x ’ ∗G∗x + g ’ ∗ x ;
312
313 f u n c t i o n c = c o n s t r a i n t s (A , b , x )
314 c = A’ ∗ x − b ;
315
316 f u n c t i o n l = l a g r a n g i a n (G, g , A , b , x , mu)
317 L = o b j e c t i v e (G, g , A , b , x , mu) − mu ( : ) ’ ∗ c o n s t r a i n t s (G, g , A, b , x , mu) ;
D.2 Inequality Constrained QP’s 129

dual active set method.m

1 function [ x , mu, i n f o , p e r f ] = d u a l a c t i v e s e t m e t h o d (G, g , C , b , w non , pbc , o p t s , t r a c e

)
2
3 % DUAL ACTIVE SET METHOD S o l v i n g an i n e q u a l i t y c o n s t r a i n e d QP o f t h e
4 % form :
5 % min f ( x ) = 0 . 5 ∗ x ’ ∗G∗x + g ∗x
6 % s . t . A∗x >= b ,
7 % by s o l v i n g a s e q u e n c e o f e q u a l i t y c o n s t r a i n e d QP’ s u s i n g t h e d u a l
8 % a c t i v e s e t method . The method u s e s t h e r a n g e s p a c e p r o c e d u r e o r t h e n u l l
9 % s p a c e p r o c e d u r e t o s o l v e t h e KKT s y s t e m . Both t h e r a n g e s p a c e and t h e
10 % n u l l s p ace p r o c e d u r e s has been p rov i d ed with f a c t o r i z a t i o n u p d ates .
11 %
12 % Call
13 % x = d u a l a c t i v e s e t m e t h o d (G, g , A, b , w non , pbc )
14 % x = d u a l a c t i v e s e t m e t h o d (G, g , A, b , w non , pbc , o p t s )
15 % [ x , mu, i n f o , p e r f ] = d u a l a c t i v e s e t m e t h o d ( ... )
16 %
17 % Input parameters
18 % G : The H e s s i a n m a t r i x o f t h e o b j e c t i v e f u n c t i o n , s i z e nxn .
19 % g : The l i n e a r term o f t h e o b j e c t i v e f u n c t i o n , s i z e nx1 .
20 % A : The c o n s t r a i n t m a t r i x h o l d i n g t h e c o n s t r a i n t s , s i z e nxm .
21 % b : The r i g h t −hand s i d e o f t h e c o n s t r a i n t s , s i z e mx1 .
22 % x : S t a r t i n g p o i n t , s i z e nx1 .
23 % w non : L i s t o f i n a c t i v e c o n s t r a i n t s , p o i n t i n g on c o n s t r a i n t s i n A .
24 % pbc : L i s t o f c o r r e s p o n d i n g c o n s t r a i n t s , p o i n t i n g on c o n s t r a i n t s i n
25 % A . Can be empty .
26 % op ts : V ector with 3 el em en ts :
27 % o p t s ( 1 ) = T o l e r a n c e u s e d t o s t a b i l i z e t h e m ethods n u m e r i c a l l y .
28 % I f | v a l u e | <= o p t s ( 1 ) , t h e n v a l u e i s r e g a r d e d a s z e r o .
29 % o p t s ( 2 ) = maximum no . o f i t e r a t i o n s t e p s .
30 % opts ( 3 ) = 1 : Using n u l l space procedure .
31 % 2 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
32 % update .
33 % 3 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
34 % u p d a t e b a s e d on f i x e d and f r e e v a r i a b l e s . Can o n l y
35 % be c a l l e d , i f t h e i n e q u a l i t y c o n s t r a i n e d QP i s
36 % s e t u p on t h e form s e e n i n Q P s o l v e r
37 % 4 : Using range space procedure .
38 % 5 : Using ran ge s p ace p roced u re with f a c t o r i z a t i o n
39 % update .
40 % I f o p t s i s n o t g i v e n o r empty , t h e d e f a u l t o p t s = [ 1 e−8 1 0 0 0 3 ] .
41 %
42 % Output parameters
43 % x : The o p t i m a l s o l u t i o n .
44 % mu : The L a g r a n g e m u l t i p l i e r s a t t h e o p t i m a l s o l u t i o n .
45 % info : Performace i n form ati on , v e c t o r with 3 el em en ts :
46 % inf o (1) = f i n a l v al ues o f the o b j e c t i v e fu n ct i o n .
47 % inf o (2) = no . o f i t e r a t i o n s t e p s .
48 % inf o (3) = 1 : F e a s i b l e s o l u t i o n found .
49 % 2 : No . o f i t e r a t i o n s t e p s e x c e e d e d .
50 % 3 : Probl em i s i n f e a s i b l e .
51 % perf : Performace , s t r u c t h o l d i n g :
52 % perf . x : V a l u e s o f x , s i z e i s nx ( i t +1) .
53 % perf . f : Values o f the o b j e c t i v e f u n c t i o n , s i z e i s 1 x ( i t +1) .
54 % p e r f . mu : V a l u e s o f mu, s i z e i s nx ( i t +1) .
55 % perf . c : V a l u e s o f c ( x ) , s i z e i s mx( i t +1) .
56 % p e r f .Wa : A c t i v e s e t , s i z e i s mx( i t +1) .
57 % p e r f . Wi : I n a c t i v e s e t , s i z e i s mx( i t +1) .
58 %
59 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 .
60 % Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
61 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n .
62 % M. Sc . , IMM, DTU, DK−2800 Lyngby .
63 % Supervisor : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r .
64 % Per Grove Thomsen , P r o f e s s o r .
65 % D ate : 0 7 . J une 2 0 0 7 .
66
67 % i n i t i a l i z e options . . .
68 t o l = opts (1 ) ;
69 it max = opts ( 2) ;
70 method = o p t s ( 3 ) ;
71 [ n ,m] = s i z e (C) ;
72 z = z e r o s (m, 1 ) ;
73
74 % i n i t i a l i z e containers . . .
75 %t r a c e = ( n a r g o u t > 3 ) ;
130 Matlab-code

76 perf = [ ] ;
77 i f trace
78 X = repm at ( z e r o s ( n , 1 ) , 1 , i t m a x ) ;
79 F = repm at ( 0 , 1 , i t m a x ) ;
80 Mu = repm at ( z , 1 , i t m a x ) ;
81 Con = repm at ( z , 1 , i t m a x ) ;
82 W act = repm at ( z , 1 , i t m a x ) ;
83 W non = repm at ( z , 1 , i t m a x ) ;
84 end
85
86 if method == 3 % n u l l s p a c e w i t h FXFR−u p d a t e
87 nb = n ∗ 2 ; % number o f bounds
88 else
89 nb = 0 ;
90 end
91 nab = 0 ; % number of active bouns
92 P = eye ( n ) ;
93
94 x = −G\ g ;
95 mu = z e r o s (m, 1 ) ;
96 w act = [ ] ;
97
98 x0 = x ;
99 f 0 = o b j e c t i v e (G, C , g , b , x , mu) ;
100 mu0 = mu ;
101 c o n 0 = c o n s t r a i n t s (G, C ( : , w non ) , g , b ( w non ) , x , mu) ;
102 w act0 = z ;
103 w non0 = ( 1 : 1 :m) ’ ;
104
105 Q = [ ] ; T = [ ] ; L = [ ] ; R = [ ] ; rem = [ ] ; % both f o r r a n g e− and
n u l l s p a c e u p d a t e and f o r n u l l s p a c e u p d a t e F X F R
106 if method == 4 | | method == 5 % r a n g e s p a c e o r r a n g e s p a c e update
107 c h o l G = c h o l (G) ;
108 end
109 i t t o t = 0;
110 i t = 0;
111 max itr = it max ;
112 stop = 0 ;
113
114 w hi l e ˜ stop
115
116 c = c o n s t r a i n t s (G, C ( : , w non ) , g , b ( w non ) , x , mu) ;
117 i f c >= − t o l ;%−1e−12%−s q r t ( e p s ) % a l l e l e m e n t s must be >= 0
118 % d i s p ( [ ’ / / / / / / i t r : ’ , i n t 2 s t r ( i t +1) , ’ / / / / / / / / / / / / / / ’ ] )
119 % d i s p ( ’ STOP : a l l i n a c t i v e c o n s t r a i n t s >= 0 ’ )
120 stop = 1 ;
121 else
122 % we f i n d t h e most n e g a t i v e v a l u e o f c
123 [ c r , r ] = min ( c ) ;
124 r = w non ( r ) ;
125 end
126
127 it = i t + 1;
128 i f i t >= m a x i t r % no c o n v e r g e n c e
129 d i s p ( [ ’ / / / / / / i t r : ’ , i n t 2 s t r ( i t +1) , ’ / / / / / / / / / / / / / / ’ ] )
130 d i s p ( ’STOP : i t >= m a x i t r ( o u t e r w h i l e l o o p ) ’ )
131 stop = 2;
132 end
133
134 it2 = 0;
135 s t o p 2 = max ( 0 , s t o p ) ;
136 w h i l e ˜ s t o p 2 %c r < −s q r t ( e p s )
137 it2 = it2 + 1;
138
139 i f method == 1
140 [ p , v ] = n u l l s p a c e (G, C ( : , w a c t ) ,−C ( : , r ) ,− z e r o s ( l e n g t h ( w a c t ) , 1 ) ) ;
141 end
142 i f method == 2
143 [ p , v , Q, T , L ] = n u l l s p a c e u p d a t e (G, C ( : , w a c t ) ,−C ( : , r ) , z e r o s ( l e n g t h (
w a c t ) , 1 ) ,Q, T , L , rem ) ;
144 end
145 i f method == 3
146 Cr = C ( : , r ) ;
147 A = C ( : , w a c t ( nab + 1: end ) ) ;
148 % a j u s t C ( : , r ) t o make i t c o r r e s p o n d t o t h e f a c t o r i z a t i o n s
of the
149 % F i x e d v a r i a b l e s ( w h e n e v e r −1 a p p e a r s a t v a r i a b l e i C ( : , r ) i
s houl d change s i g n )
150 i f nab % some bounds a r e i n t h e a c t i v e s e t
151 u i d x = f i n d ( w a c t > nb /2 & w a c t< nb+1) ;
152 v a r = n−nab+u i d x ;
153 Cr ( v a r ) = −Cr ( v a r ) ;
154 A ( v ar , : ) = −A ( v ar , : ) ;
155 end
156 [ p , v , Q, T , L ] = n u l l s p a c e u p d a t e F R F X (Q, T , L , G, A ,−Cr , z e r o s ( l e n g t h (
w a c t ) , 1 ) , nab , rem−nab ) ;
157 end
D.2 Inequality Constrained QP’s 131

158 if method == 4
159 [ p , v ] = r a n g e s p a c e ( c h o l G , C ( : , w a c t ) ,−C ( : , r ) ,− z e r o s ( l e n g t h ( w a c t )
,1) ) ;
160 end
161 i f method == 5
162 [ p , v , Q, R ] = r a n g e s p a c e u p d a t e ( c h o l G , C ( : , w a c t ) ,−C ( : , r ) , z e r o s (
l e n g t h ( w a c t ) , 1 ) ,Q, R, rem ) ;
163 end
164
165 if isempty ( v )
166 v = [];
167 end
168
169 arp = C ( : , r ) ’∗p ;
170 i f a b s ( a r p ) <= t o l % l i n e a r d e p e n d e n c y
171 i f v >= 0 % s o l u t i o n d o e s n o t e x i s t
172 d i s p ( [ ’ / / / / / / i t r : ’ , i n t 2 s t r ( i t +1) , ’ / / / / / / / / / / / / / / ’ ] )
173 d i s p ( ’STOP : v >= 0 , PROBLEM I S INFEASIBLE ! ! ! ’ )
174 stop = 3 ;
175 stop2 = stop ;
176 else
177 t = inf ;
178 f or k = 1: length (v)
179 i f v(k) < 0
180 temp = −mu( w a c t ( k ) ) / v ( k ) ;
181 i f temp < t
182 t = temp ;
183 rem = k ;
184 end
185 end
186 end
187 mu( w a c t ) = mu( w a c t ) + t ∗v ;
188 mu( r ) = mu( r ) + t ;
189 % rem ov e l i n e a r d e p e n d e n t c o n s t r a i n t from A
190 [ w a c t w non C P x nab G g ] = r e m o v e c o n s t r a i n t ( rem , C , w act ,
w non , x , P , nb , nab , n , G, g , pbc , b ) ; % rem i s i n d e x o f w act
191 end
192 else
193 % s t e p s i z e in dual space
194 t1 = i n f ;
195 for k = 1: length (v)
196 i f v(k) < 0
197 temp = −mu( w a c t ( k ) ) / v ( k ) ;
198 i f temp < t 1
199 t 1 = temp ;
200 rem = k ;
201 end
202 end
203 end
204 % s t e p s i z e i n primal space
205 t 2 = −c o n s t r a i n t s (G, C ( : , r ) , g , b ( r ) , x , mu) / a r p ;
206 i f t 2 <= t 1
207 x = x + t 2 ∗p ;
208 mu( w a c t ) = mu( w a c t ) + t 2 ∗v ;
209 mu( r ) = mu( r ) + t 2 ;
210 % append c o n s t r a i n t t o a c t i v e s e t
211 [ w a c t w non C P x nab G g Q ] = a p p e n d c o n s t r a i n t ( r , C , w act ,
w non , x , P , nb , nab , n , G, g , Q, pbc , b ) ; % r i s i n d e x o f C
212 else
213 x = x + t 1 ∗p ;
214 mu( w a c t ) = mu( w a c t ) + t 1 ∗v ;
215 mu( r ) = mu( r ) + t 1 ;
216 % rem ov e c o n s t r a i n t from a c t i v e s e t
217 [ w a c t w non C P x nab G g ] = r e m o v e c o n s t r a i n t ( rem , C , w act ,
w non , x , P , nb , nab , n , G, g , pbc , b ) ; % rem i s i n d e x o f w act
218 end
219 end
220 c r = c o n s t r a i n t s (G, C ( : , r ) , g , b ( r ) , x , mu) ;
221 i f c r > −t o l
222 s t o p 2 = 1 ; % l e a v e t h e i n n e r w h i l e−l o o p but d o e s n t s t o p t h e
algorithm
223 end
224
225 if i t 2 >= m a x i t r % no c o n v e r g e n c e ( t e r m i n a t e t h e a l g o r i t h m )
226 d i s p ( [ ’ / / / / / / i t r : ’ , i n t 2 s t r ( i t +1) , ’ / / / / / / / / / / / / / / ’ ] )
227 d i s p ( ’STOP : i t >= m a x i t r ( i n n e r w h i l e l o o p ) ’ )
228 stop = 2 ;
229 stop2 = stop ;
230 end
231
232 % c o l l e c t i n g output i n c o n t a i n e r s . . .
233 i f trace
234 i f nb % method 3 i s u s e d
235 X( : , i t ) = P’ ∗ x ;
236 else
237 X( : , i t ) = x ;
238 end
132 Matlab-code

239 F ( i t ) = o b j e c t i v e (G, C , g , b , x , mu) ;

240 Mu ( : , i t ) = mu ;
241 Con ( w non , i t ) = c o n s t r a i n t s (G, C ( : , w non ) , g , b ( w non ) , x , mu) ;
242 W act ( w a c t , i t ) = w a c t ;
243 W non ( w non , i t ) = w non ;
244 end
245 end % w h i l e
246 it tot = it tot + it2 ;
247 end % w h i l e
248 it tot = it tot + it ;
249 i f nb % method 3 i s u s e d
250 x = P’ ∗ x ;
251 % f i g u r e ; s p y (C ( : , w a c t ) ) , p a u s e
252 end
253
254 % building info . . .
255 i n f o = [ o b j e c t i v e (G, C , g , b , x , mu) i t t o t s t o p ] ;
256 % building perf . . .
257 i f trace
258 X = X ( : , 1 : i t ) ; X = [ x0 X ] ;
259 F = (1: i t ) ; F = [ f0 F ] ;
260 Mu = Mu ( : , 1 : i t ) ; Mu = [ mu0 Mu ] ;
261 Con = Con ( : , 1 : i t ) ; Con = [ c o n 0 Con ] ;
262 W act = W act ( : , 1 : i t ) ; W act = [ w a c t 0 W act ] ;
263 W non = W non ( : , 1 : i t ) ; W non = [ w non0 W non ] ;
264 p e r f = s t r u c t ( ’ x ’ , {X} , ’ f ’ , { F} , ’mu ’ , {Mu} , ’ c ’ , { Con } , ’Wa ’ , { W act } , ’ Wi ’ , { W non
}) ;
265 end
266
267 f u n c t i o n c = c o n s t r a i n t s (G, C , g , b , x , mu)
268 c = C’ ∗ x − b ;
269
270 f u n c t i o n [ w a c t w non C P x nab G g ] = r e m o v e c o n s t r a i n t ( wi , C , w a c t , w non , x , P ,
nb , nab , n , G, g , pbc , b ) % w i i s i n d e x o f w a c t
271 j = w a c t ( wi ) ;
272 i f j < nb+1 % j i s a bound and we hav e t o
r e o r g a n i z e the v a r i a b l e s
273 v a r 1 = n−nab + 1;
274 v a r 2 = n−nab+w i ;
275
276 temp = C( v ar1 , : ) ;
277 C( v ar1 , : ) = C( v ar2 , : ) ;
278 C( v ar2 , : ) = temp ;
279
280 temp = x ( v a r 1 ) ;
281 x ( var1 ) = x ( var2 ) ;
282 x ( v a r 2 ) = temp ;
283
284 temp = P ( v ar1 , : ) ;
285 P ( v ar1 , : ) = P ( v ar2 , : ) ;
286 P ( v ar2 , : ) = temp ;
287
288 temp = G( v ar1 , v a r 1 ) ;
289 G( v ar1 , v a r 1 ) = G( v ar2 , v a r 2 ) ;
290 G( v ar2 , v a r 2 ) = temp ;
291
292 temp = g ( v a r 1 ) ;
293 g ( var1 ) = g ( var2 ) ;
294 g ( v a r 2 ) = temp ;
295 nab = nab − 1 ;
296
297 temp = w a c t ( w i ) ;
298 w a c t ( wi ) = w a c t ( 1 ) ;
299 w a c t ( 1 ) = temp ;
300 j = w act (1 ) ;
301 end
302 w a c t = w a c t ( f i n d ( w a c t ˜= j ) ) ; % bound / general constraint j is
rem ov ed from a c t i v e s e t
303 w non = [ w non j ] ; % bound / general constraint j
appended t o n o n a c t i v e s e t
304
305 if ˜ i s e m p t y ( pbc )
306 i f ˜ i s i n f ( b ( pbc ( j ) ) )
307 w non = [ w non pbc ( j ) ] ; % append bound / c o n s t r a i n t pbc
( j ) to nonacti v e set , if not unbounded
308 end
309 end
310
311 f u n c t i o n [ w a c t w non C P x nab G g Q ] = a p p e n d c o n s t r a i n t ( j , C , w a c t , w non , x , P ,
nb , nab , n , G, g , Q, pbc , b ) % j i s i n d e x o f C
312 i f j < nb+1 % j i s a bound and we hav e t o
r e o r g a n i z e the v a r i a b l e s
313 v a r 1 = f i n d ( a b s (C ( : , j ) ) ==1) ;
314 v a r 2 = n−nab ;
315
316 temp = C( v ar1 , : ) ;
317 C( v ar1 , : ) = C( v ar2 , : ) ;
D.2 Inequality Constrained QP’s 133

318 C( v ar2 , : ) = temp ;

319
320 temp = Q( v ar1 , : ) ;
321 Q( v ar1 , : ) = Q( v ar2 , : ) ;
322 Q( v ar2 , : ) = temp ;
323
324 temp = x ( v a r 1 ) ;
325 x ( var1 ) = x ( var2 ) ;
326 x ( v a r 2 ) = temp ;
327
328 temp = P( v ar1 , : ) ;
329 P ( v ar1 , : ) = P ( v ar2 , : ) ;
330 P ( v ar2 , : ) = temp ;
331
332 temp = G( v ar1 , v a r 1 ) ;
333 G( v ar1 , v a r 1 ) = G( v ar2 , v a r 2 ) ;
334 G( v ar2 , v a r 2 ) = temp ;
335
336 temp = g ( v a r 1 ) ;
337 g ( var1 ) = g ( var2 ) ;
338 g ( v a r 2 ) = temp ;
339 nab = nab + 1 ;
340 w act = [ j w act ] ; % j ( is a bound ) is appended to
active set
341 else
342 w act = [ w act j ] ; % j ( is a general constraint )
i s appended t o a c t i v e s e t
343 end
344 w non = w non ( f i n d ( w non ˜= j ) ) ; % bound / general constraint j
i s rem ov ed fom n o n a c t i v e s e t
345
346 if ˜ i s e m p t y ( pbc )
347 i f ˜ i s i n f ( b ( pbc ( j ) ) )
348 w non = w non ( f i n d ( w non ˜= pbc ( j ) ) ) ; % rem ov e bound / c o n s t r a i n t
pbc ( j ) from n o n a c t i v e s e t , i f n o t unbounded
349 end
350 end
351
352 f u n c t i o n f = o b j e c t i v e (G, C , g , b , x , mu)
353 f = 0 . 5 ∗ x ’ ∗G∗x + g ’ ∗ x ;

QP solver.m

1 function [ x , i n f o , p e r f ] = Q P s o l v e r (H , g , l , u , A , b l , bu , x , o p t s )
2
3 % QP SOLVER S o l v i n g an i n e q u a l i t y c o n s t r a i n e d QP o f t h e form :
4 % min f ( x ) = 0 . 5 ∗ x ’ ∗H∗x + g ∗x
5 % s.t. l <= x <= u
6 % b l <= A∗x <= bu ,
7 % u s i n g t h e p r i m a l a c t i v e s e t method o r t h e d u a l a c t i v e s e t method . The
8 % a c t i v e s e t m ethods u s e s t h e r a n g e s p a c e p r o c e d u r e o r t h e n u l l s p a c e
9 % p r o c e d u r e t o s o l v e t h e KKT s y s t e m . Both t h e r a n g e s p a c e and t h e n u l l
10 % s p ace p r o c e d u r e s has been p rov i d ed with f a c t o r i z a t i o n u p dates . E q u a l i t y
11 % c o n s t r a i n t s a r e d e f i n e d a s l = u and b l = bu r e s p e c t i v e l y .
12 %
13 % Call
14 % x = Q P s o l v e r (H, g , l , u , A, bl , bu )
15 % x = Q P s o l v e r (H, g , l , u , A, bl , bu , x , opts )
16 % [ x , info , p e r f ] = QP solver ( ... )
17 %
18 % Input parameters
19 % H : The H e s s i a n m a t r i x o f t h e o b j e c t i v e f u n c t i o n .
20 % g : The l i n e a r term o f t h e o b j e c t i v e f u n c t i o n .
21 % l : Lower l i m i t s o f bounds . S e t a s I n f , i f unbounded .
22 % u : Upper l i m i t s o f bounds . S e t a s −I n f , i f unbounded .
23 % A : The c o n s t r a i n t m a t r i x h o l d i n g t h e g e n e r a l c o n s t r a i n t s a s row s .
24 % bl : Lower l i m i t s o f g e n e r a l c o n s t r a i n t s . S e t a s I n f , i f unbounded .
25 % bu : Upper l i m i t s o f g e n e r a l c o n s t r a i n t s . S e t a s −I n f , i f unbounded .
26 % x : S t a r t i n g p o i n t . I f x i s n o t g i v e n o r empty , t h e n t h e d u a l a c t i v e
27 % s e t method i s us ed , o t h e r w i s e t h e p r i m a l a c t i v e s e t method i s
28 % used .
29 % op ts : V ector with 3 el em en ts :
30 % o p t s ( 1 ) = T o l e r a n c e u s e d t o s t a b i l i z e t h e m ethods n u m e r i c a l l y .
31 % I f | v a l u e | <= o p t s ( 1 ) , t h e n v a l u e i s r e g a r d e d a s z e r o .
32 % o p t s ( 2 ) = maximum no . o f i t e r a t i o n s t e p s .
33 % P r i m a l a c t i v e s e t method :
34 % opts ( 3 ) = 1 : Using n u l l space procedure .
35 % 2 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
36 % update .
37 % 3 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
134 Matlab-code

38 % u p d a t e b a s e d on f i x e d and f r e e v a r i a b l e s .
39 % I f op ts ( 3 ) > 3 , then op ts ( 3 ) i s s e t to 3 a u t o m a t i c a l l y .
40 % D ual a c t i v e s e t method :
41 % opts ( 3 ) = 1 : Using n u l l space procedure .
42 % 2 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
43 % update .
44 % 3 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
45 % u p d a t e b a s e d on f i x e d and f r e e v a r i a b l e s .
46 % 4 : Using range space procedure .
47 % 5 : Using ran ge s p ace p roced u re with f a c t o r i z a t i o n
48 % update .
49 % I f o p t s i s n o t g i v e n o r empty , t h e d e f a u l t o p t s = [ 1 e−8 1 0 0 0 3 ] .
50 %
51 % Output p a r a m e t e r s
52 % x : The o p t i m a l s o l u t i o n .
53 % i n f o : Performace i n form ati on , v e c t o r with 3 el em en ts :
54 % inf o (1) = f i n a l v al ue s o f the o b j e c t i v e f un ct i on .
55 % inf o (2) = no . o f i t e r a t i o n s t e p s .
56 % P r i m a l a c t i v e s e t method :
57 % inf o (3) = 1 : F e a s i b l e s o l u t i o n found .
58 % 2 : No . o f i t e r a t i o n s t e p s e x c e e d e d .
59 % D ual a c t i v e s e t method :
60 % inf o (3) = 1 : F e a s i b l e s o l u t i o n found .
61 % 2 : No . o f i t e r a t i o n s t e p s e x c e e d e d .
62 % 3 : Probl em i s i n f e a s i b l e .
63 % p e r f : Performace , s t r u c t h o l d i n g :
64 % perf . x : V a l u e s o f x , s i z e i s nx ( i t +1) .
65 % perf . f : V a l u e s o f t h e o b j e c t i v e f u n c t i o n , s i z e i s 1x ( i t +1) .
66 % p e r f . mu : V a l u e s o f mu, s i z e i s nx ( i t +1) .
67 % perf . c : V a l u e s o f c ( x ) , s i z e i s ( n+n+m+m) x ( i t +1) .
68 % p e r f .Wa : A c t i v e s e t , s i z e i s ( n+n+m+m) x ( i t +1) .
69 % p e r f . Wi : I n a c t i v e s e t , s i z e i s ( n+n+m+m) x ( i t +1) .
70 % S i z e ( n+n+m+m) x ( i t +1) i s r e f e r i n g t o i n d i c e s i l = 1 : n , i u = ( n+1) : 2
71 % n , i b l = ( 2 n+1) : ( 2 n+m) and i b u = ( 2 n+m+1) : ( 2 n+2m) .
72 %
73 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 .
74 % Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
75 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n .
76 % M. Sc . , IMM, DTU, DK−2800 Lyngby .
77 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r .
78 % Per Grove Thomsen , P r o f e s s o r .
79 % D ate : 0 7 . J une 2 0 0 7 .
80
81 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
82 % Tune i n p u t and g a t h e r i n f o r m a t i o n %
83 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
84 % Tune . . .
85 l = l (:) ; u = u(:) ;
86 b l = b l ( : ) ; bu = bu ( : ) ;
87 g = g(:) ;
88 % Gather . . .
89 [ m, n ] = s i z e (A) ;
90
91 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
92 % Set o ption s %
93 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
94 i f n argi n < 9 | isempty ( op ts )
95 t o l = 1 e −8;
96 it max = 1000;
97 method = 3 ;
98 o p t s = [ t o l i t m a x method ] ;
99 else
100 opts = opts ( : ) ’ ;
101 end
102 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
103 % Check n a r g i n / n a r g o u t %
104 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
105 e r r o r ( nargchk ( 7 , 9 , n argi n ) )
106 e r r o r ( nargoutchk ( 1 , 3 , nargout ) )
107 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
108 % Check i n p u t / o u t p u t %
109 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
110 % Check H . . .
111 s i z e H = s i z e (H) ;
112 i f s i z e H ( 1 ) ˜= n | s i z e H ( 2 ) ˜= n
113 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o H must be o f s i z e ’ ,
i n t 2 s t r (n) , ’x ’ , in t 2 s t r (n) , ’ . ’ ] )
114 end
115 Hdiff = H − H’ ;
116 i f norm ( H d i f f ( : ) , i n f ) > e p s ∗norm (H ( : ) , i n f ) % r e l a t i v e c h e c k o f b i g g e s t a b s o l u t e
value in Hd if f
117 e r r o r ( ’H must be s y m m e t r i c . ’ )
118 end
119 [ dummy , p ] = c h o l (H) ;
120 if p
121 e r r o r ( ’H must be p o s i t i v e d e f i n i t e . ’ )
122 end
D.2 Inequality Constrained QP’s 135

123 % Check g . . .
124 sizeg = size (g) ;
125 i f s i z e g ( 1 ) ˜= n | s i z e g ( 2 ) ˜= 1
126 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o g must be a v e c t o r
o f ’ , i n t 2 s t r (n ) , ’ elements . ’ ] )
127 end
128 % Check l and u . . .
129 %l ( 4 0 , 1 ) = i n f ; % ? ? ?
130 sizel = size ( l ) ;
131 i f s i z e l ( 1 ) ˜= n | s i z e l ( 2 ) ˜= 1
132 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o l must be a v e c t o r
o f ’ , i n t 2 s t r (n ) , ’ elements . ’ ] )
133 end
134 siz eu = s i z e (u) ;
135 i f s i z e u ( 1 ) ˜= n | s i z e u ( 2 ) ˜= 1
136 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o u must be a v e c t o r
o f ’ , i n t 2 s t r (n ) , ’ elements . ’ ] )
137 end
138 for i = 1:n
139 i f l ( i ,1) > u( i ,1)
140 e r r o r ( [ ’ l ( ’ , i n t 2 s t r ( i ) , ’ ) must be s m a l l e r t h a n o r e q u a l t o u ( ’ , i n t 2 s t r (
i ) , ’ ) . ’ ])
141 end
142 end
143 % Check b l and bu . . .
144 s i z e bl = s iz e ( bl ) ;
145 i f s i z e b l ( 1 ) ˜= n | s i z e b l ( 2 ) ˜= 1
146 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o b l must be a v e c t o r
o f ’ , i n t 2 s t r (m) , ’ e l e m e n t s . ’ ] )
147 end
148 s i z e b u = s i z e ( bu ) ;
149 i f s i z e b u ( 1 ) ˜= n | s i z e b u ( 2 ) ˜= 1
150 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o bu must be a v e c t o r
o f ’ , i n t 2 s t r (m) , ’ e l e m e n t s . ’ ] )
151 end
152 f o r i = 1 :m
153 i f b l ( i ) > bu ( i )
154 e r r o r ( [ ’ b l ( ’ , i n t 2 s t r ( i ) , ’ ) must be s m a l l e r t h a n o r e q u a l t o bu ( ’ ,
int2str ( i ) , ’ ) . ’ ])
155 end
156 end
157 % Check x . . .
158 i f n argi n > 7 & ˜ isempty ( x )
159 %o p t s ( 1 ) = 1 e −20; % ? ? ?
160 feasible = 1;
161 sizex = size (x) ;
162 i f s i z e x ( 1 ) ˜= n | s i z e x ( 2 ) ˜= 1
163 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o x must be a
v ec to r o f ’ , i n t 2 s t r (n ) , ’ elements . ’ ] )
164 end
165 i l = f i n d ( x − l < −o p t s ( 1 ) ) ; i u = f i n d ( x − u > o p t s ( 1 ) ) ;
166 i b l = f i n d (A∗x − b l < −o p t s ( 1 ) ) ; i b u = f i n d (A∗x − bu > o p t s ( 1 ) ) ;
167 i f ˜ isempty ( i l )
168 d i s p ( [ ’ F o l l o w i n g bound ( s ) v i o l a t e d , b e c a u s e x − l < ’ , num 2s tr (− o p t s ( 1 ) )
, ’ : ’ ])
169 f p r i n t f ( [ ’ \b ’ , i n t 2 s t r ( i l ’ ) , ’ . \ n ’ ] )
170 feas ible = 0;
171 end
172 i f ˜ isempty ( i u )
173 d i s p ( [ ’ F o l l o w i n g bound ( s ) v i o l a t e d , b e c a u s e x − u > ’ , num 2s tr ( o p t s ( 1 ) ) ,
’ : ’ ])
174 f p r i n t f ( [ ’ \b ’ , i n t 2 s t r ( i u ’ ) , ’ . \ n ’ ] )
175 feas ible = 0;
176 end
177 i f ˜ isempty ( i b l )
178 d i s p ( [ ’ F o l l o w i n g g e n e r a l c o n s t r a i n t ( s ) v i o l a t e d , b e c a u s e A∗x − b l < ’ ,
num 2s tr (− o p t s ( 1 ) ) , ’ : ’ ] )
179 f p r i n t f ( [ ’ \b ’ , i n t 2 s t r ( i b l ’ ) , ’ . \ n ’ ] )
180 feas ible = 0;
181 end
182 i f ˜ isempty ( i b u )
183 d i s p ( [ ’ F o l l o w i n g g e n e r a l c o n s t r a i n t ( s ) v i o l a t e d , b e c a u s e A∗x − bu > ’ ,
num 2s tr ( o p t s ( 1 ) ) , ’ : ’ ] )
184 f p r i n t f ( [ ’ \b ’ , i n t 2 s t r ( i b u ’ ) , ’ . \ n ’ ] )
185 feas ible = 0;
186 end
187 if ˜feasible
188 e r r o r ( ’ S t a r t i n g p o i n t f o r p r i m a l a c t i v e s e t method i s n o t f e a s i b l e . ’ )
189 end
190 end
191 % Check o p t s . . .
192 i f l e n g t h ( o p t s ) ˜= 3
193 e r r o r ( ’ O p t i o n s must be a v e c t o r o f 3 e l e m e n t s . ’ )
194 end
195 i = 1;
196 i f ˜ i s r e a l ( opts ( i ) ) | i s i n f ( opts ( i ) ) | i s nan ( opts ( i ) ) | opts ( i ) < 0
197 e r r o r ( ’ o p t s ( 1 ) must be p o s i t i v e . ’ )
136 Matlab-code

198 end
199 i = 2;
200 i f ˜ i s r e a l ( opts ( i ) ) | i s i n f ( opts ( i ) ) | i s nan ( opts ( i ) ) | opts ( i ) < 0 | mod ( o p t s (
i ) ,1)
201 e r r o r ( ’ o p t s ( 2 ) must be a positive integer . ’ )
202 end
203 i = 3;
204 i f ˜ i s r e a l ( opts ( i ) ) | i s i n f ( opts ( i ) ) | i s nan ( opts ( i ) ) | opts ( i ) < 1 | 5 < opts (
i ) | mod ( o p t s ( i ) , 1 )
205 e r r o r ( ’ o p t s ( 3 ) must be an integer in range 1 <= v a l u e <= 5 . ’ )
206 end
207
208 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
209 % Initialize %
210 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
211 I = eye ( n ) ;
212 At = A ’ ;
213
214 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
215 % O r g a n i z e bounds and c o n s t r a i n t s %
216 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
217 % C o n v e r t i n p u t s t r u c t u r e l <= I ∗x <= u and b l <= A∗x <= bu t o C∗x >= b , w here
C = [ I −I A −A ] = [ B A ] (A = [ A −A ] ) and b = [ l −u b l −bu ] . . .
218 B = [ I −I ] ; % l <= I ∗x <= u −−> I ∗x >= l & −I ∗x >= −u
219 A = [ At −At ] ; % b l <= A∗x <= bu −−> A∗x >= b l & −A∗x >= −bu
220 C = [B A] ;
221 b = [ l ; −u ; b l ; −bu ] ;
222
223 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
224 % B u i l d i n a c t i v e s e t and c o r r e s p o n d i n g c o n s t r a i n t s %
225 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
226 % I n i t i a l i z e inactive set . . .
227 w non = 1 : 1 : 2 ∗ ( n+m) ;
228 % Remove unbounded c o n s t r a i n t s from i n a c t i v e s e t . . .
229 w non = w non ( f i n d ( ˜ i s i n f ( b ) ) ) ;
230 % Ind ic es of corresponding c o nst ra int s . . .
231 c c = [ ( n+1) : 1 : ( 2 ∗ n ) 1 : 1 : n ( 2 ∗ n+m+1) : 1 : 2 ∗ ( n+m) 2∗ n + 1 : 1 : 2 ∗ n+m ] ; % w non = [ i l
i u i b l i b u ] −> c c = [ i u i l i b u i b l ]
232
233 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
234 % Startup i n f o %
235 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
236 % Disregarded c o n s t r a i n t s . . .
237 %l ( 4 0 , 1 ) = − i n f ; % ? ? ?
238 i l = find ( i s i n f ( l ) ) ; i u = find ( i s i n f (u) ) ;
239 i b l = f i n d ( i s i n f ( b l ) ) ; i b u = f i n d ( i s i n f ( bu ) ) ;
240 i f ˜ isempty ( i l )
241 d i s p ( ’ F o l l o w i n g c o n s t r a i n t ( s ) d i s r e g a r d e d , b e c a u s e l i s unbounded : ’ )
242 disp ( [ ’ i = [ ’ , int 2s tr ( i l ’ ) , ’ ] ’ ] )
243 end
244 i f ˜ isempty ( i u )
245 d i s p ( ’ F o l l o w i n g c o n s t r a i n t ( s ) d i s r e g a r d e d , b e c a u s e u i s unbounded : ’ )
246 disp ( [ ’ i = [ ’ , int 2s tr ( i u ’ ) , ’ ] ’ ] )
247 end
248 i f ˜ isempty ( i b l )
249 d i s p ( ’ F o l l o w i n g c o n s t r a i n t ( s ) d i s r e g a r d e d , b e c a u s e b l i s unbounded : ’ )
250 disp ( [ ’ i = [ ’ , int 2s tr ( i bl ’ ) , ’ ] ’ ] )
251 end
252 i f ˜ isempty ( i b u )
253 d i s p ( ’ F o l l o w i n g c o n s t r a i n t ( s ) d i s r e g a r d e d , b e c a u s e bu i s unbounded : ’ )
254 disp ( [ ’ i = [ ’ , i n t 2 s t r ( i bu ’ ) , ’ ] ’ ] )
255 end
256
257 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
258 % C a l l p r i m a l a c t i v e s e t o r d u a l a c t i v e s e t method %
259 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
260 t r ac e = ( nargout > 2) ; % b ui l d i n g p e r f
261 i f n argi n < 8 | isempty ( x )
262 d i s p ( ’ C a l l i n g d u a l a c t i v e s e t method . ’ )
263 [ x , mu, i n f o , p e r f ] = d u a l a c t i v e s e t m e t h o d (H , g , C , b , w non , cc , o p t s , t r a c e ) ;
264 else
265 disp ( ’ Starting point i s f e a s i b l e . ’ )
266 d i s p ( ’ C a l l i n g p r i m a l a c t i v e s e t method . ’ )
267 i f opts (3 ) > 3
268 opts (3 ) = 3;
269 end
270 [ x , mu, i n f o , p e r f ] = p r i m a l a c t i v e s e t m e t h o d (H , g , C , b , x , w non , cc , o p t s , t r a c e ) ;
271 end
272 % Display i n f o . . .
273 i f i n f o ( 1 ) == 2
274 d i s p ( ’ No s o l u t i o n found , maximum number o f i t e r a t i o n s t e p s e x c e e d e d . ’ )
275 end
276 i f i n f o ( 1 ) == 3
277 d i s p ( ’ No s o l u t i o n found , p r o b l e m i s u n f e a s i b l e . ’ )
278 end
279 d i s p ( ’ QPsolver term i nated . ’ )
D.2 Inequality Constrained QP’s 137

LP solver.m

1 function [ x , f , A , b , Aeq , beq , l , u ] = L P s o l v e r ( l , u , A , b l , bu )

2
3 % LP SOLVER F i n d i n g a f e a s i b l e p o i n t with respect to the constraints of an
4 % i n e q u a l i t y c o n s t r a i n e d QP o f t h e form :
5 %
6 % min f ( x ) = 0 . 5 ∗ x ’ ∗H∗x + g ∗x
7 % s.t. l <= x <= u
8 % b l <= A∗x <= bu ,
9 %
10 % u s i n g t h e Matl ab f u n c t i o n l i n p r o g . Equality constraints are defined as
11 % l = u and b l = bu r e s p e c t i v e l y .
12 %
13 % Call
14 % x = L P s o l v e r ( l , u , A, b l , bu )
15 % x = L P s o l v e r ( l , u , A, b l , bu , o p t s )
16 % [ x , f , A, b , Aeq , beq , l , u ] = LP solver ( ... )
17 %
18 % Input parameters
19 % l : Lower l i m i t s o f bounds . S e t a s I n f , i f unbounded .
20 % u : Upper l i m i t s o f bounds . S e t a s −I n f , i f unbounded .
21 % A : The c o n s t r a i n t m a t r i x h o l d i n g t h e g e n e r a l c o n s t r a i n t s a s row s .
22 % bl : Lower l i m i t s o f g e n e r a l c o n s t r a i n t s . S e t a s I n f , i f unbounded .
23 % bu : Upper l i m i t s o f g e n e r a l c o n s t r a i n t s . S e t a s −I n f , i f unbounded .
24 % op ts : V ector with 2 el em en ts :
25 % opts ( 1) = Tolerance de c i di ng i f c o n s t r a i n t s are e q u a l i t i e s .
26 % I f | bu − b l | <= o p t s ( 1 ) , t h e n c o n s t r a i n t i s r e g a r d e d
27 % a s an e q u a l i t y .
28 % o p t s ( 2 ) = ps eudo−i n f i n i t y , can be u s e d t o r e p l a c e (+−) I n f w i t h a
29 % r e a l v a l u e r e g a r d i n g unbounded v a r i a b l e s and g e n e r a l
30 % constraints .
31 % I f o p t s i s n o t g i v e n o r empty , t h e d e f a u l t o p t s = [ 0 i n f ] .
32 %
33 % Output p a r a m e t e r s
34 % x : Feasible point .
35 % f , A , b , Aeq , beq , l , u : Output s t r u c t u r e d for further use in linprog .
36 %
37 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 .
38 % Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
39 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n .
40 % M. Sc . , IMM, DTU, DK−2800 Lyngby .
41 % Supervisor : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r .
42 % Per Grove Thomsen , P r o f e s s o r .
43 % D ate : 0 7 . J une 2 0 0 7 .
44
45 f i n d e q u a l i t y c o n s t r a i n t s = 1 ; % s e e i n i t i a l i z a t i o n o f c o n s t r a i n t s below
46 e q u a l i t y t o l = 1 e −8; % s e e i n i t i a l i z a t i o n o f c o n s t r a i n t s b e l o w
47 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
48 % Tune i n p u t and g a t h e r i n f o r m a t i o n %
49 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
50 % Tune . . .
51 l = l (:) ; u = u(:) ;
52 b l = b l ( : ) ; bu = bu ( : ) ;
53 % Gather . . .
54 [ m, n ] = s i z e (A) ;
55
56 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
57 % Set op tion s %
58 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
59 i f n argi n < 6 | isempty ( op ts )
60 e q u a l i t y t o l = 1 e −8;
61 p s e u d o i n f = 1 e8 ;
62 else
63 opts = opts ( : ) ’ ;
64 % Check o p t s . . .
65 i f l e n g t h ( o p t s ) ˜= 2
66 e r r o r ( ’ O p t i o n s must be a v e c t o r o f 2 e l e m e n t s . ’ )
67 end
68 i = 1;
69 i f ˜ i s r e a l ( opts ( i ) ) | i s i n f ( opts ( i ) ) | i s nan ( opts ( i ) ) | opts ( i ) < 0
70 e r r o r ( ’ o p t s ( 1 ) must be p o s i t i v e . ’ )
71 end
72 i = 2;
73 i f ˜ i s r e a l ( opts ( i ) ) | i s i n f ( opts ( i ) ) | i s nan ( opts ( i ) ) | opts ( i ) < 0
74 e r r o r ( ’ o p t s ( 2 ) must be p o s i t i v e . ’ )
75 end
76 e q u a l i t y t o l = opts ( 1) ;
77 ps eudo inf = opts (2 ) ;
78 end
79 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
80 % Check n a r g i n / n a r g o u t %
81 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
82 e r r o r ( nargchk ( 5 , 6 , n argi n ) )
83 e r r o r ( nargoutchk ( 1 , 8 , nargout ) )
138 Matlab-code

84 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
85 % Check i n p u t %
86 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
87 % Check l and u . . .
88 %l ( 4 0 , 1 ) = i n f ; % ? ? ?
89 si zel = size ( l ) ;
90 i f s i z e l ( 1 ) ˜= n | s i z e l ( 2 ) ˜= 1
91 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o l must be a v e c t o r
of ’ , i n t 2 s t r (n ) , ’ elements . ’ ] )
92 end
93 siz eu = s i z e (u) ;
94 i f s i z e u ( 1 ) ˜= n | s i z e u ( 2 ) ˜= 1
95 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o u must be a v e c t o r
of ’ , i n t 2 s t r (n ) , ’ elements . ’ ] )
96 end
97 for i = 1: n
98 i f l ( i ,1) > u( i ,1)
99 e r r o r ( [ ’ l ( ’ , i n t 2 s t r ( i ) , ’ ) must be s m a l l e r t h a n o r e q u a l t o u ( ’ , i n t 2 s t r (
i ) , ’ ) . ’ ])
100 end
101 end
102 % Check b l and bu . . .
103 s i z e bl = s i ze ( bl ) ;
104 i f s i z e b l ( 1 ) ˜= n | s i z e b l ( 2 ) ˜= 1
105 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o b l must be a v e c t o r
o f ’ , i n t 2 s t r (m) , ’ e l e m e n t s . ’ ] )
106 end
107 s i z e b u = s i z e ( bu ) ;
108 i f s i z e b u ( 1 ) ˜= n | s i z e b u ( 2 ) ˜= 1
109 e r r o r ( [ ’ S i z e o f A i s ’ , i n t 2 s t r (m) , ’ x ’ , i n t 2 s t r ( n ) , ’ , s o bu must be a v e c t o r
o f ’ , i n t 2 s t r (m) , ’ e l e m e n t s . ’ ] )
110 end
111 f o r i = 1 :m
112 i f b l ( i ) > bu ( i )
113 e r r o r ( [ ’ b l ( ’ , i n t 2 s t r ( i ) , ’ ) must be s m a l l e r t h a n o r e q u a l t o bu ( ’ ,
int2str ( i ) , ’ ) . ’ ])
114 end
115 end
116
117 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
118 % I n i t i a l i z e input f o r l in p r o g %
119 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
120 % R e p l a c e +/− i n f w i t h p s e u d o i n f ( l i n p r o g r e q u i r e m e n t ) . . .
121 i f p s e u d o i n f ˜= i n f
122 l ( f i n d ( l == − i n f ) ) = −p s e u d o i n f ;
123 u ( f i n d ( u == i n f ) ) = p s e u d o i n f ;
124 b l ( f i n d ( b l == − i n f ) ) = −p s e u d o i n f ;
125 bu ( f i n d ( bu == i n f ) ) = p s e u d o i n f ;
126 end
127 % O b j e c t i v e f u n c t i o n f d e f i n e d a s a v e c t o r −> l i n p r o g i s u s i n g i n f −norm . . .
128 f = ones (n , 1 ) ;
129 % I n i t i a l i z e constraints . . .
130 if 1
131 % Fi nd i n d i c e s o f e q u a l i t y and i n e q u a l i t y c o n s t r a i n t s . . .
132 i n = 1 : 1 :m; % i n d i c e s o f a l l c o n s t r a i n t s
133 eq = f i n d ( a b s ( bu − b l ) <= e q u a l i t y t o l ) ’ ; % i n d i c e s o f e q u a l i t y c o n s t r a i n t s
134 f o r i = eq
135 i n = i n ( f i n d ( i n ˜= i ) ) ; % rem ov e i n d i c e s o f e q u a l i t y c o n s t r a i n t s
136 end
137 % S p l i t c o n s t r a i n t s i n t o e q u a l i t y and i n e q u a l i t y c o n s t r a i n t s . . .
138 A eq = A ( : , eq ) ; A i n = A ( : , i n ) ;
139 b l e q = b l ( eq ) ; b l i n = b l ( i n ) ;
140 b u e q = bu ( eq ) ; b u i n = bu ( i n ) ;
141 A = [− A i n ’ ; A i n ’ ] ; % c o n s t r a i n t m a t r i x o f i n e q u a l i t y c o n s t r a i n t s
142 b = [− b l i n ; b u i n ] ; % i n e q u a l i t y c o n s t r a i n t s
143 Aeq = A eq ’ ; % c o n s t r a i n t m a t r i x o f e q u a l i t y c o n s t r a i n t s
144 beq = ( b l e q + b u e q ) / 2 ; % e q u a l i t y c o n s t r a i n t s
145 else
146 % a l l c o n s t r a i n t s i n i t i a l i z e d as i n e q u a l i t i e s
147 A = [−A ’ ; A ’ ] ; % o n s t r a i n t m a t r i x o f i n e q u a l i t y c o n s t r a i n t s
148 b = [− b l ; bu ] ; % i n e q u a l i t y c o n s t r a i n t s
149 Aeq = [ ] ; % no e q u a l i t y c o n s t r a i n t s
150 beq = [ ] ; % no e q u a l i t y c o n s t r a i n t s
151 end
152
153 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
154 % Fi nd f e a s i b l e p o i n t and d i s p l a y u s e r i n f o %
155 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
156 % Fi nd f e a s i b l e p o i n t u s i n g l i n p r o g w i t h d e f a u l t s e t t i n g s . . .
157 disp ( ’ Calling linprog . ’ )
158 [ x , dummy , e x i t f l a g ] = l i n p r o g ( f , A, b , Aeq , beq , l , u , [ ] , o p t i m s e t ( ’ D i s p l a y ’ , ’ o f f ’ ) ) ;
159 % R e p l a c e p s e u d o l i m i t w i t h +/− i n f . . .
160 i f p s e u d o i n f ˜= i n f
161 b ( f i n d ( b == −p s e u d o i n f ) ) = − i n f ;
162 b ( f i n d ( b == p s e u d o i n f ) ) = i n f ;
163 l ( f i n d ( l == −p s e u d o i n f ) ) = − i n f ;
164 u ( f i n d ( u == p s e u d o i n f ) ) = i n f ;
D.2 Inequality Constrained QP’s 139

165 end
166 % Display i n f o . . .
167 i f e x i t f l a g ˜= 1
168 d i s p ( [ ’ No f e a s i b l e p o i n t found , e x i t f l a g = ’ , i n t 2 s t r ( e x i t f l a g ) , ’ , s e e ” h e l p
linprog ”. ’ ])
169 end
170 fo r i = 1: length (x)
171 i f x ( i ) > pseudoinf
172 di s p ( [ ’ Fe a s i b l e poi nt regarded as i n f i n i t e , x ( ’ , i n t 2 s t r ( i ) , ’ ) > ’ ,
num 2s tr ( p s e u d o i n f ) , ’ . ’ ] )
173 end
174 i f x ( i ) < −p s e u d o i n f
175 di s p ( [ ’ Fe a s i b l e poi nt regarded as i n f i n i t e , x ( ’ , i n t 2 s t r ( i ) , ’ ) < ’ ,
num 2s tr (− p s e u d o i n f ) , ’ . ’ ] )
176 end
177 end
178 di s p ( ’ LPsolver terminated . ’ )
140 Matlab-code

D.3 Nonlinear Programming

SQP solver.m

1 f u n c t i o n [ x , i n f o , p e r f ] = S Q P s o l v e r ( modfun , modsens , c o s t f u n , c o s t s e n s , x0 ,
pi0 , opts , v a r a r g i n )
2 % SQP SOLVER S o l v e s a n o n l i n e a r program o f t h e form
3 %
4 % min f (x)
5 % s . t . h ( x ) >= 0
6 %
7 % Where f : Rˆn −> R , and h : Rˆn −>Rˆm, m eani ng t h a t n i s t h e number o f
v a r i a b l e s and m
8 % i s t h e number o f c o n s t r a i n t s . SQP s o l v e s t h e program by u s e o f t h e L a g r a n g i a n
function
9 % L ( x , y ) = f ( x )−y ’ h ( x ) w hi ch means t h a t i t i s t h e f o l l o w i n g s y s t e m t h a t i s
10 % solved
11 %
12 % n a b l a x ( L ( x , y ) ) = n a b l a ( f ( x ) )−n a b l a ( h ( x ) ) y = 0
13 % n a b l a y ( L ( x , y ) ) = −h ( x ) = 0 .
14 %
15 % Newtons method i s u s e d t o a p p r o x i m a t e t h e s o l u t i o n . Each Newton s t e p i s
c a l c u l a t e d by
16 % s o l v i n g a QP d e f i n e d a s
17 %
18 % min 0 . 5 ∗ d e l t a x ’ [ n a b l a ˆ2 x x ( L ( x , y ) ) ] d e l t a x + [ n a b l a ( f ( x ) ) ] ’ d e l t a x
19 % s . t . n a b l a ( h ( x ) ) ’ d e l t a x >= −h ( x )
20 %
21 % T h i s means t h a t t h e s o l u t i o n can o n l y be f o u n d i f n a b l a ˆ2 x x ( L ( x , y ) ) i s
22 % p o s i t i v e d e f i n i t e . An BFGS−u p d a t e h a s b e e n p r o v i d e d w hi ch a p p r o x i m a t e s n a b l a
ˆ2 x x ( L ( x , y ) ) .
23 % The s o l u t i o n i s f o u n d by s o l v i n g a s e q u e n c e o f t h e s e QPs . The
24 % d u a l a c t i v e s e t method i s u s e d f o r s o l v i n g t h e QP’ s .
25 %
26 % Call
27 % [ x , i n f o , p e r f ] = S Q P s o l v e r ( @modfun , @modsens , @ c o s t f u n , @ c o s t s e n s , x0 ,
pi0 , opts )
28 %
29 % Input parameters
30 % @modfun : f u n c t i o n s that d e f i n e s : h (x ) : Rˆn −>Rˆm
31 % @modsens : f u n c t i o n s that d e f i n e s : nabla (h ( x ) ) : Rˆn −>Rˆ ( nxm )
32 % @costfun : f u n c t i o n s that d e f i n e s : f (x ) : Rˆn −> R
33 % @costsens : f u n c t i o n s that d e f i n e s : nabla ( f ( x ) ) : Rˆn −> Rˆn
34 % x0 : starting quess
35 % pi0 : l agra nge m u l t i p l i e r s f o r the c o n s t r a i n t s .
36 % ( c o u l d be a z e r o −v e c t o r o f l e n g t h m) .
37 % opts :
38 % op ts : V ector with 3 el em en t :
39 % o p t s ( 1 ) = T o l e r a n c e u s e d t o s t a b i l i z e t h e m ethods n u m e r i c a l l y .
40 % I f | v al u e | < op ts ( 1 ) , then v al u e i s regard ed as z e r o .
41 % o p t s ( 2 ) = maximum no . o f i t e r a t i o n s t e p s .
42 % opts ( 3 ) = 1 : Using n u l l space procedure .
43 % = 2 : Using n u l l s p ace p roced u re with f a c t o r i z a t i o n
44 % update .
45 % I f op ts ( 3 ) > 2 , then op ts ( 3 ) i s s e t to 2 a u t o m a t i c a l l y .
46 % I f o p t s i s n o t g i v e n o r empty , t h e d e f a u l t o p t s = [ 1 e−8 1 0 0 0 2 ] .
47 %
48 % Output p a r a m e t e r s
49 % x : The o p t i m a l s o l u t i o n .
50 % i n f o : Performace i n form ati on , v e c t o r with 3 el em en ts :
51 % inf o (1) = f i n a l values of f .
52 % inf o (2) = no . o f i t e r a t i o n s t e p s .
53 % inf o (3) = 1 : F e a s i b l e s o l u t i o n found .
54 % 2 : No . o f i t e r a t i o n s t e p s e x c e e d e d .
55 % p e r f : Performace , s t r u c t h o l d i n g :
56 % perf . x : V a l u e s o f x from e a c h i t e r a t i o n o f SQP . S i z e i s n x i t .
57 % perf . f : V a l u e s o f f ( x ) from e a c h i t e r a t i o n o f SQP . S i z e i s 1
xit .
58 % p e r f . itQP : Number o f i t e r a t i o n s from t h e d u a l a c t i v e s e t method
59 % e a c h t i m e a QP i s s o l v e d . S i z e i s 1 x i t .
60 % p e r f . stopQP : r e a s o n why t h e d u a l a c t i v e s e t method h a s
61 % t e r m i n a t e d e a c h t i m e a QP i s s o l v e d . S i z e
62 % is 1xit .
63 % p e r f . stopQP ( i ) = 1 : s o l u t i o n o f QP h a s b e e n f o u n d s u c c e s s f u l l y .
64 % p e r f . stopQP ( i ) = 2 : s o l u t i o n o f QP h a s n o t b e e n f o u n d s u c c e s s f u l l y a s
65 % i t e r a t i o n number h a s e x c e e d e d m a x i t e r a t i o n
number .
66 % p e r f . stopQP ( i ) = 3 : s o l u t i o n o f QP h a s n o t b e e n f o u n d a s t h e QP i s
infeasible .
67 %
68 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 .
69 % Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
D.3 Nonlinear Programming 141

70 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n .
71 % M. Sc . , IMM, DTU, DK−2800 Lyngby .
72 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r .
73 % Per Grove Thomsen , P r o f e s s o r .
74 % D ate : 0 7 . J une 2 0 0 7 .
75 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
76
77 if n argi n < 7 | isempty ( op ts )
78 t o l = 1 e −8;
79 it max = 1000;
80 method = 2 ;
81 o p t s = [ t o l i t m a x method ] ;
82 else
83 i f opts (3 ) > 2
84 opts (3 ) = 2 ;
85 end
86 opts = opts ( : ) ’ ;
87 end
88
89 f 0 = f e v a l ( c o s t f u n , x0 , v a r a r g i n { : } ) ;
90 g0 = f e v a l ( modfun , x0 , v a r a r g i n { : } ) ;
91 c = f e v a l ( c o s t s e n s , x0 , v a r a r g i n { : } ) ;
92 A = f e v a l ( modsens , x0 , v a r a r g i n { : } ) ;
93 W = e y e ( l e n g t h ( x0 ) ) ;
94 w non = ( 1 : 1 : l e n g t h ( g0 ) ) ;
95
96 stop = 0;
97 t o l = opts (1 ) ;
98 it max = opts ( 2) ;
99 it r = 0;
100 n = l e n g t h ( x0 ) ;
101 x i n i t = x0 ;
102 f i n i t = f0 ;
103
104 % i n i t i a l i z e containers . . .
105 t r a c e = ( nargout > 2) ;
106 i f trace
107 X = repm at ( z e r o s ( n , 1 ) , 1 , i t m a x ) ; % x o f SQP
108 F = repm at ( 0 , 1 , i t m a x ) ; % f u n c t i o n v a l u e o f SQP
109 I t = repm at ( 0 , 1 , i t m a x ) ; % no . i t e r a t i o n s o f QP
110 S t o p = repm at ( 0 , 1 , i t m a x ) ; % s t o p o f QP
111 end
112 max itr = it max ;
113
114 w hi l e ˜ stop
115 X ( : , i t r +1) = x0 ;
116 i t r = i t r + 1;
117 i f ( i t r > max itr )
118 stop = 2 ;
119 end
120
121 [ d e l t a x , mu, i n f o ] = d u a l a c t i v e s e t m e t h o d (W, c , A,−g0 , w non , [ ] , o p t s ) ;
122
123 if( a b s ( c ’ ∗ d e l t a x ) + a b s (mu’ ∗ g0 ) ) < t o l
124 d i s p ( ’ s o l u t i o n has been found ’ )
125 stop = 1 ;
126 else
127
128 if i t r == 1
129 s i g m a = a b s (mu) ;
130 else
131 for i = 1: l e n g t h (mu)
132 s i g m a ( i ) = max ( a b s (mu( i ) ) , 0 . 5 ∗ ( s i g m a ( i )+a b s (mu( i ) ) ) ) ;
133 end
134 end
135
136 [ a l p h a , x , f , g ] = l i n e s e a r c h a l g o r i t h m ( modfun , c o s t f u n , f 0 , g0 , c , x0 , d e l t a x
, s i gm a , 1 e −4) ;
137
138 p i i = p i 0 + a l p h a ∗ (mu−p i 0 ) ;
139
140 n a b l a L 0 = c−A∗ p i i ;
141 c = f e v a l ( costsens , x , varargin {:}) ;
142 A = f e v a l ( modsens , x , v a r a r g i n { : } ) ;
143 n a b l a L = c−A∗ p i i ;
144 s = x − x0 ;
145 y = nabla L − nabla L0 ;
146 sy = s ’ ∗ y ;
147 sWs = s ’ ∗W∗ s ;
148 i f ( s y >= 0 . 2 ∗ sWs )
149 theta = 1;
150 else
151 t h e t a = ( 0 . 8 ∗ sWs ) / ( sWs−s y ) ;
152 end
153 Ws = W∗ s ;
154 sW = s ’ ∗W;
155 r = t h e t a ∗y+(1− t h e t a ) ∗Ws ;
142 Matlab-code

156 W = W−(Ws∗sW) /sWs+( r ∗ r ’ ) / ( s ’ ∗ r ) ;

157 x0 = x ;
158 pi0 = p i i ;
159 f0 = f ;
160 g0 = g ;
161 end
162
163 % c o l l e c t i n g output i n c o n t a i n e r s . . .
164 i f trace
165 X ( : , i t r ) = x0 ;
166 F( i t r ) = f 0 ;
167 I t ( i t r ) = in fo (2) ;
168 Stop ( i t r ) = i n f o ( 3 ) ;
169 end
170 end
171
172 i n f o = [ f 0 i t r s t o p ] ; % SQP i n f o
173 x = x0 ;
174 % building perf . . .
175 i f trace
176 X = X (: ,1: itr ) ; X = [ xinit X ] ;
177 F = F( 1: i t r ) ; F = [ f i n i t F ] ;
178 It = It (1: i t r ) ; It = [0 It ] ;
179 Stop = Stop ( 1 : i t r ) ; Stop = [ 0 Stop ] ;
180 p e r f = s t r u c t ( ’ x ’ , {X} , ’ f ’ , { F} , ’ itQP ’ , { I t } , ’ stopQP ’ , { S t o p } ) ;
181 end
D.4 Updating the Matrix Factorizations 143

D.4 Updating the Matrix Factorizations

givens rotation matrix.m

1 function [ c , s ] = givens rotation matrix (a , b)

2
3 % GIVENS ROTATION MATRIX : c a l c u l a t e s t h e e l e m e n t s c and s w hi ch a r e u s e d t o
4 % i n t r o d u c e one z e r o i n a v e c t o r o f two e l e m e n t s
5 %
6 % Call
7 % [ c s ] = givens rotation matrix (a , b)
8 %
9 % Input parameters
10 % a and b a r e t h e two e l e m e n t s o f t h e v e c t o r w here we want t o
11 % i n t r o d u c e one z e r o .
12 %
13 % Output p a r a m e t e r s
14 % c and s i s u s e d t o c o n s t r u c t t h e g i v e n s r o t a t i o n m a t r i x Q g i v e n s : [ c −s ;
s c ].
15 % Now one z e r o i s i n t r o d u c e d : Q g i v e n s ∗ [ a b ] ’ = [ gamma 0 ] ,
16 % w here gamma i s t h e l e n g t h o f [ a b ] i s a b s ( gamma)
17 %
18 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
19 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
20 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
21 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
22 % D ate : 31. october 2006.
23 % Reference : −−−−−−−−−−−−−−−−−−−−−
24
25
26
27 % i f ( b==0)
28 % c = 1;
29 % s = 0;
30 % else
31 % i f ( a b s ( b )>a b s ( a ) )
32 % t a u = −a /b ;
33 % s = 1/ s q r t (1+ t a u ∗ t a u ) ;
34 % c = tau ∗ s ;
35 % else
36 % t a u = −b / a ;
37 % c = 1/ s q r t (1+ t a u ∗ t a u ) ;
38 % s = tau ∗c ;
39 % end
40 % end
41 G = givens (a , b) ;
42 c = G( 1 , 1 ) ;
43 s = G( 2 , 1 ) ;

range space update.m

1 f u n c t i o n [ x , u , Q, R ] = r a n g e s p a c e u p d a t e ( L , A, g , b , Q, R, c o l r e m )
2 % RANGE SPACE UPDATE u s e s t h e r a n g e−s p a c e p r o c e d u r e f o r s o l v i n g a QP p r o b l e m :
min f ( x ) = 0. 5∗x ’ Gx+g ’ x s t : A ’ x=b ,
3 % w here A c o n t a i n s m c o n s t r a i n t s and t h e s y s t e m h a s n v a r i a b l e s .
RANGE SPACE UPDATE c o n t a i n s m ethods f o r
4 % updati ng the f a c t o r i z a t i o n s u s i n g Givens r o t a t i o n s .
5
6 % ∗∗∗ when s o l v i n g an i n e q u a l i t y c o n s t r a i n e d QP, a s e q e n c e o f e q u a l i t y
7 % c o n s t r a i n e d QPs a r e s o l v e d . The d i f f e r e n c e b e t w e e n two o f t h e s e
8 % f o l l o w i n g e q u a l i t y c o n s t r a i n e d QP i s one appended c o n s t r a i n t a t t h e
9 % l a s t i n d e x o f A , o r a c o n s t r a i n t rem ov ed a t i n d e x c o l r e m o f A .
10 %
11 % Call
12 % [ x , u , Q, R ] = r a n g e s p a c e u p d a t e ( L , A, g , b , Q, R, c o l r e m )
13 %
14
15 % Input parameters
16 % L : is the Cholesky factorization of the Hessian matrix G
o f f ( x ) . L i s nxn
17 % A : is the constraint matrix . The constraints are columns
i n A . A i s nxm
18 % g : c on ta i ns n elements
19 % b : c on ta i ns m elements
144 Matlab-code

20 % Q and R : i s t h e QR−f a c t o r i z a t i o n o f t h e QP w hi ch h a s j u s t b e e n s o l v e d
21 % ( i f not the f i r s t i t e r a t i o n ) i n the s eq uence d e s c i b e d i n ∗∗∗.
22 % col rem : i s t h e i n d e x a t w hi ch a c o n s t r a i n t h a s b e e n rem ov ed
from A .
23 %
24 % Q, R and c o l r e m can be empty [ ] w hi ch means that The QP
25 % i s t h e f i r s t one i n the sequence ( s e e ∗∗∗) .
26
27 % Output p a r a m e t e r s
28 % x : i s the optimized poi nt
29 % u : i s the corres pondi ng Lagrangian Multipliers
30 % Q and R : i s t h e QR−f a c t o r i z a t i o n o f A
31
32 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
33 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
34 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
35 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
36 % D ate : 11. february 2007.
37 % Reference : −−−−−−−−−−−−−−−−−−−−−
38
39 [ nA ,mA] = s i z e (A) ;
40 [ nR ,mR] = s i z e (R) ;
41 K = L\A ;
42 w = L\ g ;
43 z = b+(w’ ∗K) ’ ;
44 i f i s e m p t y (Q) && i s e m p t y (R)
45 % di s p ( ’ complete f a c t o r i z a t i o n ’ ) ;
46 [ Q, R ] = q r (K) ;
47 e l s e i f mR < mA % new col um n h a s b e e n appended t o A
48 % d i s p ( ’ append update ’ ) ;
49 [ Q, R ] = q r f a c t u p d a t e a p p c o l (Q, R, K ( : , end ) ) ;
50 e l s e i f mR > mA % col um n h a s b e e n rem ov ed from A a t i n d e x col rem
51 % d i s p ( ’ rem ov e u p d a t e ’ ) ;
52 [ Q, R ] = q r f a c t u p d a t e r e m c o l (Q, R, c o l r e m ) ;
53 end
54 u = R( 1 : l en gth ( z ) , : ) ’\ z ;
55 u = R ( 1 : l e n g t h ( z ) , : ) \u ;
56 y = K∗u−w ;
57 x = L’\y ;

qr fact update app col.m

1 f u n c t i o n [ Q, R ] = q r f a c t u p d a t e a p p c o l (Q, R, c o l n e w )
2 % QR FACT UPDATE APP COL u p d a t e s t h e q r−f a c t o r i z a t i o n when a s i n g l e col um n i s
3 % appended a t i n d e x m+ 1. And t h e f a c t o r i z a t i o n from b e f o r e a d d i n g t h e col um n
i s known
4 % : ( Q o l d and R o l d )
5 %
6 % Call
7 % [ q r ] = q r f a c t u p d a t e a p p c o l ( Q ol d , R o l d , c o l n e w )
8 %
9 % Input parameters
10 % Q o l d i s t h e Q p a r t o f t h e QR−f a c t o r i z a t i o n from t h e f o r m e r
11 % m a t r i x A ( t h e m a t r i x we want t o append one col um n a t i n d e x m+1) .
12 % R o l d i s t h e R p a r t from t h e QR−f a c t o r i z a t i o n from t h e f o r m e r
13 % m a t r i x A ( t h e m a t r i x we want t o append one col um n a t i n d e x m+1) .
14 % c o l n e w i s t h e col um n we want t o append
15 %
16 % Output p a r a m e t e r s
17 % Q i s t h e u p d a t e d Q−m a t r i x
18 % R i s t h e u p d a t e d R−m a t r i x ( e v e r y t h i n g but t h e u p p e r mxm m a t r i x i s z e r o s
)
19 %
20 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
21 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
22 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
23 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
24 % D ate : 31. october 2006.
25 % Reference : −−−−−−−−−−−−−−−−−−−−−
26
27 [ n m] = s i z e (R) ;
28 sw = ( c o l n e w ’ ∗Q) ’ ;
29 for j = n : − 1 :m+2
30 i = j −1;
31 [ c s ] = g i v e n s r o t a t i o n m a t r i x ( sw ( i ) , sw ( j ) ) ;
32 e 1 = sw ( i ) ∗ c − sw ( j ) ∗ s ;
33 sw ( j ) = sw ( i ) ∗ s + sw ( j ) ∗ c ;
34 sw ( i ) = e 1 ;
D.4 Updating the Matrix Factorizations 145

35 v1 = Q ( : , i ) ∗ c − Q ( : , j ) ∗ s ;
36 Q( : , j ) = Q( : , i ) ∗ s + Q( : , j ) ∗c ;
37 Q ( : , i ) = v1 ;
38 end
39 R = [ R sw ] ;

qr fact update rem col.m

1 f u n c t i o n [ Q, R ] = q r f a c t u p d a t e r e m c o l (Q, R, c o l i n d e x )
2 % QR FACT UPDATE REM COL u p d a t e s t h e q r−f a c t o r i z a t i o n when a s i n g l e col um n i s
3 % rem ov ed .
4 %
5 % Call
6 % [ q r ] = q r f a c t u p d a t e r e m c o l ( Q ol d , R o l d , c o l n e w )
7 %
8 % Input parameters
9 % Q o l d i s t h e Q p a r t from t h e QR−f a c t o r i z a t i o n from t h e f o r m e r
10 % m a t r i x A ( t h e m a t r i x we want t o rem ov e one col um n ) .
11 % R o l d i s t h e R p a r t from t h e QR−f a c t o r i z a t i o n from t h e f o r m e r
12 % m a t r i x A ( t h e m a t r i x we want t o rem ov e one col um n ) .
13 % c o l i n d e x i s t h e i n d e x o f t h e col um n we want t o rem ov e
14 %
15 % Output p a r a m e t e r s
16 % Q i s t h e u p d a t e d Q−m a t r i x
17 % R i s t h e u p d a t e d R−m a t r i x ( e v e r y t h i n g but t h e u p p e r mxm m a t r i x i s z e r o s
)
18 %
19 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
20 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
21 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
22 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
23 % D ate : 31. october 2006.
24 % Reference : −−−−−−−−−−−−−−−−−−−−−
25
26 [ n m] = s i z e (R) ;
27 t = m − col index ;
28 for i = 1:1: t
29 j = i + 1;
30 [c s ] = g i v e n s r o t a t i o n m a t r i x (R( c o l i n d e x +i −1 , c o l i n d e x +i ) , R( c o l i n d e x +j
−1 , c o l i n d e x + i ) ) ;
31 v1 = R( c o l i n d e x +i −1 , c o l i n d e x + 1: end ) ∗ c − R( c o l i n d e x +j −1 , c o l i n d e x + 1: end ) ∗
s;
32 R( c o l i n d e x +j −1 , c o l i n d e x + 1: end ) = R( c o l i n d e x +i −1 , c o l i n d e x + 1: end ) ∗ s + R(
c o l i n d e x+j −1 , c o l i n d e x + 1: end ) ∗ c ;
33 R( c o l i n d e x +i −1 , c o l i n d e x + 1: end ) = v1 ;
34 q1 = Q ( : , c o l i n d e x+i −1) ∗ c − Q ( : , c o l i n d e x +j −1) ∗ s ;
35 Q ( : , c o l i n d e x +j −1) = Q ( : , c o l i n d e x +i −1) ∗ s + Q ( : , c o l i n d e x +j −1) ∗ c ;
36 Q ( : , c o l i n d e x +i −1) = q1 ;
37 end
38 R = [ R ( : , 1 : c o l i n d e x −1) R ( : , c o l i n d e x + 1: end ) ] ;

null space update.m

1 f u n c t i o n [ x , u , Q new , T new , L new ] = n u l l s p a c e u p d a t e (G, A , g , b , Q ol d , T o l d , L o l d ,

col rem )
2
3 % NULL SPACE UPDATE u s e s t h e n u l l −s p a c e p r o c e d u r e f o r solving a QP p r o b l e m : min
f ( x ) = 0. 5∗ x ’ Gx+g ’ x s t : A ’ x=b ,
4 % w here A c o n t a i n s m c o n s t r a i n t s and t h e s y s t e m h a s n variables .
NULL SPACE UPDATE c o n t a i n s m ethods f o r
5 % updati ng the f a c t o r i z a t i o n s u s i n g Givens r o t a t i o n s .
6
7 % ∗∗∗ when s o l v i n g an i n e q u a l i t y c o n s t r a i n e d QP, a s e q e n c e o f e q u a l i t y
8 % c o n s t r a i n e d QPs a r e s o l v e d . The d i f f e r e n c e b e t w e e n two o f t h e s e
9 % f o l l o w i n g e q u a l i t y c o n s t r a i n e d QP i s one appended c o n s t r a i n t a t t h e
10 % l a s t i n d e x o f A , o r a c o n s t r a i n t rem ov ed a t i n d e x c o l r e m o f A .
11 %
12 % Call
13 % [ x , u , Q new , T new , L new ] = n u l l s p a c e u p d a t e (G, A , g , b , Q ol d , T o l d , L o l d ,
col rem )
14 %
15
146 Matlab-code

16 % Input parameters
17 % G : i s t h e H e s s i a n m a t r i x o f f ( x ) . G i s nxn
18 % A : i s t h e c o n s t r a i n t m a t r i x . The c o n s t r a i n t s a r e c o l u m n s
i n A . A i s nxm
19 % g : c o nt ai ns n elements
20 % b : c o nt ai ns m elements
21 % Q o l d and T o l d : i s t h e QT−f a c t o r i z a t i o n o f t h e QP w hi ch h a s j u s t b e e n
solved
22 % ( i f n o t t h e f i r s t i t e r a t i o n ) i n t h e s e q u e n c e d e s c i b e d i n ∗ ∗ ∗ . The
23 % T p a r t o f t h e QT−f a c t o r i z a t i o n i s l o w e r t r i a n g u l a r
24 % L old : i s the Cholesky f a c t o r i z a t i o n o f the reduced H es s i an
25 % m a t r i x o f t h e QP j u s t s o l v e d ( s e e ∗ ∗ ∗ ) .
26 % col rem : i s t h e i n d e x a t w hi ch a c o n s t r a i n t h a s b e e n rem ov ed
from A .
27 %
28 % Q ol d , T old , L ol d and c o l r e m can be empty [] w hi ch means t h a t The QP
29 % i s the f i r s t one i n the s eq u en ce ( s e e ∗∗∗) .
30
31 % Output p a r a m e t e r s
32 % x : is the optimized poi nt
33 % u : is the corres pondi ng Lagrangian M u l t i p l i e r s
34 % Q new and T new : is t h e QT−f a c t o r i z a t i o n o f A
35 % L new : is the Cholesky f a c t o r i z a t i o n o f the reduced Hessian
matrix .
36
37 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
38 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
39 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
40 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
41 % D ate : 08. february 2007.
42
43 [ nA ,mA] = s i z e (A) ;
44 [ nT ,mT] = s i z e ( T o l d ) ;
45
46 di m N ul Space = nA−mA;
47
48 Q new = Q o l d ;
49 T new = T o l d ;
50 L new = L o l d ;
51 i f i s e m p t y ( Q o l d ) && i s e m p t y ( T o l d ) && i s e m p t y ( L o l d )
52 %d i s p ( ’ c o m p l e t e f a c t o r i z a t i o n ’ ) ;
53 [ Q, R ] = q r (A) ;
54 I t i l d e = f l i p u d ( e y e (nA ) ) ;
55 T new = I t i l d e ∗R ;
56 Q new = Q∗ I t i l d e ;
57 Q1 = Q new ( : , 1 : di m N ul Space ) ;
58 Gz = Q1 ’ ∗G∗Q1 ;
59 L new = c h o l ( Gz ) ’ ;
60 e l s e i f mT < mA % new col um n h a s b e e n appended t o A
61 % d i s p ( ’ append update ’ ) ;
62 [ Q new , T new , L new ] = n u l l s p a c e u p d a t e f a c t a p p c o l ( Q ol d , T old , L old ,
A ( : , end ) ) ;
63 e l s e i f mT > mA% col um n h a s b e e n rem ov ed from A a t i n d e x c o l r e m
64 % d i s p ( ’ rem ov e u p d a t e ’ ) ;
65 [ Q new , T new , L new ] = n u l l s p a c e u p d a t e f a c t r e m c o l ( Q ol d , T old , L old ,
G, c o l r e m ) ;
66 end
67
68 Q1 = Q new ( : , 1 : di m N ul Space ) ;
69 Q2 = Q new ( : , di m N ul Space + 1:nA ) ;
70 T newMark = T new ( di m N ul Space + 1: end , : ) ;
71 py = T newMark ’ \ b ;
72 g z = −((G∗ (Q2∗py ) + g ) ’ ∗Q1 ) ’ ;
73 z = L new \ g z ;
74 pz = L new ’ \ z ;
75 x = Q2∗py + Q1∗ pz ;
76 u = ( (G∗x + g ) ’ ∗Q2 ) ’ ;
77 u = T newMark \u ;

null space update fact app col.m

1 f u n c t i o n [ Q, T , L ] = n u l l s p a c e u p d a t e f a c t a p p c o l (Q, T , L , c o l n e w )
2 % NULL SPACE UPDATE FACT APP COL u p d a t e s t h e QT−f a c t o r i z a t i o n o f A when a
3 % s i n g l e col um n c o l n e w i s appended t o A a s t h e l a s t col um n . The r e s u l t i n g
4 % c o n s t r a i n t m a t r i x i s Abar = [ A c o l n e w ] . The c o r r e s p o n d i n g QP p r o b l e m h a s a
reduced Hessian
5 % m a t r i x redH and t h e c h o l e s k y f a c t o r i z a t i o n o f redH i s L o l d .
6
7 % Call
D.4 Updating the Matrix Factorizations 147

8 % [ Q, T , L ] = n u l l s p a c e u p d a t e f a c t a p p c o l (Q, T , L, col new )

9
10 % Input parameters
11 % Q and T : i s t h e QT−f a c t o r i z a t i o n o f A
12 % L : i s the chol es k y f a c t o r i z a t i o n o f the reduced Hessian
m a t r i x o f t h e c o r r e s p o n d i n g QP p r o b l e m .
13 % col new : i s t h e col um n t h a t i s appended t o A : Abar = [ A
col new ]
14
15 % Output p a r a m e t e r s
16 % Q and T : i s t h e QT−f a c t o r i z a t i o n o f Abar
17 % L : i s the Cholesky f a c t o r i z a t i o n o f the reduced Hessian
m a t r i x o f t h e new QP p r o b l e m .
18
19 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
20 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
21 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
22 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
23 % D ate : 08. february 2007.
24 % Reference : −−−−−−−−−−−−−−−−−−−−−
25
26 [ n ,m] = s i z e (T) ;
27 d i m N u l l S p a c e = n − m;
28 wv = ( c o l n e w ’ ∗Q) ’ ;
29 f o r i = 1 : d i m N u l l S p a c e−1
30 j = i + 1;
31 [ s , c ] = g i v e n s r o t a t i o n m a t r i x ( wv ( i ) , wv ( j ) ) ;
32 temp = wv ( i ) ∗ c + wv ( j ) ∗ s ;
33 wv ( j ) = wv ( j ) ∗ c − wv ( i ) ∗ s ;
34 wv ( i ) = temp ;
35 temp = Q ( : , i ) ∗ c + Q ( : , j ) ∗ s ;
36 Q( : , j ) = Q( : , j ) ∗ c − Q( : , i ) ∗ s ;
37 Q ( : , i ) = temp ;
38 temp = L ( i , : ) ∗ c + L ( j , : ) ∗ s ;
39 L( j , : ) = L( j , : ) ∗c − L( i , : ) ∗ s ;
40 L ( i , : ) = temp ;
41 end
42 f o r i = 1 : d i m N u l l S p a c e−1
43 j = i + 1;
44 [ c , s ] = g i v e n s r o t a t i o n m a t r i x (L( i , i ) , L( i , j ) ) ;
45 temp = L ( : , i ) ∗ c − L ( : , j ) ∗ s ;
46 L ( : , j ) = L ( : , j )∗c + L ( : , i )∗ s ;
47 L ( : , i ) = temp ;
48 end
49 T = [ T wv ] ;
50 L = L ( 1 : d i m N u l l S p a c e − 1 , 1 : d i m N u l l S p a c e −1) ;

null space update fact rem col.m

1 f u n c t i o n [ Q, T , L ] = n u l l s p a c e u p d a t e f a c t r e m c o l (Q, T , L , G, c o l r e m )
2 % NULL SPACE UPDATE FACT REM COL u p d a t e s t h e QT−f a c t o r i z a t i o n o f A when a
3 % col um n i s rem ov ed from A a t column−i n d e x c o l r e m . The new C o n s t r a i n t m a t r i x
i s c a l l e d Abar .
4 % The c o r r e s p o n d i n g QP p r o b l e m h a s a r e d u c e d H e s s i a n m a t r i x redH and t h e
cholesky f a c to ri z a ti o n
5 % o f redH i s L .
6
7 % Call
8 % [ Q, T , L ] = n u l l s p a c e u p d a t e f a c t r e m c o l (Q, T , L , G, col rem )
9
10 % Input parameters
11 % Q and T : i s t h e QT−f a c t o r i z a t i o n o f A
12 % L : i s the chol es k y f a c t o r i z a t i o n o f the reduced Hessian
m a t r i x o f t h e c o r r e s p o n d i n g QP p r o b l e m .
13 % G : i s t h e H e s s i a n m a t r i x o f t h e QP p r o b l e m .
14 % col rem : i s t h e column−i n d e x a t w hi ch a col um n h a s b e e n
rem ov ed from A
15
16 % Output p a r a m e t e r s
17 % Q and T : i s t h e QT−f a c t o r i z a t i o n o f Abar
18 % L : i s the Cholesky f a c t o r i z a t i o n o f the reduced Hessian
m a t r i x o f t h e new QP p r o b l e m .
19
20 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
21 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
22 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
23 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
24 % D ate : 08. february 2007.
148 Matlab-code

25 % Reference : −−−−−−−−−−−−−−−−−−−−−
26
27 [ n ,m] = s i z e (T) ;
28 di m N ul Space = n−m;
29 j = col rem ;
30 mm = m−j ;
31 nn = mm+ 1;
32
33
34 for i = 1 : 1 :mm
35 i d x 1 = nn−i ;
36 i d x 2 = i d x 1 + 1;
37 [ s , c ] = g i v e n s r o t a t i o n m a t r i x (T( di m N ul Space+i d x 1 , j+ i ) ,T( di m N ul Space+i d x 2 , j
+i ) ) ;
38 temp = T( di m N ul Space+i d x 1 , j + 1: end ) ∗ c + T( di m N ul Space+i d x 2 , j + 1: end ) ∗ s ;
39 T( di m N ul Space+i d x 2 , j + 1: end ) = −T( di m N ul Space+i d x 1 , j + 1: end ) ∗ s + T(
di m N ul Space+i d x 2 , j + 1: end ) ∗ c ;
40 T( di m N ul Space+i d x 1 , j + 1: end ) = temp ;
41 temp = Q ( : , di m N ul Space+i d x 1 ) ∗ c + Q ( : , di m N ul Space+i d x 2 ) ∗ s ;
42 Q ( : , di m N ul Space+i d x 2 ) = −Q ( : , di m N ul Space+i d x 1 ) ∗ s + Q ( : , di m N ul Space+i d x 2 ) ∗ c ;
43 Q ( : , di m N ul Space+i d x 1 ) = temp ;
44 end
45
46 T = [ T ( : , 1 : j −1) T ( : , j + 1: end ) ] ;
47 z = Q ( : , di m N ul Space +1) ;
48 l = L \ ( (G∗ z ) ’ ∗Q ( : , 1 : di m N ul Space ) ) ’ ;
49 d e l t a = s q r t ( z ’ ∗G∗ z−l ’ ∗ l ) ;
50 L = [ L z e r o s ( di m N ul Space , 1 ) ; l ’ d e l t a ] ;

null space updateFRFX.m

1 f u n c t i o n [ x , u , Q f r , T f r , L f r ] = n u l l s p a c e u p d a t e F R F X ( Q f r , T f r , L f r , G, A, g , b ,
dim fx , c o l r e m )
2
3 % NULL SPACE UPDATE FRFX u s e s t h e same p r o c e d u r e a s NULL SPACE UPDATE f o r
solving f ( x ) = 0. 5∗x ’ Gx+g ’ x s t : A’ x=b ,
4 %( s o p l e a s e t a k e a l o o k a t i t ) . The d i f f e r e n c e i s t h a t NULL SPACE UPDATE FRFX
t a k e s a d v a n t a g e o f t h e f a c t t h a t some o f t h e a c t i v e c o n s t r a i n t s
5 % a r e bounds ( u s u a l l y ) . An a c t i v e bound c o r r e s p o n d t o one f i x e d v a r i a b l e . T h i s
means t h a t x can be d e v i d e d i n t o [ x f r e e , x f i x e d ] ’
6 % w here x f i x e d a r e t h o s e v a r i a b l e s w hi ch a r e f i x e d . The p a r t o f t h e
7 % f a c t o r i z a t i o n s w hi ch c o r r e s p o n d t o t h e f i x e d v a r i a b l e s can n o t be c h a n g e s
8 % ( a s t h e y a r e f i x e d ) and t h i s means t h a t we a r e o n l y r e q u i r e d t o
9 % r e f a c t o r i z e t h e p a r t w hi ch c o r r e s p o n d t o t h e f r e e v a r i a b l e s .
10
11 % ∗∗∗ when s o l v i n g an i n e q u a l i t y c o n s t r a i n e d QP, a s e q e n c e o f e q u a l i t y
12 % c o n s t r a i n e d QPs a r e s o l v e d . The d i f f e r e n c e b e t w e e n two o f t h e s e
13 % f o l l o w i n g e q u a l i t y c o n s t r a i n e d QP i s one appended c o n s t r a i n t o r one
14 % rem ov ed c o n s t a i n t
15 %
16 % Call
17 % [ x , u , Q f r , T f r , L f r ] = n u l l s p a c e u p d a t e F R F X ( Q f r , T f r , L f r , G, A, g , b ,
dim fx , c o l r e m )
18 %
19 % Input parameters
20 % G : i s t h e H e s s i a n m a t r i x o f f ( x ) . G i s nxn
21 % A : i s t h e c o n s t r a i n t m a t r i x w hi ch o n l y c o n t a i n s a c t i v e
g e n e r a l c o n s t r a i n t s ( t h e bound−c o n s t r a i n t s h a s b e e n rem ov ed ) .
22 % The d i m e n s i o n o f A i s n x m f r ( n i s number
23 % o f v a r i a b l e s and m f r i s t h e number o f
24 % active general c o nstra ints )
25 % g : i s t h e g r a d i e n t o f f ( x ) and t h e d i m e n s i o n i s nx1
26 % b : c o n t a i n s t h e max v a l u e s o f t h e c o n s t r a i n t s ( b o t h
27 % g e n e r a l and an bound c o n s t r a i n t s ) and t h e r e f o r e
the dimension i s
28 % ( m f r+mfx ) x1 .
29 % Q f r and T f r a r e t h e f r e e p a r t o f t h e QT−f a c t o r i z a t i o n o f t h e p a r t o f
t h e QP w hi ch h a s j u s t b e e n s o l v e d
30 % ( i f n o t t h e f i r s t i t e r a t i o n ) i n t h e s e q u e n c e d e s c i b e d i n ∗ ∗ ∗ . The
31 % T p a r t o f t h e QT−f a c t o r i z a t i o n i s l o w e r t r i a n g u l a r
32 % L old : i s the Cholesky f a c t o r i z a t i o n o f the reduced H es s i an
33 % m a t r i x o f t h e QP j u s t s o l v e d ( s e e ∗ ∗ ∗ ) .
34 % col rem : i s t h e i n d e x a t w hi ch a c o n s t r a i n t h a s b e e n rem ov ed
from A ( i f a c o n s t r a i n t h a s b e e n appended t h i s
35 % v a r i a b l e i s unus ed .
36 %
37 % Q ol d , T o l d , L o l d and c o l r e m can be empty [] w hi ch means t h a t The QP
38 % i s the f i r s t one i n t h e s e q u e n c e ( s e e ∗ ∗ ∗ ) .
39 % dim fx : number o f f i x e d v a r i a b l e s
40 %
D.4 Updating the Matrix Factorizations 149

41 % Output p a r a m e t e r s
42 % x : is the s o l u t i o n
43 % u : is the corres pondi ng Lagrangian M u l t i p l i e r s
44 % Q f r and T f r : is t h e QT−f a c t o r i z a t i o n o f A c o r r e s p o n d i n g t o t h e
45 % free variables .
46 % L fr : i s the Cholesky f a c t o r i z a t i o n o f the reduced
47 % H es s i an matrix .
48 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
49 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
50 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
51 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
52 % D ate : 08. february 2007.
53 % Reference : −−−−−−−−−−−−−−−−−−−−−
54
55 [ nT mT] = s i z e ( T f r ) ;
56 [ nA mA] = s i z e (A) ;
57 d i m f x o l d = nA−nT ;
58
59 if i s e m p t y (A) % n o t h i n g t o f a c t o r i z e
60 % d i s p ( ’ A i s empty ’ )
61 C = eye ( di m fx ) ;
62 C = [ z e r o s ( l e n g t h ( g )−d i m f x , d i m f x ) ; C ] ;
63 [ x u ] = n u l l s p a c e (G, C , g , b ) ;
64 Q fr = [ ] ; T f r = [ ] ; L f r = [ ] ; A fx = [ ] ;
65
66 e l s e i f i s e m p t y ( T f r ) | | ( (mA == mT) && ( d i m f x == d i m f x o l d ) ) % c o m p l e t e
factorization
67 % di s p ( ’ complete f a c t o r i z a t i o n ’ )
68 A f r = A ( 1 : end−d i m f x , : ) ;
69 [ n m] = s i z e ( A f r ) ;
70 G f r f r = G( 1 : n , 1 : n ) ;
71 dns = n−m;
72 [ Q, R ] = q r ( A f r ) ;
73 I t i l d e = f l i p u d ( eye ( n ) ) ;
74 T f r = I t i l d e ∗R ;
75 Q f r = Q∗ I t i l d e ;
76 Qz = Q f r ( : , 1 : dns ) ;
77 Gz = Qz ’ ∗ G f r f r ∗Qz ;
78 L f r = c h o l ( Gz ) ’ ;
79 A f x = A( end−d i m f x + 1: end , : ) ;
80 [ x u ] = h e l p f u n ( Q f r , T f r , L f r , A fx , G, g , b ) ;
81
82 e l s e i f mA > mT % one g e n e r a l c o n s t r a i n t h a s b e e n appended
83 % d i s p ( ’ append g e n e r a l c o n s t r a i n t ’ )
84 [ Q fr , T fr , L f r ] = n u l l s p a c e u p d a t e f a c t a p p g e n e r a l F R F X ( Q fr , T fr ,
L f r , A ( : , end ) ) ;
85 dim fr = s i z e ( T fr , 1 ) ;
86 A f x = A( d i m f r + 1: end , : ) ;
87 [ x u ] = h e l p f u n ( Q f r , T f r , L f r , A fx , G, g , b ) ;
88 e l s e i f mA < mT % one g e n e r a l c o n s t r a i n t h a s b e e n rem ov ed a t i n d x c o l r e m
89 % d i s p ( ’ rem ov e g e n e r a l c o n s t r a i n t ’ )
90 dim fr = s i z e ( T fr , 1 ) ;
91 G f r f r = G( 1 : d i m f r , 1 : d i m f r ) ;
92 [ Q fr , T fr , L f r ] = n u l l s p a c e u p d a t e f a c t r e m g e n e r a l F R F X ( Q fr , T fr ,
L fr , G frfr , col rem ) ;
93 A f x = A( d i m f r + 1: end , : ) ;
94 [ x u ] = h e l p f u n ( Q f r , T f r , L f r , A fx , G, g , b ) ;
95
96 e l s e i f d i m f x > d i m f x o l d % one bound h a s b e e n appended
97 % d i s p ( ’ append bound ’ )
98 [ Q fr , T fr , L f r ] = null space update fact app bound FRFX ( Q fr , T fr , L fr
);
99 dim fr = s i z e ( T fr , 1 ) ;
100 A f x = A( d i m f r + 1: end , : ) ;
101 [ x u ] = h e l p f u n ( Q f r , T f r , L f r , A fx , G, g , b ) ;
102
103 e l s e i f d i m f x < d i m f x o l d % one bound h a s b e e n rem ov ed
104 % d i s p ( ’ rem ov e bound ’ )
105 [ nT mT] = s i z e ( T f r ) ;
106 dns = nT−mT;
107 T f r = T f r ( dns + 1: end , : ) ;
108 T f r = [ T f r ; A( nT + 1 , : ) ] ;
109 [ Q fr , T fr , L f r ] = null space update fact rem bound FRFX ( Q fr , T fr , L fr
, G) ;
110 A f x = A( nT+ 2: end , : ) ;
111 [ x u ] = h e l p f u n ( Q f r , T f r , L f r , A fx , G, g , b ) ;
112 end
113
114 f u n c t i o n [ x new u new ] = h e l p f u n ( Q f r , T f r , L f r , A fx , G, g , b )
115 % disp ( ’ help fun ’ )
116 [ nT ,mT] = s i z e ( T f r ) ;
117 dns = nT−mT;
118 d i m f r = nT ;
119 d i m f x = l e n g t h ( g )−d i m f r ;
120 Q1 = Q f r ( : , 1 : dns ) ;
121 Q2 = Q f r ( : , dns + 1:nT ) ;
150 Matlab-code

122 T f r = T f r ( dns + 1: end , : ) ;

123 b f r = b ( d i m f x + 1: end ) ;
124 x f x = b ( 1 : dim fx ) ;
125 i f dim fx
126 temp = ( x f x ’ ∗ A f x ) ’ ;
127 b f r = b f r −temp ;
128 end
129 py = T f r ’ \ b f r ;
130 G f r f r = G( 1 : d i m f r , 1 : d i m f r ) ;
131 g f r = g (1: dim fr ) ;
132 g z = −(( G f r f r ∗ (Q2∗py ) + g f r ) ’ ∗ Q1) ’ ;
133 z = L f r \ gz ;
134 pz = L f r ’ \ z ;
135 x f r = Q2∗py + Q1∗ pz ;
136 %com pute L a g r a n g i a n m u l t i p l i e r s
137 c = G∗ [ x f r ; x f x ] + g ;
138 c f r = c (1: dim fr ) ;
139 c f x = c ( d i m f r + 1: end ) ;
140 Y f r = Q f r ( 1 : d i m f r , dns + 1: d i m f r ) ;
141 u I = T fr \( c f r ’ ∗ Y fr ) ’ ;
142 u B = c f x −A f x ∗ u I ;
143 x new = [ x f r ; x f x ] ;
144 u new = [ u B ; u I ] ;

null space update fact app general FRFX.m

1 f u n c t i o n [ Q fr , T fr , L f r ] = n u l l s p a c e u p d a t e f a c t a p p g e n e r a l F R F X ( Q fr ,
T fr , L fr , col new )
2
3 % NULL SPACE UPDATE FACT APP GENERAL FRFX u p d a t e s t h e QT−f a c t o r i z a t i o n o f A
when a
4 % g e n e r a l c o n s t r a i n t : c o l n e w i s appended t o A a s t h e l a s t col um n . The
resulting
5 % c o n s t r a i n t m a t r i x i s Abar = [ A c o l n e w ] . The c o r r e s p o n d i n g QP p r o b l e m h a s a
reduced Hessian
6 % m a t r i x redH and t h e c h o l e s k y f a c t o r i z a t i o n o f redH i s L f r . I t i s o n l y
7 % t h e p a r t c o r r e s p o n d i n g t o t h e f r e e v a r i a b l e s w hi ch a r e u p d a t e d ( t h e f i x e d
8 % part are not changi ng )
9 % Call
10 % [ Q fr , T fr , L f r ] = n u l l s p a c e u p d a t e f a c t a p p g e n e r a l F R F X ( Q fr , T fr
, L fr , col new )
11 %
12 % Input parameters :
13 % Q f r and T f r : i s t h e QT−f a c t o r i z a t i o n o f A ( t h e p a r t
14 % corres pondi ng to the f r e e v a r i a b l e s )
15 % L fr : i s the chol es k y f a c t o r i z a t i o n o f the reduced Hessian
m a t r i x o f t h e c o r r e s p o n d i n g QP p r o b l e m .
16 % col new : i s t h e g e n e r a l c o n s t r a i n t t h a t i s appended t o A : Abar
= [A col new ]
17
18 % Output p a r a m e t e r s
19 % Q f r and T f r : is
t h e QT−f a c t o r i z a t i o n o f Abar ( t h e p a r t
20 % corres pondi ng to the f r e e v a r i a b l e s )
21 % L fr : i s the Cholesky f a c t o r i z a t i o n o f the reduced Hessian
matrix o f t h e new QP p r o b l e m .
22
23 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
24 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
25 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
26 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
27 % D ate : 08. february 2007.
28 % Reference : −−−−−−−−−−−−−−−−−−−−−
29
30 [ n ,m] = s i z e ( T f r ) ;
31 dns = n−m;
32 Z f r = Q f r ( : , 1 : dns ) ;
33 Y f r = Q f r ( : , dns + 1: end ) ;
34 T f r = T f r ( dns + 1: end , : ) ;
35 a f r = col new ( 1 : n ) ;
36 wv = ( a f r ’ ∗ Q f r ) ’ ;
37 w = wv ( 1 : dns ) ;
38 v = wv ( dns + 1: end ) ;
39 f o r i = 1: l e n g t h (w)−1
40 j = i + 1;
41 [ s , c ] = g i v e n s r o t a t i o n m a t r i x (w( i ) ,w( j ) ) ;
42
43 temp = w( i ) ∗ c + w( j ) ∗ s ;
44 w( j ) = −w( i ) ∗ s + w( j ) ∗ c ;
45 w( i ) = temp ;
D.4 Updating the Matrix Factorizations 151

46
47 temp = Z f r ( : , i ) ∗ c + Z f r ( : , j ) ∗ s ;
48 Z f r ( : , j ) = −Z f r ( : , i ) ∗ s + Z f r ( : , j ) ∗ c ;
49 Z f r ( : , i ) = temp ;
50
51 temp = L f r ( i , : ) ∗ c + L f r ( j , : ) ∗ s ;
52 L f r ( j , : ) = −L f r ( i , : ) ∗ s + L f r ( j , : ) ∗ c ;
53 L f r ( i , : ) = temp ;
54 end
55 gamma = w( end ) ;
56 T f r = [ z e r o s ( 1 , s i z e ( T f r , 2 ) ) gamma ; T f r v ] ;
57 L f r = L f r ( 1 : end − 1 , : ) ;
58 [ nn mm] = s i z e ( L f r ) ;
59 f o r i = 1 : 1 : nn
60 j= i + 1;
61 [ c , s ] = givens rotation matrix ( L fr ( i , i ) , L fr ( i , j ) ) ;
62
63 temp = L f r ( : , i ) ∗ c − L f r ( : , j ) ∗ s ;
64 L fr ( : , j ) = L fr ( : , i )∗s + L fr ( : , j )∗c ;
65 L f r ( : , i ) = temp ;
66 end
67 Q fr = [ Z f r Y fr ] ;
68 T f r = [ z e r o s ( dns −1 , s i z e ( T f r , 2 ) ) ; T f r ] ;
69 L f r = L f r ( : , 1 : dns −1) ;

null space update fact rem general FRFX.m

1 f u n c t i o n [ Q new , T new , L new ] = n u l l s p a c e u p d a t e f a c t r e m g e n e r a l F R F X ( Q f r ,

T fr , L fr , G frfr , j )
2
3 % NULL SPACE UPDATE FACT REM GENERAL FRFX u p d a t e s t h e QT−f a c t o r i z a t i o n
c o r r e s p o n d i n g t o t h e f r e e v a r i a b l e s o f A when a
4 % g e n e r a l c o n s t r a i n t i s rem ov ed from A a t column−i n d e x j . The new C o n s t r a i n t
m a t r i x i s c a l l e d Abar .
5 % The c o r r e s p o n d i n g QP p r o b l e m h a s a r e d u c e d H e s s i a n m a t r i x redH and t h e
cholesky f a c to ri z a ti o n
6 % o f redH i s L f r .
7
8 % Call
9 % [ Q new , T new , L new ] = n u l l s p a c e u p d a t e f a c t r e m c o l ( Q f r , T fr , L fr
, G frfr , col rem )
10
11 % Input parameters
12 % Q f r and T f r : i s t h e QT−f a c t o r i z a t i o n o f A ( t h e p a r t
13 % corres pondi ng to the f r e e v a r i a b l e s )
14 % L fr : i s the chol es k y f a c t o r i z a t i o n o f the reduced Hessian
matrix o f t h e c o r r e s p o n d i n g QP p r o b l e m ( t h e p a r t
15 % corres pondi ng to the f r e e v a r i a b l e s ) .
16 % G frfr : i s t h e H e s s i a n m a t r i x o f t h e QP p r o b l e m ( t h e p a r t
17 % corres pondi ng to the f r e e v a r i a b l e s ) .
18 % col rem : i s t h e column−i n d e x a t w hi ch a g e n e r a l c o n s t r a i n t h a s
b e e n rem ov ed from A
19
20 % Output p a r a m e t e r s
21 % Q new and T new : is
t h e QT−f a c t o r i z a t i o n o f Abar ( t h e p a r t
22 % corres pondi ng to the f r e e v a r i a b l e s ) .
23 % L new : i s the Cholesky f a c t o r i z a t i o n o f the reduced Hessian
matrix o f t h e new QP p r o b l e m ( t h e p a r t
24 % corres pondi ng to the f r e e v a r i a b l e s ) .
25
26 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
27 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
28 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
29 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
30 % D ate : 08. february 2007.
31 % Reference : −−−−−−−−−−−−−−−−−−−−−
32
33 [ n ,m] = s i z e ( T f r ) ;
34 dns = n−m;
35 T f r = T f r ( dns + 1: end , : ) ;
36 T11 = T f r (m−j + 2:m, 1 : j −1) ;
37 N = T f r ( 1 : m−j +1 , j + 1: end ) ;
38 M = T f r (m−j + 2: end , j + 1: end ) ;
39 Q1 = Q f r ( : , 1 : dns ) ;
40 Q21 = Q f r ( : , dns + 1:n−j +1) ;
41 Q22 = Q f r ( : , n−j + 2: end ) ;
42 [ nn mm] = s i z e (N) ;
43
44 for i = 1 : 1 :mm
152 Matlab-code

45 i d x 1 = nn−i ;
46 i d x 2 = i d x 1 + 1;
47 [ s , c ] = g i v e n s r o t a t i o n m a t r i x (N( i d x 1 , i ) ,N( i d x 2 , i ) ) ;
48
49 temp = N( i d x 1 , : ) ∗ c + N( i d x 2 , : ) ∗ s ;
50 N( i d x 2 , : ) = −N( i d x 1 , : ) ∗ s +N( i d x 2 , : ) ∗ c ;
51 N( i d x 1 , : ) = temp ;
52
53 temp = Q21 ( : , i d x 1 ) ∗ c + Q21 ( : , i d x 2 ) ∗ s ;
54 Q21 ( : , i d x 2 ) = −Q21 ( : , i d x 1 ) ∗ s + Q21 ( : , i d x 2 ) ∗ c ;
55 Q21 ( : , i d x 1 ) = temp ;
56 end
57 N = N ( 2 : end , : ) ;
58 T new = [ z e r o s ( nn −1 , j −1) N ; T11 M] ;
59 T new = [ z e r o s ( dns +1 ,m−1) ; T new ] ;
60 Q new = [ Q1 Q21 Q22 ] ;
61 z = Q new ( : , dns +1) ;
62 l = L f r \ ( ( G f r f r ∗ z ) ’ ∗ Q1) ’ ;
63 d e l t a = s q r t ( z ’ ∗ G f r f r ∗ z−l ’ ∗ l ) ;
64 L new = [ L f r z e r o s ( dns , 1 ) ; l ’ d e l t a ] ;

null space update fact app bound FRFX.m

1 f u n c t i o n [ Q fr , T fr , L f r ] = null space update fact app bound FRFX ( Q fr , T fr ,

L fr )
2
3 % NULL SPACE UPDATE FACT APP BOUND FRFX u p d a t e s t h e QT−f a c t o r i z a t i o n o f A when
a
4 % bound i s appended t o t h e c o n s t r a i n t m a t r i x . The c o r r e s p o n d i n g QP p r o b l e m h a s
a reduced Hessian
5 % m a t r i x redH and t h e c h o l e s k y f a c t o r i z a t i o n o f redH i s L f r . The
6 % QT−f a c t o r i z a t i o n c o r r e s p o n d t o t h e t h e g e n e r a l c o n s t r a i n t m a tr i x and o n l y
7 % the part corres pondi ng to the f r e e v a r i a b l e s .
8
9 % Call
10 % [ Q fr , T fr , L f r ] = null space update fact app bound FRFX ( Q fr , T fr ,
L fr )
11
12 % Input parameters
13 % Q f r and T f r : is t h e QT−f a c t o r i z a t i o n o f A, (A i s t h e g e n e r a l
14 % c o n s t r a i n t m a t r i x and o n l y t h e p a r t
15 % corres pondi ng to the f r e e v a r i a b l e s )
16 % L fr : i s the chol es k y f a c t o r i z a t i o n o f the reduced Hessian
matrix o f the c o r r e s p o n d i n g QP p r o b l e m .
17
18 % Output p a r a m e t e r s
19 % Q f r and T f r : i s t h e QT−f a c t o r i z a t i o n o f t h e g e n e r a l c o n s t r a i n t
20 % matrix f o r th e p art c o r r e s p o n d i n g to th e f r e e
variables .
21 % L fr : i s the Cholesky f a c t o r i z a t i o n o f the reduced H es s i an
m a t r i x o f t h e new QP p r o b l e m .
22
23 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
24 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
25 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
26 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
27 % D ate : 08. february 2007.
28 % Reference : −−−−−−−−−−−−−−−−−−−−−
29 [ n ,m] = s i z e ( T f r ) ;
30 dns = n−m;
31 q = Q f r ( end , : ) ’ ;
32 TL = z e r o s ( n ) ;
33 TL ( 1 : dns , 1 : dns ) = L f r ;
34 TL ( : , dns + 1: end ) = T f r ;
35 f o r i = 1: l e n g t h ( q )−1
36 j = i + 1;
37 [ s , c ] = givens rotation matrix (q( i ) ,q( j ) ) ;
38
39 temp = q ( i ) ∗ c + q ( j ) ∗ s ;
40 q ( j ) = −q ( i ) ∗ s + q ( j ) ∗ c ;
41 q ( i ) = temp ;
42
43 temp = Q f r ( : , i ) ∗ c + Q f r ( : , j ) ∗ s ;
44 Q f r ( : , j ) =− Q f r ( : , i ) ∗ s + Q f r ( : , j ) ∗ c ;
45 Q f r ( : , i ) = temp ;
46
47 temp = TL( i , : ) ∗ c + TL( j , : ) ∗ s ;
48 TL( j , : ) = −TL( i , : ) ∗ s + TL( j , : ) ∗ c ;
49 TL( i , : ) = temp ;
D.4 Updating the Matrix Factorizations 153

50 end
51
52 Q f r = Q f r ( 1 : end − 1 , 1 : end −1) ;
53 T f r = TL ( 1 : end −1 , dns + 1: end ) ;
54 L new = TL ( 1 : dns − 1 , 1 : dns ) ;
55 [ nn mm] = s i z e ( L new ) ;
56 f o r i = 1 : 1 : nn
57 j= i + 1;
58 [ c , s ] = g i v e n s r o t a t i o n m a t r i x ( L new ( i , i ) , L new ( i , j ) ) ;
59
60 temp = L new ( : , i ) ∗ c − L new ( : , j ) ∗ s ;
61 L new ( : , j ) = L new ( : , i ) ∗ s + L new ( : , j ) ∗ c ;
62 L new ( : , i ) = temp ;
63 end
64 L f r = L new ( : , 1 : end −1) ;

null space update fact rem bound FRFX.m

1 f u n c t i o n [ Q fr , T fr , L f r ] = null space update fact rem bound FRFX ( Q fr , T fr ,

L f r , G)
2
3 % NULL SPACE UPDATE FACT REM BOUND FRFX u p d a t e s t h e QT−f a c t o r i z a t i o n of the
g e n e r a l c o n s t r a i n t m a t r i x ( and o n l y t h e p a r t
4 % c o r r e s p o n d i n g t o t h e f r e e v a r i a b l e s ) when a bound i s rem ov ed .
5 % The c o r r e s p o n d i n g QP p r o b l e m h a s a r e d u c e d H e s s i a n m a t r i x redH and the
cholesky f a c to ri z a ti o n
6 % o f redH i s L o l d .
7
8 % Call
9 % [ Q fr , T fr , L f r ] = null space update fact rem bound FRFX ( Q fr , T fr ,
L f r , G)
10
11 % Input parameters
12 % Q f r and T f r : i s t h e QT−f a c t o r i z a t i o n o f t h e c o n s t r a i n t m a t r i x ( and
only the part corres pondi ng to the f r e e v a r i a b l e s ) .
13 % L fr : i s the chol es k y f a c t o r i z a t i o n o f the reduced Hessian
m a t r i x o f t h e c o r r e s p o n d i n g QP p r o b l e m .
14 % G : i s t h e H e s s i a n m a t r i x o f t h e QP p r o b l e m .
15
16 % Output p a r a m e t e r s
17 % Q f r and T f r : i s t h e QT−f a c t o r i z a t i o n o f t h e new g e n e r a l
18 % c o n s t r a i n t m a t r i x ( and o n l y t h e p a r t
corres pondi ng to the f r e e v a r i a b l e s )
19 % L fr : i s the Cholesky f a c t o r i z a t i o n o f the reduced Hessian
m a t r i x o f t h e new QP p r o b l e m .
20
21 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
22 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
23 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
24 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
25 % D ate : 08. february 2007.
26 % Reference : −−−−−−−−−−−−−−−−−−−−−
27
28 n = s i z e ( Q fr , 1 ) ;
29 m = s i z e ( T f r , 1 ) −1;
30 dns = n−m;
31 G f r f r = G( 1 : n , 1 : n ) ;
32 Z f r = Q f r ( : , 1 : dns ) ;
33 Y f r = Q f r ( : , dns + 1: end ) ;
34 [ nn mm] = s i z e ( T f r ) ;
35 Y fr = [ Y fr z e ro s ( s i z e ( Y fr , 1 ) ,1) ; z ero s (1 , s i z e ( Y fr , 2 ) ) 1 ] ;
36 f o r i = 1 : 1 :mm
37 i d x 1 = nn−i ;
38 i d x 2 = i d x 1 + 1;
39 [ s , c ] = g i v e n s r o t a t i o n m a t r i x ( T f r ( idx1 , i ) , T f r ( idx2 , i ) ) ;
40
41 temp = T f r ( i d x 1 , : ) ∗ c + T f r ( i d x 2 , : ) ∗ s ;
42 T f r ( i d x 2 , : ) = −T f r ( i d x 1 , : ) ∗ s +T f r ( i d x 2 , : ) ∗ c ;
43 T f r ( i d x 1 , : ) = temp ;
44
45 temp = Y f r ( : , i d x 1 ) ∗ c + Y f r ( : , i d x 2 ) ∗ s ;
46 Y f r ( : , i d x 2 ) =− Y f r ( : , i d x 1 ) ∗ s + Y f r ( : , i d x 2 ) ∗ c ;
47 Y f r ( : , i d x 1 ) = temp ;
48 end
49 T fr = T f r ( 2 : end , : ) ;
50 Z fr = [ Z fr ; zeros (1 , si z e ( Z fr ,2 ) ) ] ;
51 Q fr = [ Z fr Y fr ] ;
52 T fr = [ z e r o s ( s i z e ( Z f r , 2 ) +1 , s i z e ( T f r , 2 ) ) ; T f r ] ;
53 Z fr b a r = Q f r ( : , 1 : n−m+1) ;
154 Matlab-code

54 Z f r = Z f r b a r ( 1 : end − 1 , 1 : end −1) ;

55 z = Z f r b a r ( 1 : end −1 , end ) ;
56 r h o = Z f r b a r ( end , end ) ;
57 h = G( 1 : n , n+1) ;
58 omega = G( n +1 ,n+1) ;
59 l = L f r \ ( ( G f r f r ∗ z+r h o ∗h ) ’ ∗ Z f r ) ’ ;
60 d e l t a = s q r t ( z ’ ∗ ( G f r f r ∗ z +2∗ r h o ∗h )+omega ∗ r h o ∗ rho−l ’ ∗ l ) ;
61 L fr = [ L fr zeros ( s i ze ( L fr ,1) ,1) ; l ’ delta ] ;
D.5 Demos 155

D.5 Demos

QP demo.m

1 function QP demo ( method , f u n t o p l o t )

2
3 % QP DEMO I n t e r a c t i v e demonstration of the primal active set and
4 % the dual a c t i v e s e t m ethods .
5 %
6 % Call
7 % QP demo ( method , f u n t o p l o t )
8 %
9 % I n p u t parameter
10 % method : ’ p r i m a l ’ : D e m o n s t r a t i n g t h e p r i m a l a c t i v e s e t method .
11 % ’ dual ’ : D e m o n s t r a t i n g t h e d u a l a c t i v e s e t method .
12 % funtoplot : ’ objective ’ : P l o t t i n g the o b j e c t i v e fu nc t i o n .
13 % ’ l agrangi an ’ : P l o t t i n g the Lagrangian fu nc t i o n .
14 %
15 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 .
16 % Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
17 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n .
18 % M. Sc . , IMM, DTU, DK−2800 Lyngby .
19 % Supervisor : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r .
20 % Per Grove Thomsen , P r o f e s s o r .
21 % D ate : 0 7 . J une 2 0 0 7 .
22
23 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
24 % Check n a r g i n / n a r g o u t %
25 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
26 e r r o r ( nargchk ( 2 , 2 , n argi n ) )
27 e r r o r ( nargoutchk ( 0 , 0 , nargout ) )
28 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
29 % Check i n p u t %
30 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
31 % c h e c k method . . .
32 i f ˜ s t r c m p ( method , ’ p r i m a l ’ ) & ˜ s t r c m p ( method , ’ d u a l ’ )
33 e r r o r ( ’ Method must be ’ ’ p r i m a l ’ ’ o r ’ ’ d u a l ’ ’ . ’ )
34 end
35 % check f u n t o p l o t . . .
36 ftp = 0; % plot obj ective function
37 i f ˜ strcmp ( f u n t o p l o t , ’ o b j e c t i v e ’ ) & ˜ strcmp ( f u n t o p l o t , ’ l a g r a n g i a n ’ )
38 e r r o r ( ’ F u n t o p l o t must be ’ ’ o b j e c t i v e ’ ’ o r ’ ’ l a g r a n g i a n ’ ’ . ’ )
39 e l s e i f strcmp ( f u n t o p l o t , ’ l a g r a n g i a n ’ )
40 ftp = 1; % plot lagrangian
41 end
42 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
43 % S e t u p and run demo %
44 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
45 % S e t u p demo . . .
46 G = [1 0;0 1 ] ;
47 g = [0 0 ] ’;
48 A = [0.5 1;
49 0 1;
50 2 1.75;
51 3 −1;
52 1 0];
53 b = [3 1 8.5 3 2 . 2 ] ’ ;
54 % Run demo . . .
55 i f s t r c m p ( method , ’ p r i m a l ’ )
56 p r i m a l a c t i v e s e t d e m o (G, g , A’ , b , f t p )
57 else
58 d u a l a c t i v e s e t d e m o (G, g , A’ , b , f t p )
59 end
60 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
61 % Auxilery function ( s ) %
62 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
63 f u n c t i o n p r i m a l a c t i v e s e t d e m o (G, g , A , b , f t p )
64 % initialize ...
65 [ n ,m] = s i z e (A) ;
66 At = A ’ ;
67 w non = 1 : 1 :m;
68 w act = [ ] ;
69 mu = z e r o s (m, 1 ) ;
70 % i n i t i a l i z e options . . .
71 t o l = sqr t ( eps ) ;
72 it max = 100;
73 % i n i t i a l i z e c o u n t e r s and c o n t a i n e r s . . .
74 it = 0;
75 X = repm at ( z e r o s ( n , 1 ) , 1 , i t m a x ) ;
76 % plot . . .
77 x = a c t i v e s e t p l o t (G, At , g , b , [ ] , mu, w a c t , [ − 4 8; −4 8 ] , [ 2 0 20 50 100 t o l f t p ] ) ;
78 X( : , 1 ) = x ;
156 Matlab-code

79 % check f e a s i b i l i t y o f x . . .
80 i b = f i n d ( At∗x − b < − t o l ) ;
81 i f ˜ isempty ( i b )
82 d i s p ( [ ’ F o l l o w i n g c o n s t r a i n t ( s ) v i o l a t e d , b e c a u s e A∗x < b : ’ ] )
83 f p r i n t f ( [ ’ \b ’ , i n t 2 s t r ( i b ’ ) , ’ . \ n ’ ] )
84 e r r o r ( ’ S t a r t i n g p o i n t f o r p r i m a l a c t i v e s e t method i s n o t f e a s i b l e , run
demo a g a i n . ’ )
85 end
86 % iterate . . .
87 stop = 0 ;
88 w hi l e ˜ stop
89 it = i t + 1;
90 i f i t >= i t m a x
91 d i s p ( ’ No . o r i t e r a t i o n s s t e p s e x e e d e d . ’ )
92 s t o p = 2 ; % maximum no i t e r a t i o n s e x c e e d e d
93 end
94 % c a l l range / n u l l space procedure . . .
95 mu = z e r o s (m, 1 ) ;
96 [ p , m u act ] = n u l l s p a c e d e m o (G, A ( : , w a c t ) ,G∗x+g , z e r o s ( l e n g t h ( w a c t ) , 1 ) ) ;
97 mu( w a c t ) = m u act ;
98 % plot . . .
99 a c t i v e s e t p l o t (G, At , g , b , X ( : , 1 : i t ) ,mu, w a c t , [ − 4 8; −4 8 ] , [ 2 0 20 50 100 t o l
ftp ]) ;
100 d i s p ( ’ P r e s s any k ey t o c o n t i n u e . . . ’ )
101 pause
102 % check i f s o l u t i o n found . . .
103 i f norm ( p ) <= t o l
104 i f mu >= − t o l
105 s top = 1 ; % s o l u t i o n found
106 d i s p ( ’ S o l u t i o n f o u n d by p r i m a l a c t i v e s e t method , demo t e r m i n a t e d . ’
)
107 else
108 % com pute i n d e x j o f bound / c o n s t r a i n t t o be rem ov ed . . .
109 [ dummy , j ] = min (mu) ;
110 w a c t = w a c t ( f i n d ( w a c t ˜= j ) ) ; % rem ov e c o n s t r a i n t j from a c t i v e
set
111 w non = [ w non j ] ; % append c o n s t r a i n t j t o n o n a c t i v e s e t f u n c t i o n
112 end
113 else
114 % com pute s t e p l e n g t h and i n d e x j o f bound / c o n s t r a i n t t o be appended . . .
115 alpha = 1 ;
116 f o r app = w non
117 ap = At ( app , : ) ∗p ; % At ( app , : ) = A ( : , app ) ’
118 i f ap < − t o l
119 temp = ( b ( app ) − At ( app , : ) ∗ x ) / ap ;
120 i f −t o l < temp & temp < a l p h a
121 a l p h a = temp ; % s m a l l e s t s t e p l e n g t h
122 j = app ; % i n d e x j o f bound t o be appended
123 end
124 end
125 end
126 i f alpha < 1
127 % make c o n s t r a i n e d s t e p . . .
128 x = x + a l p h a ∗p ;
129 w a c t = [ w a c t j ] ; % append c o n s t r a i n t j t o a c t i v e s e t
130 w non = w non ( f i n d ( w non ˜= j ) ) ; % rem ov e c o n s t r a i n t j from
nonactive set
131 else
132 % make f u l l s t e p . . .
133 x = x + p;
134 end
135 end
136 X ( : , i t +1) = x ;
137 % % plot . . .
138 % i f ˜ stop
139 % %d i s p ( ’ P r e s s any k ey t o c o n t i n u e . . . ’ )
140 % %p a u s e
141 % a c t i v e s e t p l o t (G, At , g , b , X ( : , 1 : i t +1) ,mu, w a c t , [ − 4 8; −4 8 ] , [ 2 0 20 50
100 t o l f t p ] ) ;
142 % d i s p ( ’ P r e s s any k ey t o c o n t i n u e . . . ’ )
143 % pause
144 % end
145 end
146
147 f u n c t i o n d u a l a c t i v e s e t d e m o (G, g , A , b , f t p )
148 % initialize ...
149 [ n ,m] = s i z e (A) ;
150 C = A;
151 w non = 1 : 1 :m;
152 w act = [ ] ;
153 x = −G\ g ; x = x ( : ) ;
154 mu = z e r o s (m, 1 ) ;
155 % i n i t i a l i z e options . . .
156 t o l = sq r t ( eps ) ;
157 max itr = 100;
158 % i n i t i a l i z e c o u n t e r s and c o n t a i n e r s . . .
159 i t = 0;
D.5 Demos 157

160 it draw = 1;
161 X = repm at ( z e r o s ( n , 1 ) , 1 , m a x i t r ) ;
162 % plot . . .
163 a c t i v e s e t p l o t (G, C ’ , g , b , x , mu, w a c t , [ − 4 8; −4 8 ] , [ 2 0 20 50 100 t o l f t p ] ) ;
164 d i s p ( ’ P r e s s any k ey t o c o n t i n u e . . . ’ )
165 pause
166 X( : , 1 ) = x ;
167 % iterate . . .
168 stop = 0;
169 w hi l e ˜ stop
170 c = c o n s t r a i n t s (G, C ( : , w non ) , g , b ( w non ) , x , mu) ;
171 i f c >= − t o l ;
172 stop = 1 ;
173 %d i s p ( ’ STOP : a l l i n a c t i v e c o n s t r a i n t s >= 0 ’ )
174 d i s p ( ’ S o l u t i o n f o u n d by d u a l a c t i v e s e t method , demo t e r m i n a t e d . ’ )
175 else
176 % we f i n d t h e l e a s t n e g a t i v e v a l u e o f c
177 c r = max ( c ( f i n d ( c < −s q r t ( e p s ) ) ) ) ;
178 r = f i n d ( c == c r ) ;
179 r = r (1) ;
180 end
181 it = i t + 1;
182 i f i t >= m a x i t r
183 d i s p ( ’ No . o r i t e r a t i o n s s t e p s e x e e d e d ( o u t e r l o o p ) . ’ )
184 s t o p = 3 ; % maximum no i t e r a t i o n s e x c e e d e d
185 end
186 % iterate . . .
187 it2 = 0;
188 s t o p 2 = max ( 0 , s t o p ) ;
189 w hi l e ˜ stop2
190 it2 = it2 + 1;
191 i f i t 2 >= m a x i t r
192 d i s p ( ’ No . o r i t e r a t i o n s s t e p s e x e e d e d ( i n n e r l o o p ) . ’ )
193 stop = 3 ;
194 stop2 = stop ;
195 end
196 % c a l l range / n u l l space procedure . . .
197 [ p , v ] = n u l l s p a c e d e m o (G, C ( : , w a c t ) ,−C ( : , r ) , z e r o s ( l e n g t h ( w a c t ) , 1 ) ) ;
198 i f isempty ( v )
199 v = [];
200 end
201 arp = C ( : , r ) ’∗p ;
202 i f a b s ( a r p ) <= t o l % l i n e a r d e p e n d e n c y
203 i f v >= 0 % s o l u t i o n d o e s n o t e x i s t
204 d i s p ( ’ Probl em i s i n f e a s i b l e , demo t e r m i n a t e d . ’ )
205 stop = 2 ;
206 stop2 = stop ;
207 else
208 t = inf ;
209 f or k = 1: length (v)
210 i f v(k) < 0
211 temp = −mu( w a c t ( k ) ) / v ( k ) ;
212 i f temp < t
213 t = temp ;
214 rem = k ;
215 end
216 end
217 end
218 mu( w a c t ) = mu( w a c t ) + t ∗v ;
219 mu( r ) = mu( r ) + t ;
220 w a c t = w a c t ( f i n d ( w a c t ˜= w a c t ( rem ) ) ) ;
221 end
222 else
223 % s t e p s i z e in dual space . . .
224 t1 = i n f ;
225 for k = 1: length (v)
226 i f v(k) < 0
227 temp = −mu( w a c t ( k ) ) / v ( k ) ;
228 i f temp < t 1
229 t 1 = temp ;
230 rem = k ;
231 end
232 end
233 end
234 % s t e p s i z e i n primal space . . .
235 t 2 = −c o n s t r a i n t s (G, C ( : , r ) , g , b ( r ) , x , mu) / a r p ;
236 i f t 2 <= t 1
237 x = x + t 2 ∗p ;
238 mu( w a c t ) = mu( w a c t ) + t 2 ∗v ;
239 mu( r ) = mu( r ) + t 2 ;
240 w act = [ w act r ] ;
241 else
242 x = x + t 1 ∗p ;
243 mu( w a c t ) = mu( w a c t ) + t 1 ∗v ;
244 mu( r ) = mu( r ) + t 1 ;
245 w a c t = w a c t ( f i n d ( w a c t ˜= w a c t ( rem ) ) ) ;
246 end
158 Matlab-code

247 end
248 c r = c o n s t r a i n t s (G, C ( : , r ) , g , b ( r ) , x , mu) ;
249 i f c r > −t o l
250 stop2 = 1 ; % l e a v e s the inner w h i l e−l o o p but does not stop the
algorithm
251 end
252 it draw = it draw + 1;
253 X( : , it d r aw ) = x ;
254 %p l o t . . .
255 i f ˜ stop
256 a c t i v e s e t p l o t (G, C ’ , g , b , X ( : , 1 : i t d r a w ) , mu, w a c t , [ − 4 8; −4 8] ,[20 20
50 100 t o l f t p ]) ;
257 d i s p ( ’ P r e s s any k ey to conti nue . . . ’ )
258 pause
259 end
260 end % w h i l e
261 end % w h i l e
262
263 f u n c t i o n [ x , mu ] = n u l l s p a c e d e m o (G, A , g , b )
264 % initialize ...
265 [ n m] = s i z e (A) ;
266 % QR f a c t o r i z a t i o n o f A s o t h a t A = [ Y Z ] ∗ [ R 0 ] ’ . . .
267 [ Q, R ] = q r (A) ; % m atl ab ’ s i m p l e m e n t a t i o n
268 Y = Q ( : , 1 : m) ;
269 Z = Q ( : ,m+ 1:n ) ;
270 R = R ( 1 : m, : ) ;
271 Zt = Z ’ ;
272 % S o l v e f o r t h e r a n g e s p a c e com ponent py . . .
273 py = R’ \ b ;
274 % Compute t h e r e d u c e d g r a d i e n t . . .
275 g z = Zt ∗ (G∗ (Y∗py ) + g ) ;
276 % Compute t h e r e d u c e d H e s s i a n and com pute i t s C h o l e s k y factorization . . .
277 Gz = Zt ∗G∗Z ;
278 L = c h o l ( Gz ) ’ ;
279 % S o l v e f o r t h e n u l l s p a c e com ponent pz . . .
280 pz = L\−g z ;
281 pz = L ’ \ pz ;
282 % Compute t h e s o l u t i o n . . .
283 x = Y∗py + Z∗ pz ;
284 % Compute t h e L a g r a n g e m u l t i p l i e r s . . .
285 mu = R\ (Y’ ∗ ( G∗x + g ) ) ;
286
287 f u n c t i o n f = o b j e c t i v e (G, A , g , b , x , mu)
288 f = x ’ ∗G∗x + g ’ ∗ x ;
289
290 f u n c t i o n c = c o n s t r a i n t s (G, A , g , b , x , mu)
291 c = A’ ∗ x − b ;
292
293 f u n c t i o n l = l a g r a n g i a n (G, A , g , b , x , mu)
294 L = o b j e c t i v e (G, A , g , b , x , mu) − mu’ ∗ c o n s t r a i n t s (G, A , g , b , x , mu) ;

active set plot.m

1 function [ x , w a c t ] = a c t i v e s e t p l o t (G, A, g , b , x , mu, w a c t , D , o p t s )

2
3 % ACTIVE SET PLOT P l o t t i n g t h e o b j e c t i v e o r t h e L a g r a n g i a n f u n c t i o n and the
4 % c o n s t r a i n t s w i t h f e a s i b l e r e g i o n s . The c o n s t r a i n t s must on t h e form
5 % A∗x >= b . Can o n l y p l o t f o r t h r e e d i m e n s i o n s .
6 %
7 % Call
8 % a c t i v e s e t p l o t (G, A, g , b , x , mu, wa , D)
9 % a c t i v e s e t p l o t (G, A, g , b , x , mu, wa , D, opts )
10 % [ x , wa ] = a c t i v e s e t p l o t ( . . . )
11 %
12 % Input parameters
13 % G : The H e s s i a n o f t h e o b e j c t i v e f u n c t i o n .
14 % A : The c o n s t r a i n t m a t r i x o f s i z e mx2 , w here m i s t h e number o f
15 % constraints .
16 % g : C o e f f i c i e n t s o f l i n e a r term i n o b j e c t i v e f u n c t i o n .
17 % b : Righthandside o f c o n s t r a i n t s .
18 % x : S t a r t i n g p o i n t . I f x i s a m a t r i x o f s i z e 2 xn , n = 1 , 2 , 3 , . . . ,
19 % t h e n t h e i t e r a t i o n p a t h i s p l o t t e t . I f x i s empty , t h e u s e r w i l l
20 % be a s k e d t o e n t e r a s t a r t i n g p o i n t .
21 % mu : The L a g r a n g i a n m u l t i p l i e r s . I f mu i s empty , a l l m u l t i p l i e r s w i l l
22 % be s e t t o z e r o .
23 % wa : Working s e t l i s t i n g t h e a c t i v e c o n s t r a i n t s . I f wa i s empty , t h e n
24 % a c o n s t r a i n t w i l l be f o u n d a s a c t i v e , i f x i s w i t h i n a r a n g e o f
25 % opts ( 5) to that c o n s t r a i n t .
26 % D : Domain t o be p l o t t e t , g i v e n a s [ x1 ( 1 ) x1 ( 2 ) ; x2 ( 1 ) x2 ( 2 ) ] .
27 % op ts : V ector with s i x el em en ts .
D.5 Demos 159

28 % opts ( 1 : 2 ) : Number o f g r i d p o i n t s i n t h e f i r s t and s e c o n d

29 % direction .
30 % opts ( 3) : Number o f c o n t o u r l e v e l s .
31 % opts ( 4) : Number o f l i n e a r l y s p a c e d p o i n t s u s e d f o r p l o t t i n g
32 % the c o n s t r a i n t s .
33 % opts ( 5) : A c o n s t r a i n t w i l l be f o u n d a s a c t i v e , i f x i s
34 % within a range of opts ( 5) to that c o n s t r a i n t .
35 % opts ( 6) : 0 : Pl ot t i ng the contours of the o b j e c t i v e fu nc t i o n .
36 % 1 : Pl ot t i ng the contours of the Lagrangian f u nc t i o n .
37 % If opts not , then t h e d e f a u l t o p t s = [ 2 0 20 50 100 s q r t ( e p s ) 0 ] .
38 %
39 % Output p a r a m e t e r s
40 % x : Same a s i n p u t x . I f i n p u t x i s empty , t h e n t h e s t a r t i n g p o i n t
41 % e n t e r e d by t h e u s e r .
42 % w : Same a s i n p u t w a c t . I f i n p u t w a c t i s empty , t h e n t h e l i s t o f
43 % a c t i v e c o n s t r a i n t f o u n d upon t h e i n p u t / e n t e r e d s t a r t i n g p o i n t .
44 %
45 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
46 % In cours e : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
47 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
48 % Supervisor : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove Thomsen ,
49 % Professor .
50 % D ate : 28 t h J a n u a r y 2 0 0 7 .
51
52 % checking input . . .
53 e r r o r ( nargchk ( 8 , 9 , n argi n ) )
54 A = A’ ;
55 [ n ,m] = s i z e (A) ;
56 i f i s e m p t y (mu)
57 mu = z e r o s (m, 1 ) ;
58 end
59 [ u , v ] = s i z e (D) ;
60 i f u ˜= 2 | v ˜= 2
61 e r r o r ( ’ The domain must be a m a t r i x o f s i z e 2 x2 . ’ )
62 end
63 i f nargin > 8
64 [ u , v ] = s i z e ( opts ( : ) ) ;
65 i f u ˜= 6 | v ˜= 1
66 e r r o r ( ’ Opts must be a v e c t o r o f l e n g t h 6. ’ )
67 end
68 end
69
70 % d e f a u l t opts . . .
71 i f n argi n < 9 | isempty ( op ts )
72 o p t s = [ 2 0 20 20 20 s q r t ( e p s ) 0 ] ; %[ 2 0 20 50 100 s qr t ( eps ) 0];
73 end
74
75 % fu n ct i o n to pl o t . . .
76 fun = @ o b j e c t i v e ;
77 i f opts ( 6)
78 fun = @ lagrangian ;
79 end
80
81 % i n t e r n a l parameters . . .
82 f s i z e = 12; % font s iz e
83
84 % pl ot the contours of the o b j e c t i v e or the Lagrangian function . . .
85 f i gu re (1) , c l f
86 c o n t p l o t ( fun , G, A , g , b , mu, D , o p t s )
87 x l a b e l ( ’ x 1 ’ , ’ FontSize ’ , f s i z e )
88 y l a b e l ( ’ x 2 ’ , ’ FontSize ’ , f s i z e )
89 h o l d on
90
91 % pl ot the c o n s t r a i n t s . . .
92 i f n argou t & isempty ( x )
93 c o n s t p l o t ( @ c o n s t r a i n t s , G, A, g , b , mu, D, w a c t , m, o p t s , f s i z e )
94 % t i t l e ( ’ x = ( , ) , f ( x ) = , W a = [ ] , \mu = [ ] ’ , ’ F o n t S i z e ’ , f s i z e )
95 % ask u s e r to e n t e r s t a r t i n g p o i n t . . .
96 w h i l e isempty ( x )
97 d i s p ( ’ L e f t c l i c k on p l o t t o s e l e c t s t a r t i n g p o i n t o r p r e s s any k ey t o
enter st a r t in g point in console . ’ )
98 [ u , v , but ] = g i n p u t ( 1 ) ;
99 i f but == 1
100 x = [u v ];
101 else
102 w h i l e i s e m p t y ( x ) | l e n g t h ( x ) ˜= 2 | sum ( i s n a n ( x ) ) | sum ( i s i n f ( x ) ) |
sum ( ˜ i s r e a l ( x ) ) | i s c h a r ( x )
103 x = i n p u t ( ’ E n t e r s t a r t i n g p o i n t [ x1 x2 ] : ’ ) ;
104 end
105 end
106 end
107 x = x(:) ;
108 f ig u r e (1)
109 % find active constraints . . .
110 i f nargout > 1
111 w a c t = f i n d ( a b s (A ( 2 , : ) ’ ∗ x ( 2 ) + f e v a l ( @ c o n s t r a i n t s , G, A ( 1 , : ) , g , b , x ( 1 ) ,mu
) ) <= o p t s ( 5 ) ) ’ ; % A( 2 ) ∗ x2 + (A( 1 ) ∗x1 − b ) <= e p s
160 Matlab-code

112 i f w act
113 c o n s t p l o t ( @ c o n s t r a i n t s , G, A , g , b , mu, D , w a c t , m, o p t s , f s i z e )
114 end
115 t i t l e ( [ ’ x = ( ’ , num 2s tr ( x ( 1 , end ) , 2 ) , ’ , ’ , num 2s tr ( x ( 2 , end ) , 2 ) , ’ ) , f ( x ) =
’ , num 2s tr ( o b j e c t i v e (G, A, g , b , x ( : , end ) ,mu) , 2 ) , ’ , W a = [ ’ , i n t 2 s t r (
w a c t ) , ’ ] , \mu = [ ’ , num 2s tr (mu’ , 2 ) , ’ ] ’ ] , ’ F o n t S i z e ’ , f s i z e )
116 end
117 else
118 %i f isempty ( w act )
119 % w a c t = f i n d ( a b s (A ( 2 , : ) ’ ∗ x ( 2 ) + f e v a l ( @ c o n s t r a i n t s , G, A ( 1 , : ) , g , b , x ( 1 ) ,
mu) ) <= o p t s ( 5 ) ) ’ ; % A( 2 ) ∗ x2 + (A( 1 ) ∗ x1 − b ) <= e p s
120 %end
121 c o n s t p l o t ( @ c o n s t r a i n t s , G, A, g , b , mu, D , w a c t , m, o p t s , fsize )
122 t i t l e ( [ ’ x = ( ’ , num 2s tr ( x ( 1 , end ) , 2 ) , ’ , ’ , num 2s tr ( x ( 2 , end ) , 2 ) , ’ ) , f ( x ) = ’ ,
num 2s tr ( o b j e c t i v e (G, A , g , b , x ( : , end ) ,mu) , 2 ) , ’ , W a = [ ’ , i n t 2 s t r ( w act ) , ’
] , \mu = [ ’ , num 2s tr (mu’ , 2 ) , ’ ] ’ ] , ’ F o n t S i z e ’ , fsize )
123 end
124
125 % p l o t th e path . . .
126 pathplot (x)
127 hold o f f
128
129 f u n c t i o n c o n t p l o t ( fun , G, A , g , b , mu, D, o p t s )
130 [ X1 , X2 ] = m e s h g r i d ( l i n s p a c e (D( 1 , 1 ) ,D ( 1 , 2 ) , o p t s ( 1 ) ) , l i n s p a c e (D( 2 , 1 ) ,D( 2 , 2 ) , o p t s
(2) ) ) ;
131 F = z e r o s ( opts ( 1 : 2 ) ) ;
132 f o r i = 1: opts (1 )
133 f o r j = 1 : opts ( 2)
134 F ( i , j ) = norm ( f e v a l ( fun , G, A , g , b , [ X1 ( i , j ) ; X2 ( i , j ) ] , mu) , 2 ) ;
135 end
136 end
137 c o n t o u r ( X1 , X2 , F , o p t s ( 3 ) )
138
139 f u n c t i o n c o n s t p l o t ( fun , G, A , g , b , mu, D , w a c t , m, o p t s , f s i z e )
140 f c o l o r = [ . 4 . 4 . 4 ] ; f a l p h a = . 4 ; % c o l o r and a l p h a v a l u e s o f f a c e s m ark i ng
unfeasable region
141 b c o l o r = [ . 8 . 8 . 8 ] ; % b a c k g r o u n d c o l o r o f c o n s t r a i n t num beri ng
142 x1 = l i n s p a c e (D ( 1 , 1 ) ,D( 1 , 2 ) , o p t s ( 4 ) ) ;
143 x2 = l i n s p a c e (D ( 2 , 1 ) ,D( 2 , 2 ) , o p t s ( 4 ) ) ;
144 C = z e r o s (m, o p t s ( 4 ) ) ;
145 f o r j = 1: opts (4 )
146 f o r i = 1 :m
147 i f A( 2 , i ) % i f A( 2 ) ˜= 0
148 C( i , j ) = − f e v a l ( fun , G, A( 1 , i ) , g , b ( i ) , x1 ( j ) ,mu) /A( 2 , i ) ; % x2 = −(A( 1 )
∗ x1 − b ) /A( 2 )
149 else
150 C( i , j ) = b ( i ) /A( 1 , i ) ; % A( 2 ) = 0 => x1 = b/A( 1 ) , must be p l o t t e t
r e v e r s e l y a s (C( i , : ) , x2 )
151 end
152 end
153 end
154 f o r i = 1 :m
155 i f any ( i == w a c t )
156 l w i d t h = 1 ; c o l o r = [ 1 0 0 ] ; % l i n e w i d t h and c o l o r o f a c t i v e
constraints
157 else
158 l w i d t h = 1 ; c o l o r = [ 0 0 0 ] ; % l i n e w i d t h and c o l o r o f i n a c t i v e
constraints
159 end
160 i f A( 2 , i ) % i f A( 2 ) ˜= 0
161 i f A( 2 , i ) > 0 % i f A( 2 ) > 0
162 f i l l ( [ D( 1 , 1 ) D ( 1 , 2 ) D( 1 , 2 ) D( 1 , 1 ) ] , [ C( i , 1 ) C( i , end ) min (D( 2 , 1 ) ,C( i ,
end ) ) min (D( 2 , 1 ) ,C( i , 1 ) ) ] , f c o l o r , ’ FaceA l pha ’ , f a l p h a )
163 else
164 f i l l ( [ D( 1 , 1 ) D ( 1 , 2 ) D( 1 , 2 ) D( 1 , 1 ) ] , [ C( i , 1 ) C( i , end ) max (D( 2 , 2 ) ,C( i ,
end ) ) max (D( 2 , 2 ) ,C( i , 1 ) ) ] , f c o l o r , ’ FaceA l pha ’ , f a l p h a )
165 end
166 p l o t ( x1 , C( i , : ) , ’− ’ , ’ Li neWi dth ’ , l w i d t h , ’ C o l o r ’ , c o l o r )
167 i f C( i , 1 ) < D( 1 , 1 )% | C( i , 1 ) < D( 2 , 1 )
168 t e x t (− f e v a l ( fun , G, A( 2 , i ) , g , b ( i ) ,D ( 2 , 1 ) ,mu) /A( 1 , i ) ,D( 2 , 1 ) , i n t 2 s t r ( i )
, ’ C ol or ’ , ’ k ’ , ’ EdgeColor ’ , c o l o r , ’ BackgroundColor ’ , b col or , ’
F o n t S i z e ’ , f s i z e ) % x1 = −(A( 2 ) ∗ x2 − b ) /A( 1 )
169 else
170 i f C( i , 1 ) > D ( 2 , 2 )
171 t e x t (− f e v a l ( fun , G, A( 2 , i ) , g , b ( i ) ,D( 2 , 1 ) ,mu) /A( 1 , i ) ,D ( 2 , 1 ) ,
i n t 2 s t r ( i ) , ’ C ol or ’ , ’ k ’ , ’ EdgeColor ’ , c o l o r , ’ BackgroundColor ’
, b c o l o r , ’ F o n t S i z e ’ , f s i z e ) % x1 = −(A( 2 ) ∗ x2 − b ) /A( 1 )
172 else
173 t e x t (D( 1 , 1 ) ,C( i , 1 ) , i n t 2 s t r ( i ) , ’ C o l o r ’ , ’ k ’ , ’ E d g e C o l o r ’ , c o l o r , ’
BackgroundColor ’ , b col or , ’ F on tS i ze ’ , f s i z e )
174 end
175 end
176 else
177 i f A( 1 , i ) > 0 % i f A( 1 ) > 0
178 f i l l ( [ D( 1 , 1 ) C( i , 1 ) C( i , end ) D( 1 , 1 ) ] , [ D( 2 , 1 ) D( 2 , 1 ) D ( 2 , 2 ) D( 2 , 2 ) ] ,
f c o l o r , ’ FaceA l pha ’ , f a l p h a )
179 else
D.5 Demos 161

180 f i l l ( [ C( i , 1 ) D( 1 , 2 ) D ( 1 , 2 ) C( i , end ) ] , [ D( 2 , 1 ) D( 2 , 1 ) D ( 2 , 2 ) D( 2 , 2 ) ] ,
f c o l o r , ’ FaceA l pha ’ , f a l p h a )
181 end
182 p l o t (C( i , : ) , x2 , ’− ’ , ’ Li neWi dth ’ , l w i d t h , ’ C o l o r ’ , c o l o r )
183 t e x t (C( i , 1 ) ,D( 2 , 1 ) , i n t 2 s t r ( i ) , ’ C o l o r ’ , ’ k ’ , ’ E d g e C o l o r ’ , c o l o r , ’
BackgroundColor ’ , b col or , ’ F on tS i ze ’ , f s i z e )
184 end
185 end
186
187 function pathplot (x)
188 lwidth = 2 ; msize = 6 ;
189 p l o t ( x ( 1 , 1 ) , x ( 2 , 1 ) , ’ ob ’ , ’ Li neWi dth ’ , l w i d t h , ’ M a r k e r s i z e ’ , m s i z e ) % s t a r t i n g
position
190 p l o t ( x ( 1 , : ) , x ( 2 , : ) , ’ Li neWi dth ’ , l w i d t h ) % p a t h
191 p l o t ( x ( 1 , end ) , x ( 2 , end ) , ’ og ’ , ’ Li neWi dth ’ , l w i d t h , ’ M a r k e r s i z e ’ , m s i z e ) % c u r r e n t
position
192
193 f u n c t i o n f = o b j e c t i v e (G, A , g , b , x , mu)
194 f = 0 . 5 ∗ x ’ ∗G∗x + g ’ ∗ x ;
195
196 f u n c t i o n c = c o n s t r a i n t s (G, A, g , b , x , mu)
197 c = A’ ∗ x − b ;
198
199 f u n c t i o n l = l a g r a n g i a n (G, A , g , b , x , mu)
200 l = o b j e c t i v e (G, A , g , b , x , mu) − mu’ ∗ c o n s t r a i n t s (G, A, g , b , x , mu) ;

quad tank demo.m

1 function q u a d t a n k d e m o ( t , N , r , F , dF , gam , w , pd )
2
3 % QUAD TANK DEMO D e m o n s t r a t i o n o f t h e q u a d r u p l e t a n k p r o c e s s . The w a t e r
4 % l e v e l s i n t a n k 1 and 2 a r e c o n t r o l l e d a c c o r d i n g t o t h e s e t p o i n t s . The
5 % h e i g h t s o f a l l f o u r t a n k s a r e 50 cm . The w o r k s p a c e i s s a v e d a s
6 % ’ q u a d r u p l e t a n k p r o c e s s . mat ’ i n c u r r e n t d i r e c t o r y , s o i t i s p o s s i b l e t o
7 % run t h e a n i m a t i o n a g a i n by c a l l i n g q u a d t a n k a n i m a t e w i t h o u t r e c o m p u t i n g
8 % the setup .
9 % NOTE: A new c a l l o f q u a d t a n k d e m o w i l l o v e r w r i t e t h e s a v e d w o r k s p a c e
10 % ’ q u a d r u p l e t a n k p r o c e s s . mat ’ . The f i l e must be d e l e t e d m a n u a l l y .
11 %
12 % Call
13 % q u a d t a n k d e m o ( t , N, r , F , dF , gam , pd )
14 %
15 % Input parameters
16 % t : [ min ] S i m u l a t i o n t i m e o f t a n k p r o c e s s . 1 <= t <= 3 0 . D e f a u l t i s
17 % 5 . The t i m e i s p l o t t e t a s s e c o n d s . The l a s t d i s c r e t e p o i n t i s n o t
18 % animated / p l o t t e t .
19 % N : D i s c r e t i z a t i o n o f t . 5 <= N <= 1 0 0 , must be an i n t e g e r . D e f a u l t
20 % i s 1 0 . Number o f v a r i a b l e s i s 6∗N and number o f c o n s t r a i n t s i s
21 % 24∗N .
22 % r : [ cm ] S e t p o i n t s o f t a n k 1 and 2 . 0 <= r ( i ) <= 5 0 . D e f a u l t i s
23 % [30 3 0] .
24 % F : [ l / min ] Max f l o w r a t e s o f pump 1 and 2 . 0 <= F ( i ) <= 1 0 0 0 .
25 % Default i s [500 50 0 ].
26 % dF : [ l / min ˆ 2 ] Min/max c h a n g e i n f l o w r a t e s o f pump 1 and 2 . −100 <=p o i n t
27 % dF ( i ) <= 1 0 0 . D e f a u l t i s [ pump1 pump2 ] = [ −50 50 −50 5 0 ] .
28 % gam : F r a c t i o n o f f l o w from pump 1 and 2 g o i n g d i r e c t l y t o t a n k 1 and
29 % 2 . 0 <= gam ( i ) <= 1 . D e f a u l t i s [ 0 . 4 5 0 . 4 0 ] .
30 % w : S e t t i n g p r i o r i t y o f c o n t r o l l i n g w a t e r l e v e l i n t a n k 1 and 2
31 % r e l a t i v e t o one a n o t h e r . 1 <= w( i ) <= 1 0 0 0 . D e f a u l t i s [ 1 1 ] .
32 % pd : 1 : U s i n g p r i m a l a c t i v e s e t method , 2 : U s i n g d u a l a c t i v e s e t
33 % method . D e f a u l t i s 2 .
34 % I f i n p u t p a r a m e t e r s a r e empty , t h e n d e f a u l t v a l u e s a r e u s e d .
35
36 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
37 % Check n a r g i n / n a r g o u t %
38 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
39 e r r o r ( nargchk ( 7 , 8 , n argi n ) )
40 e r r o r ( nargoutchk ( 0 , 0 , nargout ) )
41 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
42 % Check i n p u t %
43 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
44 % Check t . . .
45 i f isempty ( t )
46 t = 5 ∗ 6 0 ; % t = min∗ s e c
47 else
48 t = c h e c k i n p u t ( t , 1 , 3 0 , 1 ) ∗ 6 0 ; % t = min∗ s e c
49 end
50 % Check N . . .
51 i f i s e m p t y (N)
52 N = 10;
162 Matlab-code

53 else
54 i f mod (N, 1 )
55 e r r o r ( ’N must be an i n t e g e r . ’ )
56 end
57 N = c h e c k i n p u t (N, 5 , 1 0 0 , 1 ) ;
58 end
59 % Check r . . .
60 i f isempty ( r )
61 r = [30 30 ];
62 else
63 r = check input ( r ,0 ,50 , 2) ;
64 end
65 % Check F . . .
66 i f isempty (F)
67 F = [500 500] ;
68 else
69 F = c h e c k i n p u t (F , 0 , 1 0 0 0 , 2 ) ;
70 end
71 % Check dF . . .
72 i f i s e m p t y ( dF )
73 dF = [ −50 50 −50 5 0 ] ;
74 else
75 dF = c h e c k i n p u t ( dF , − 1 0 0 , 1 0 0 , 4 ) ;
76 end
77 % Check gam . . .
78 i f i s e m p t y ( gam )
79 gam = [ 0 . 4 5 0 . 4 ] ;
80 else
81 gam = c h e c k i n p u t ( gam , 0 , 1 , 2 ) ;
82 end
83 % Check w . . .
84 i f i s e m p t y (w)
85 w = [1 1 ] ;
86 else
87 w = c h e c k i n p u t (w, 1 , 1 0 0 0 , 2 ) ;
88 end
89 % Check pd . . .
90 i f n a r g i n < 8 | i s e m p t y ( pd )
91 pd = 2 ;
92 else
93 i f mod ( pd , 1 )
94 e r r o r ( ’ pd must be an i n t e g e r . ’ )
95 end
96 pd = c h e c k i n p u t ( pd , 1 , 2 , 1 ) ;
97 end
98 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
99 % Startup i n f o %
100 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
101 i f N >= 30
102 c o n t = ’ do ’ ;
103 w h i l e ˜ strcmp ( l ow er ( con t ) , ’ y ’ ) & ˜ strcmp ( l ow er ( con t ) , ’ n ’ )
104 c o n t = i n p u t ( ’N >= 3 0 , s o c o m p u t a t i o n a l t i m e w i l l be s e v e r a l m i n u t e s ,
do you want t o c o n t i n u e ? y /n [ y ] : ’ , ’ s ’ ) ;
105 i f isempty ( con t )
106 cont = ’ y ’ ;
107 f p r i n t f ( ’ \b ’ )
108 disp ( ’y ’ )
109 end
110 %d i s p ( ’ ’ )
111 end
112 i f c o n t == ’ n ’
113 d i s p ( ’ S i m u l a t i o n t e r m i n a t e d by u s e r . ’ )
114 return
115 end
116 end
117 d i s p ( [ ’N = ’ , i n t 2 s t r (N) , ’ , s o number o f v a r i a b l e s i s ’ , i n t 2 s t r ( 6 ∗N) , ’ and
number o f c o n s t r a i n t s i s ’ , i n t 2 s t r ( 2 4 ∗ N) , ’ . ’ ] )
118 d i s p ( [ ’ t = ’ , num 2s tr ( t , 2 ) , ’ , gam = [ ’ , num 2s tr ( gam , 2 ) , ’ ] , w = [ ’ , num 2s tr (w , 2 ) , ’
]. ’ ])
119 d i s p ( [ ’ r = [ ’ , num 2s tr ( r ) , ’ ] , F = [ ’ , num 2s tr (F , 2 ) , ’ ] , dF = [ ’ , num 2s tr ( dF , 2 ) , ’ ] . ’
])
120 d i s p ( ’ Computing s i m u l a t i o n , p l e a s e w a i t . . . ’ )
121 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
122 % S e t u p demo %
123 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
124 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
125 % UI %
126 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
127 % physics . . .
128 g = 5 ; % g r a v i t y i s s m a l l due t o l i n e a r i z e d s y s t e m
129 % t i m e s pan and number o f s a m p l i n g p o i n t s . . .
130 t s p a n = [ 0 t ] ; %3 6 0 ] ;
131 % weights m atri ces . . .
132 Q = [ w ( 1 ) 0 ; 0 w ( 2 ) ] ; % w e i g h t m a t r i x , u s e d i n Q−norm , s e t t i n g p r i o r i t y o f h1
and h2 r e l a t i v e t o e a c h o t h e r
133 Hw = 1 e 6 ; % w e i g h i n g h1 and h2 (= Hw) i n r e l a t i o n t o h3 , h4 , u1 and u2 (= 1 )
134 % pump 1 . . .
D.5 Demos 163

135 Fmin1 = 0 ; Fmax1 = F ( 1 ) ; % minmax f l o w s

136 dFmin1 = dF ( 1 ) ; dFmax1 = dF ( 2 ) ; % minmax r a t e of change in flow
137 F10 = 0 ; % i n i t i a l v a l u e
138 % pump 2 . . .
139 Fmin2 = 0 ; Fmax2 = F ( 2 ) ; % minmax f l o w s
140 dFmin2 = dF ( 3 ) ; dFmax2 = dF ( 4 ) ; % minmax r a t e of change in flow
141 F20 = 0 ; % i n i t i a l v a l u e
142 % valve 1 . . .
143 gam1 = gam ( 1 ) ;
144 % valve 2 . . .
145 gam2 = gam ( 2 ) ;
146 % tank 1 . . .
147 r1 = r ( 1) ; % s e t poi nt
148 hmin1 = 0 ; hmax1 = 5 0 ; % minmax h e i g h t s
149 h10 = 0 ; % i n i t i a l v a l u e
150 % tank 2 . . .
151 r2 = r ( 2) ; % s e t poi nt
152 hmin2 = 0 ; hmax2 = 5 0 ; % minmax h e i g h t s
153 h20 = 0 ; % i n i t i a l v a l u e
154 % tank 3 . . .
155 hmin3 = 0 ; hmax3 = 5 0 ; % minmax h e i g h t s
156 h30 = 0 ; % i n i t i a l v a l u e
157 % tank 4 . . .
158 hmin4 = 0 ; hmax4 = 5 0 ; % minmax h e i g h t s
159 h40 = 0 ; % i n i t i a l v a l u e
160
161 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
162 % In i tiat e variables %
163 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
164 % pumps . . .
165 umin = [ Fmin1 Fmin2 ] ’ ;
166 umax = [ Fmax1 Fmax2 ] ’ ;
167 dumin = [ dFmin1 dFmin2 ] ’ ;
168 dumax = [ dFmax1 dFmax2 ] ’ ;
169 % valves . . .
170 gam = [ gam1 gam2 ] ;
171 % tanks . . .
172 bmin = [ hmin1 hmin2 hmin3 hmin4 ] ’ ;
173 bmax = [ hmax1 hmax2 hmax3 hmax4 ] ’ ;
174 % set points . . .
175 r = [ r1 r2 ] ’ ;
176 % i n i t i a l values . . .
177 x0 = [ h10 h20 h30 h40 ] ’ ;
178 %u0 = [ F10 F20 ] ’ ;
179 u0m i nus 1 = [ F10 F20 ] ’ ;
180 Q = Hw∗Q; % w e i g h t m a t r i x , u s e d i n Q−norm , s e t t i n g p r i o r i t y o f h1 and h2
r e l a t i v e to each o t h e r
181 d t = ( t s p a n ( 2 ) − t s p a n ( 1 ) ) /N ;
182
183 nx = l e n g t h ( x0 ) ;
184 nu = l e n g t h ( u0m i nus 1 ) ;
185
186 a1 = 1.2272;
187 a2 = 1.2272;
188 a3 = 1.2272;
189 a4 = 1.2272;
190 A1 = 380.1327;
191 A2 = 380.1327;
192 A3 = 380.1327;
193 A4 = 380.1327;
194
195 Ac = 2∗ g ∗[ − a1 /A1 0 a3 /A1 0 ;
196 0 −a2 /A2 0 a4 /A2 ;
197 0 0 −a3 /A3 0 ;
198 0 0 0 −a4 /A4 ] ;
199
200 Bc = [ gam1 /A1 0 ;
201 0 gam2 /A2 ;
202 0 ( 1 − gam2 ) /A3 ;
203 ( 1 − gam1 ) /A4 0 ] ;
204
205 Cc = [ 1 0 0 0 ;
206 0 1 0 0];
207
208 % %######################################################
209 % % b u i l d t h e o b j e c t f u n c t i o n H e s s i a n and g r a d i e n t :
210 % %######################################################
211
212 Qx = d t ∗Cc ’ ∗Q∗Cc ; % s h o u l d t h i s hav e non−z e r o
elements i n the d i o g o n a l ??
213 Qx = add2mat ( Qx , e y e ( 2 ) , 3 , 3 , ’ r e p ’ ) ; % => ASSURES THAT HESSIAN I S POSITIVE
DEFINITE
214 Qu = e y e ( nu ) ; % o n l y p u r p o s e i s t o make d i m e n s i o n s f i t
and t o r e m a i n p o s i t i v e d e f i n i t e
215 %Qu = z e r o s ( nu ) ;
216 qx = −d t ∗Cc ’ ∗Q∗ r ; % qk i n t e x t
164 Matlab-code

217 qu = z e r o s ( 2 , 1 ) ; % only purpose is t o make

dimensions f i t
218
219 H = z e r o s (N∗ ( nx+nu ) ) ; % Hessian
220 g = z e r o s (N∗ ( nx+nu ) , 1 ) ; % gradient
221 f o r i = 1:N
222 % i f i > f l o o r (N/ 2 )
223 % r ( 2 ) = 20
224 % qx = −d t ∗Cc ’ ∗Q∗ r
225 % end
226 j = 1+( i −1) ∗ ( nu+nx ) ;
227 H = add2mat (H , Qu, j , j , ’ r e p ’ ) ;
228 g = add2mat ( g , qu , j , 1 , ’ r e p ’ ) ;
229 H = add2mat (H , Qx , j+nu , j+nu , ’ r e p ’ ) ;
230 g = add2mat ( g , qx , j+nu , 1 , ’ r e p ’ ) ;
231 end
232
233 % ############################################################################
234 % % B u i l d A c ( g e n e r a l c o n s t r a i n t m a t r i x ) , u p p e r and l o w e r bounds f o r
235 % % g e n e r a l c o n s t r a i n t s ( b l and bu ) and u p p e r and l o w e r bounds f o r
236 % % v a r i a b l e s ( u and l )
237 % % ###########################################################################
238
239 I x = e y e ( nx ) ;
240 I u = e y e ( nu ) ;
241 A = I x + d t ∗Ac ;
242 B = d t ∗Bc ;
243 Ax0 = A∗ x0 ;
244 z e r o x = z e r o s ( nx , 1 ) ;
245 z e r o u = z e r o s ( nu , 1 ) ;
246
247 n = N∗ ( nx+nu ) ; % number o f v a r i a b l e s
248 m = N∗ ( nx+nu ) ; % number o f g e n e r a l c o n s t r a i n t s
249 A c = z e r o s (m, n ) ; % new A m a t r i x ( c a r s t e n ) ( g e n e r a l c o n s t r a i t n s are row s
=> we w i l l t r a n s p o s e it later )
250 l = zeros (n ,1 ) ; % l o w e r bounds f o r v a r i a b l e s
251 u = zeros (n ,1 ) ; % u p p e r bounds f o r v a r i a b l e s
252 bl = z e r o s (m, 1 ) ; % l o w e r bounds f o r g e n e r a l c o n s t r a i n t s
253 bu = z e r o s (m, 1 ) ; % u p p e r bounds f o r g e n e r a l c o n s t r a i n t s
254
255 row = 1 ;
256 col = 1;
257 A c = add2mat ( A c , B , row , c o l , ’ r e p ’ ) ;
258 A c = add2mat ( A c ,− I x , row , c o l +nu , ’ r e p ’ ) ;
259 b l = add2mat ( b l ,−Ax0 , row , 1 , ’ r e p ’ ) ;
260 bu = add2mat ( bu , Ax0 , row , 1 , ’ r e p ’ ) ;
261 f o r i = 1:N−1
262 row = 1+ i ∗nx ; % s t a r t row f o r new k
263 c o l = 3+( i −1) ∗ ( nx+nu ) ; % s t a r t col um n f o r new k
264 A c = add2mat ( A c , A, row , c o l , ’ r e p ’ ) ;
265 A c = add2mat ( A c , B , row , c o l +nx , ’ r e p ’ ) ;
266 A c = add2mat ( A c ,− I x , row , c o l +nx+nu , ’ r e p ’ ) ;
267 b l = add2mat ( b l , z e r o x , row , 1 , ’ r e p ’ ) ;
268 bu = add2mat ( bu , z e r o x , row , 1 , ’ r e p ’ ) ;
269 end
270
271 row = N∗nx + 1;
272 A c = add2mat ( A c , I u , row , 1 , ’ r e p ’ ) ;
273 b l = add2mat ( b l , dumin+u0m i nus 1 , row , 1 , ’ r e p ’ ) ;
274 bu = add2mat ( bu , dumax−u0m i nus 1 , row , 1 , ’ r e p ’ ) ;
275 f o r i = 1:N−1
276 row = row+nu ;
277 c o l = 1+( i ) ∗ ( nu+nx ) ;
278
279 A c = add2mat ( A c , I u , row , col , ’ rep ’ ) ;
280 A c = add2mat ( A c ,− I u , row , c o l −(nx+nu ) , ’ r e p ’ ) ;
281 b l = add2mat ( b l , dumin , row , 1 , ’ rep ’ ) ;
282 bu = add2mat ( bu , dumax , row , 1 , ’ rep ’ ) ;
283 end
284
285 for i = 0:N−1
286 row = 1+ i ∗ ( nx+nu ) ;
287 l = add2mat ( l , umin , row , 1 , ’ r e p ’ ) ;
288 u = add2mat ( u , umax , row , 1 , ’ r e p ’ ) ;
289 l = add2mat ( l , bmin , row+nu , 1 , ’ r e p ’ ) ;
290 u = add2mat ( u , bmax , row+nu , 1 , ’ r e p ’ ) ;
291 end
292
293 if pd == 1
294 x = L P s o l v e r ( l , u , A c , b l , bu ) ;
295 else
296 x = [];
297 end
298 [ x , i n f o ] = Q P s o l v e r (H, g , l , u , A c , b l , bu , x ) ;
299 d i s p ( ’ Performance i n f o r m a t i o n o f a c t i v e s e t method : ’ )
300 info
301
D.5 Demos 165

302 % ##############################################################
303 % p l o t s o f quad−t a n k p r o c e s s
304 % ##############################################################
305 output = x ;
306 % making t . . .
307 t = tspan ( 1 ) : dt : tspan ( 2 ) ;
308 % making h , F , d f . . .
309 u0 = o u t p u t ( 1 : 2 ) ;
310 x1 = o u t p u t ( 3 : 6 ) ;
311 n u l 0 = B∗u0−I x ∗ x1+Ax0 ;
312 nul0 = nul0 ’ ∗ nul0 ;
313 x k = x1 ;
314 h = x k (1:2) ;
315 h e i g h t s ( : , 1 ) =x0 ;
316 h e i g h t s ( : , 2 ) =x k ;
317 f l o w ( : , 1 ) = u0m i nus 1 ;
318 f l o w ( : , 2 ) = u0 ;
319 f o r k = 1 : N−1
320 k s = 3+(k −1) ∗ ( nx+nu ) ;
321 u k = o u t p u t ( k s + 4: k s +5) ;
322 x k p l u s = o u t p u t ( k s + 6: k s +9) ;
323
324 n u l k = A∗ x k+B∗ u k−I x ∗ x k p l u s ;
325 nul k = nul k ’∗ nul k ;
326
327 x k = x k plus ;
328 % h = x k (1:2)
329 h e i g h t s ( : , k +2)=x k ;
330 f l o w ( : , k+2) = u k ;
331 end
332 dF = [ d i f f ( f l o w ( 1 , : ) ) ; d i f f ( f l o w ( 2 , : ) ) ] ;
333 % plot . . .
334 f s i z e = 10;
335 f i gu re (1) , c l f
336 subplot (4 ,2 ,1)
337 p l o t ( t , h e i g h t s ( 1 , : ) , ’−o ’ )
338 h o l d on
339 p l o t ( t , r ( 1 ) ∗ o n e s ( 1 , N+1) , ’ r ’ )
340 p l o t ( t , hmax1∗ o n e s ( 1 , N+1) , ’ k ’ )
341 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ h 1 [ cm ] ’ , ’ F o n t S i z e ’ , f s i z e )% , l e g e n d
( ’h 1 ’ , ’ r 1 ’)
342 a x i s ( [ tspan (1 ) tspan (2 ) 0 5 0 ] )
343 subplot (4 ,2 ,2)
344 p l o t ( t , h e i g h t s ( 2 , : ) , ’−o ’ )
345 h o l d on
346 p l o t ( t , r ( 2 ) ∗ o n e s ( 1 , N+1) , ’ r ’ )
347 p l o t ( t , hmax2∗ o n e s ( 1 , N+1) , ’ k ’ )
348 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ h 2 [ cm ] ’ , ’ F o n t S i z e ’ , f s i z e )% , l e g e n d
( ’h 2 ’ , ’ r 2 ’)
349 a x i s ( [ tspan (1 ) tspan (2 ) 0 5 0 ] )
350 subplot (4 ,2 ,3)
351 p l o t ( t , h e i g h t s ( 3 , : ) , ’−o ’ )
352 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ h 3 [ cm ] ’ , ’ F o n t S i z e ’ , f s i z e )
353 h o l d on
354 p l o t ( t , hmax3∗ o n e s ( 1 , N+1) , ’ k ’ )
355 a x i s ( [ tspan (1 ) tspan (2 ) 0 5 0 ] )
356 subplot (4 ,2 ,4)
357 p l o t ( t , h e i g h t s ( 4 , : ) , ’−o ’ )
358 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ h 4 [ cm ] ’ , ’ F o n t S i z e ’ , f s i z e )
359 h o l d on
360 p l o t ( t , hmax4∗ o n e s ( 1 , N+1) , ’ k ’ )
361 a x i s ( [ tspan (1 ) tspan (2 ) 0 5 0 ] )
362 subplot (4 ,2 ,5)
363 p l o t ( t ( 1 : end )−dt , f l o w ( 1 , : ) , ’−o ’ )
364 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ F 1 [ cmˆ3/ s ] ’ , ’ F o n t S i z e ’ , f s i z e )
365 a x i s ( [ t ( 1 )−d t t s p a n ( 2 ) 0 Fmax1 ] )
366 subplot (4 ,2 ,6)
367 p l o t ( t ( 1 : end )−dt , f l o w ( 2 , : ) , ’−o ’ )
368 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ F 2 [ cmˆ3/ s ] ’ , ’ F o n t S i z e ’ , f s i z e )
369 %s t a i r s ( f l o w ( 2 , : ) )
370 a x i s ( [ t ( 1 )−d t t s p a n ( 2 ) 0 Fmax2 ] )
371 subplot (4 ,2 ,7)
372 p l o t ( t ( 1 : end −1)−dt , dF ( 1 , : ) , ’−o ’ )
373 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ \ D e l t a F 1 [ cmˆ3/ s ˆ 2 ] ’ , ’ F o n t S i z e ’ ,
fsize )
374 a x i s ( [ t ( 1 )−d t t s p a n ( 2 ) dFmin1 dFmax1 ] )
375 subplot (4 ,2 ,8)
376 p l o t ( t ( 1 : end −1)−dt , dF ( 2 , : ) , ’−o ’ )
377 x l a b e l ( ’ t [ s ] ’ , ’ F o n t S i z e ’ , f s i z e ) , y l a b e l ( ’ \ D e l t a F 2 [ cmˆ3/ s ˆ 2 ] ’ , ’ F o n t S i z e ’ ,
fsize )
378 a x i s ( [ t ( 1 )−d t t s p a n ( 2 ) dFmin2 dFmax2 ] )
379 hold o f f
380 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
381 % Animate demo %
382 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
383 save qu a dr up le t an k pr oc e ss
384 quad tank animate
166 Matlab-code

385 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
386 % Auxilery function ( s ) %
387 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
388 function v = check input (v , l , u , n)
389 v = v (:) ’;
390 m = length (v) ;
391 i f m ˜= n
392 e r r o r ( [ num 2s tr ( i nputnam e ( 1 ) ) , ’ must be a v e c t o r o f l e n g t h ’ , i n t 2 s t r ( n ) , ’ . ’
])
393 end
394 for i = 1: n
395 i f isch ar (v( i ) ) | ˜ i s r e a l (v( i ) ) | i s i n f (v( i ) ) | isnan (v( i ) ) | v( i ) < l | u
< v( i )
396 e r r o r ( [ num 2s tr ( i nputnam e ( 1 ) ) , ’ ( ’ , i n t 2 s t r ( i ) , ’ ) must be i n r a n g e ’ ,
num 2s tr ( l ) , ’ <= v a l u e <= ’ , num 2s tr ( u ) , ’ . ’ ] )
397 end
398 end

quad tank animate.m

1 % ##############################################################
2 % a n i m a t i o n o f quad−t a n k p r o c e s s
3 % ##############################################################
4 load ( ’ qua dr up le t a nk pr oc e ss ’ )
5 output = x ;
6 % making t . . .
7 t = tspan ( 1 ) : dt : tspan ( 2 ) ;
8 % making h , F , d f . . .
9 u0 = o u t p u t ( 1 : 2 ) ;
10 x1 = o u t p u t ( 3 : 6 ) ;
11 n u l 0 = B∗u0−I x ∗ x1+Ax0 ;
12 nul0 = nul0 ’ ∗ nul0 ;
13 x k = x1 ;
14 h = x k (1:2) ;
15 h e i g h t s ( : , 1 ) =x0 ;
16 h e i g h t s ( : , 2 ) =x k ;
17 f l o w ( : , 1 ) = u0m i nus 1 ;
18 f l o w ( : , 2 ) = u0 ;
19 f o r k = 1 : N−1
20 k s = 3+(k −1) ∗ ( nx+nu ) ;
21 u k = o u t p u t ( k s + 4: k s +5) ;
22 x k p l u s = o u t p u t ( k s + 6: k s +9) ;
23
24 n u l k = A∗ x k+B∗ u k−I x ∗ x k p l u s ;
25 nul k = nul k ’∗ nul k ;
26
27 x k = x k plus ;
28 % h = x k (1:2)
29 h e i g h t s ( : , k +2)=x k ;
30 f l o w ( : , k +2) = u k ;
31 end
32 dF = [ d i f f ( f l o w ( 1 , : ) ) ; d i f f ( f l o w ( 2 , : ) ) ] ;
33 f o r i = 1 : l e n g t h ( t )−1
34 f i gur e (2)
35 q u a d t a n k p l o t ( t ( i ) , h e i g h t s ( : , i ) , bmax , r , f l o w ( : , i ) , dF ( : , i ) , gam )
36 M( i ) = g e t f r a m e ;
37 end
38
39 % make m ov i e f o r p r e s e n t a t i o n . . .
40 r e p l y = i n p u t ( ’ Do you want t o make 4 t a n k d e m o m o v i e . a v i f i l e ? y /n [y]: ’ , ’s ’)
;
41 i f i s e m p t y ( r e p l y ) | r e p l y == ’ y ’
42 fp s = input ( ’ Sp e ci fy fp s in avi f i l e ? [ 1 5 ] : ’ ) ;
43 i f isempty ( f p s )
44 fps = 15;
45 end
46 d i s p ( ’ Making a v i f i l e , p l e a s e w a i t . . . ’ )
47 m o v i e 2 a v i (M, ’ 4 t a n k d e m o m o v i e . a v i ’ , ’ f p s ’ , f p s )
48 d i s p ( ’ F i n i s h e d making a v i f i l e . . . ’ )
49 end

SQP demo.m
D.5 Demos 167

1 function SQP demo

2
3 % SQP DEMO I n t e r a c t i v e d e m o n s t r a t i o n o f t h e SQP method . The ex am pl e p r o b l e m
4 % i s t h e f o l l o w i n g n o n l i n e a r program :
5 % min f ( x ) = x1 ˆ4 + x2 ˆ4
6 % s . t . x2 >= x1 ˆ2 − x1 + 1
7 % x2 >= x1 ˆ2 − 4 x1 + 6
8 % x2 <= −x1 ˆ2 + 3 x1 + 2
9 %
10 % Call
11 % SQP demo ( )
12 %
13 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 .
14 % Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
15 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n .
16 % M. Sc . , IMM, DTU, DK−2800 Lyngby .
17 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r .
18 % Per Grove Thomsen , P r o f e s s o r .
19 % D ate : 0 7 . J une 2 0 0 7 .
20
21 close all
22 it max = 1000;
23 method = 1 ;
24 t o l = 1 e −8; o p t s = [ t o l it max method ] ;
25
26 pi0 = [0 0 0 ] ’ ; % because we hav e three nonlinear constraints .
27 p l o t s c e n e ( @costfun ) ;
28
29 d i s p ( ’ L e f t c l i c k on p l o t t o s e l e c t s t a r t i n g p o i n t o r p r e s s any k ey e n t e r
startin g point in console . ’ )
30 [ u , v , but ] = g i n p u t ( 1 ) ;
31 i f but == 1
32 x0 = [ u v ] ;
33 else
34 w h i l e i s e m p t y ( x ) | l e n g t h ( x ) ˜= 2 | sum ( i s n a n ( x ) ) | sum ( i s i n f ( x ) ) | sum ( ˜
i s r ea l (x) ) | ischar (x)
35 x0 = i n p u t ( ’ E n t e r s t a r t i n g p o i n t [ x1 x2 ] : ’ ) ;
36 end
37 end
38 x0 = x0 ’ ;
39 p a t h p l o t ( x0 )
40 f s i z e = 12;
41 f s i z e s m a l l = 10;
42 f 0 = c o s t f u n ( x0 ) ;
43 g0 = modfun ( x0 ) ;
44 c = c o s t s e n s ( x0 ) ;
45 A = m ods ens ( x0 ) ;
46 W = e y e ( l e n g t h ( x0 ) ) ;
47 w non = ( 1 : 1 : l e n g t h ( g0 ) ) ;
48 plotNewtonStep = 1 ;
49 stop = 0;
50 t o l = opts (1 ) ;
51 max itr = opts ( 2) ;
52 it r = 0;
53 w hi l e ˜ stop
54 d i s p ( ’ P r e s s any k ey t o c o n t i n u e . . . ’ )
55 pause
56 X ( : , i t r +1) = x0 ;
57 i f plotNewtonStep
58 W = h e s s i a n ( x0 , p i 0 ) ;
59 else
60 h o l d on
61 p a t h p l o t (X)
62 end
63 i t r = i t r + 1;
64 i f ( i t r > max itr )
65 stop = 1 ;
66 end
67
68 [ d e l t a x , mu, dummy ] = d u a l a c t i v e s e t m e t h o d (W, c , A,−g0 , w non , [ ] , o p t s , 0 ) ;
69
70 if ( a b s ( c ’ ∗ d e l t a x ) + a b s (mu’ ∗ g0 ) ) < t o l
71 d i s p ( ’ s o l u t i o n has been found ’ )
72 stop = 1 ;
73 else
74
75 if i t r == 1
76 s i g m a = a b s (mu) ;
77 else
78 for i = 1: l e n g t h (mu)
79 s i g m a ( i ) = max ( a b s (mu( i ) ) , 0 . 5 ∗ ( s i g m a ( i )+a b s (mu( i ) ) ) ) ;
80 end
81 end
82
83 [ a l p h a , x , f , g ] = l i n e s e a r c h a l g o r i t h m ( @modfun , @ c o s t f u n , f 0 , g0 , c , x0 ,
d e l t a x , s i gm a , 1 e −4) ;
168 Matlab-code

84
85 p i i = p i 0 + a l p h a ∗ (mu−p i 0 ) ;
86
87 % h e r e t h e new ton s t e p i s p l o t t e d
88 i f plotNewtonStep
89 t s p a n = l i n s p a c e ( −1 ,3 ,100) ;
90 f o r i = 1: l e n g t h ( t s p a n )
91 x h a t = x0+t s p a n ( i ) ∗ d e l t a x ;
92 p i h a t = p i 0+t s p a n ( i ) ∗ (mu−p i 0 ) ;
93 na b la fx = c o s t s e n s ( x hat ) ;
94 n a b l a h x = m ods ens ( x h a t ) ;
95 y va l ( : , i ) = n a bl a f x − nabla hx ∗ pi ha t ;
96 end
97
98 subplot (1 ,3 ,1)
99 hold o f f
100 p l o t s c e n e ( @costfun ) ;
101 X temp = X ;
102 X temp ( : , i t r +1) = x ;%0+ d e l t a x ;
103 x f i n = x ;%x0+a l p h a ∗ d e l t a x ;
104 p a t h p l o t ( X temp )
105 t i t l e ( { [ ’ x { o l d } = ( ’ , num 2s tr ( x0 ( 1 ) ) , ’ , ’ , num 2s tr ( x0 ( 2 ) ) , ’ ) ˆT ’ ] ; [ ’ x
{new } = ( ’ , num 2s tr ( x f i n ( 1 ) ) , ’ , ’ , num 2s tr ( x f i n ( 2 ) ) , ’ ) ˆT ’ ] } , ’
FontSize ’ , f s i z e s m a l l ) ;
106 x l a b e l ( ’ x 1 ’ , ’ FontSize ’ , f s i z e ) , y l a b e l ( ’ x 2 ’ , ’ FontSize ’ , f s i z e )
107
108 subplot (1 ,3 ,2)
109 hold o f f
110 p lot ( t span , y va l ( 1 , : ) ) ;
111
112 nabla fx = c o s t s e n s ( x0 ) ;
113 nabla hx = m ods ens ( x0 ) ;
114 startPos = n ab la f x − nabla hx ∗pi0 ;
115 startPos y = startPos (1) ;
116 startPos x = 0;
117
118 endPos x = 1 ;
119 endPos y = 0 ;
120
121 h o l d on
122 p l o t ( [ s t a r t P o s x e n d P o s x ] , [ s t a r t P o s y e n d P o s y ] , ’ Li neWi dth ’ , 2 ) %
path
123 p l o t ( [ s t a r t P o s x a l p h a ] , [ s t a r t P o s y (1− a l p h a ) ∗ s t a r t P o s y ] , ’
Li neWi dth ’ , 2 , ’ c o l o r ’ , ’ r ’ ) % p a t h
124 p l o t ( [ t s p a n ( 1 ) t s p a n ( end ) ] , [ 0 0 ] , ’−− ’ ) % y=0
125
126 pi fin = p i 0+a l p h a ∗ (mu−p i 0 ) ;
127 nabla fx f in = costsens ( x fin ) ;
128 nabla hx f i n = m ods ens ( x f i n ) ;
129 endvalue = nabla fx f in − nabla hx fin ∗ p i f in ;
130
131 t i t l e ( { [ ’ F ( x 1 ) { o l d } = ’ , num 2s tr ( s t a r t P o s y ) ] ; [ ’ F ( x 1 ) {new } = ’ ,
num 2s tr ( e n d v a l u e ( 1 ) ) ] } , ’ F o n t S i z e ’ , f s i z e s m a l l ) ;
132 x l a b e l ( ’ \ alpha ’ , ’ FontSize ’ , f s i z e ) , y l a b e l ( ’ F 1 ’ , ’ FontSize ’ , f s i z e )
133
134 subplot (1 ,3 ,3)
135 hold o f f
136 p lot ( t span , y va l ( 2 , : ) ) ;
137 startPos y = startPos (2) ;
138 h o l d on
139 p l o t ( [ s t a r t P o s x e n d P o s x ] , [ s t a r t P o s y e n d P o s y ] , ’ Li neWi dth ’ , 2 ) %
path
140 p l o t ( [ s t a r t P o s x a l p h a ] , [ s t a r t P o s y (1− a l p h a ) ∗ s t a r t P o s y ] , ’
Li neWi dth ’ , 2 , ’ c o l o r ’ , ’ r ’ ) % p a t h
141 p l o t ( [ t s p a n ( 1 ) t s p a n ( end ) ] , [ 0 0 ] , ’−− ’ ) % y=0
142 t i t l e ( { [ ’ F ( x 2 ) { o l d } = ’ , num 2s tr ( s t a r t P o s y ) ] ; [ ’ F ( x 2 ) {new } = ’ ,
num 2s tr ( e n d v a l u e ( 2 ) ) ] } , ’ F o n t S i z e ’ , f s i z e s m a l l ) ;
143 x l a b e l ( ’ \ alpha ’ , ’ FontSize ’ , f s i z e ) , y l a b e l ( ’ F 2 ’ , ’ FontSize ’ , f s i z e )
144 end
145 n a b l a L 0 = c−A∗ p i i ;
146 c = costsens (x) ;
147 A = m ods ens ( x ) ;
148 n a b l a L = c−A∗ p i i ;
149 s = x − x0 ;
150 y = nabla L − nabla L0 ;
151 sy = s ’ ∗ y ;
152 sWs = s ’ ∗W∗ s ;
153 i f ( s y >= 0 . 2 ∗ sWs )
154 theta = 1;
155 else
156 t h e t a = ( 0 . 8 ∗ sWs ) / ( sWs−s y ) ;
157 end
158 Ws = W∗ s ;
159 sW = s ’ ∗W;
160 r = t h e t a ∗y+(1− t h e t a ) ∗Ws ;
161 W = W−(Ws∗sW) /sWs+( r ∗ r ’ ) / ( s ’ ∗ r ) ;
162 x0 = x ;
D.5 Demos 169

163 pi0 = p i i ;
164 f0 = f ;
165 g0 = g ;
166 end
167 end
168
169 function pathplot (x)
170 lwidth = 2 ; msize = 6 ; f s i z e = 12;
171 p l o t ( x ( 1 , 1 ) , x ( 2 , 1 ) , ’ ob ’ , ’ Li neWi dth ’ , l w i d t h , ’ M a r k e r s i z e ’ , m s i z e ) % s t a r t i n g
position
172 p l o t ( x ( 1 , : ) , x ( 2 , : ) , ’ Li neWi dth ’ , l w i d t h ) % p a t h
173 p l o t ( x ( 1 , : ) , x ( 2 , : ) , ’ ob ’ , ’ Li neWi dth ’ , l w i d t h , ’ M a r k e r s i z e ’ , m s i z e ) % p a t h
174 p l o t ( x ( 1 , end ) , x ( 2 , end ) , ’ og ’ , ’ Li neWi dth ’ , l w i d t h , ’ M a r k e r s i z e ’ , m s i z e ) % c u r r e n t
position
175 t i t l e ( [ ’ x = ( ’ , num 2s tr ( x ( 1 , end ) ’ ) , ’ , ’ , num 2s tr ( x ( 2 , end ) ) , ’ ) ’ ] , ’ F o n t S i z e ’ , f s i z e
)
176
177
178
179 function fx = costfun (x)
180 % The f u n c t i o n t o be m i n i m i z e d
181 f x = x ( 1 ) ∗ x ( 1 ) ∗x ( 1 ) ∗ x ( 1 )+x ( 2 ) ∗ x ( 2 ) ∗x ( 2 ) ∗ x ( 2 ) ; % : c o s t = ( X1 . ˆ 4 + X2 . ˆ 4 ) ;
182
183 f u n c t i o n dx = c o s t s e n s ( x )
184 % The g r a d i e n t o f t h e c o s t f u n c t i o n
185 dx =[4∗ x ( 1 ) ∗x ( 1 ) ∗x ( 1 ) ; 4∗ x ( 2 ) ∗ x ( 2 ) ∗x ( 2 ) ] ; % : gradient of ( X1 . ˆ 4 + X2 . ˆ 4 ) ;
186
187
188 f u n c t i o n f x = modfun ( x )
189 % The c o n s t r a i n t s
190 c 1 = −x ( 1 ) ˆ2+x ( 1 )+x ( 2 ) −1; % x2 >= x1 . ˆ 2 − x1 + 1
191 c 2 = −x ( 1 ) ˆ2+4∗x ( 1 )+x ( 2 ) −6; % x2 >= x1 ˆ2 − 4 x1 + 6
192 c 3 = −x ( 1 ) . ˆ2+ 3∗ x ( 1 )−x ( 2 ) + 2; % x2 <= −x1 ˆ2 + 3 x1 + 2
193 f x = [ c1 c2 c3 ] ’ ;
194
195 f u n c t i o n d f x = m ods ens ( x )
196 % gr ad i e nt of the c o n s t r a i n t s
197
198 dc1 = [ −2∗x ( 1 ) +1 1 ] ’ ; % x2 >= x1 . ˆ 2 − x1 + 1
199 dc2 = [ −2∗x ( 1 ) +4 1 ] ’ ; % x2 >= x1 ˆ2 − 4 x1 + 6
200 dc3 = [ −2∗x ( 1 ) +3 − 1 ] ’ ; % x2 <= −x1 ˆ2 + 3 x1 + 2
201 dfx = [ dc1 dc2 dc3 ] ;
202
203 f u n c t i o n H = h e s s i a n ( x , mu)
204
205 H e s s C t r 1 = [ −2 0; 0 0]; % x2 >= x1 . ˆ 2 − x1 + 1
206 H e s s C t r 2 = [ −2 0; 0 0]; % x2 >= x1 ˆ2 − 4 x1 + 6
207 H e s s C t r 3 = [ −2 0; 0 0]; % x2 <= −x1 ˆ2 + 3 x1 + 2
208
209 HessCost = [ 12∗ x ( 1 ) ∗x ( 1 ) 0;
210 0 12∗ x ( 2 ) ∗x ( 2 ) ] ; % : c o s t = ( X1 . ˆ 4 + X2 . ˆ 4 ) ;
211
212 H = H e s s C o s t −(mu( 1 ) ∗ H e s s C t r 1 ) −(mu( 2 ) ∗ H e s s C t r 2 ) −(mu( 3 ) ∗ H e s s C t r 3 ) ;
213
214 f un c ti on p l o t s c e n e ( costfun , v ar ar gi n )
215 f c o l o r = [ . 4 . 4 . 4 ] ; f a l p h a = . 4 ; % c o l o r and a l p h a v a l u e s o f f a c e s m ark i ng
unfeasable region
216 plot left = −5;
217 plot right = 5;
218 p l o t b u t t o m = −5;
219 plot top = 5;
220 p l o td et a i l s = 30;
221 l i n s p a c e d e t ai l s = 100;
222 contours = 10;
223 ctr1 = 1;
224 ctr2 = 1;
225 ctr3 = 0;
226 ctr4 = 0;
227 ctr5 = 1;
228 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
229 % c o n s t r a i n t s d e f i n e d f o r z=0
230 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
231 x ctr = linspace ( p lot lef t , plot right , l ins pace details ) ;
232 y c t r 1 = x c t r .ˆ2 − x c t r + 1; ctr1 geq = 1; % x2 >= x1 . ˆ 2 − x1 + 1
233 y c t r 2 = x c t r . ˆ 2 − 4∗ x c t r + 6 ; ctr2 geq = 1; % x2 >= x1 ˆ2 − 4 x1 + 6
234 y ctr3 = sin ( x ctr ) + 3; ctr3 geq = 0; % x2 <= s i n ( x1 ) + 3
235 y c t r 4 = cos ( x c t r ) + 2 ; ctr4 geq = 1; % x2 >= c o s ( x1 ) + 2
236
237 y c t r 5 = − x c t r . ˆ2+ 3∗ x c t r + 2; ctr5 geq = 0; % x2 <= −x1 ˆ2 + 3 x1 + 2
238 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
239 % pl ot the c os t f u nc t i o n
240 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
241 delta = dist ( plo t le ft , p lot r ight )/ p lot de t ails ;
242 [ X1 , X2 ] = m e s h g r i d ( p l o t l e f t : d e l t a : p l o t r i g h t ) ; %c r e a t e a m a t r i x o f (X, Y) from
vector
243 f o r i = 1 : l e n g t h ( X1 )
244 f o r j = 1 : l e n g t h ( X2)
170 Matlab-code

245 c o s t ( i , j ) = f e v a l ( costfun , [ X1 ( i , j ) X2 ( i , j ) ] ) ;% , varargin {:}) ;

246 end
247 end
248 %f i g u r e ( 1 )
249 % mesh ( X1 , X2 , c o s t )
250 %f i g u r e
251 c o n t o u r ( X1 , X2 , c o s t , c o n t o u r s )
252 h o l d on
253
254 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
255 % p l ot the c o n s t r a i n t s
256 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
257 buttom final =[ ];
258 top final =[];
259 [ t o p fin al , buttom final ] = p l o t c t r ( x ctr , y ctr1 , ctr1 geq , ctr1 , p l o t l e f t ,
p l o t r i g h t , plot buttom , p lo t top , t o p f i n a l , buttom final , fc olor , falpha
);
260 [ t o p fin al , buttom final ] = p l o t c t r ( x ctr , y ctr2 , ctr2 geq , ctr2 , p l o t l e f t ,
p l o t r i g h t , plot buttom , p lo t top , t o p f i n a l , buttom final , fc olor , falpha
);
261 [ t o p fin al , buttom final ] = p l o t c t r ( x ctr , y ctr3 , ctr3 geq , ctr3 , p l o t l e f t ,
p l o t r i g h t , plot buttom , p lo t top , t o p f i n a l , buttom final , fc olor , falpha
);
262 [ t o p fin al , buttom final ] = p l o t c t r ( x ctr , y ctr4 , ctr4 geq , ctr4 , p l o t l e f t ,
p l o t r i g h t , plot buttom , p lo t top , t o p f i n a l , buttom final , fc olor , falpha
);
263
264 [ t o p f i n a l , bu tt o m f i nal ] = p l o t c t r ( x c tr , y ctr5 , ctr5 geq , ctr5 , p l o t l e f t ,
p l o t r i g h t , plot buttom , p lo t top , t o p f i n a l , buttom fi nal , f c o l o r , f al pha
);
265
266 if ˜ i s e m p t y ( t o p f i n a l ) && i s e m p t y ( b u t t o m f i n a l )
267 f i l l ( [ p l o t l e f t x c t r p l o t r i g h t ] , [ plot buttom top final plot buttom ] ,
f c o l o r , ’ FaceA l pha ’ , f a l p h a )
268 end
269 i f ˜ i s e m p t y ( b u t t o m f i n a l ) && i s e m p t y ( t o p f i n a l )
270 f i l l ( [ p l o t l e f t x ctr p l ot r ig h t ] , [ plot top buttom final plot top ] , fcolor , ’
FaceA l pha ’ , f a l p h a )
271 end
272 i f ˜ i s e m p t y ( b u t t o m f i n a l ) && ˜ i s e m p t y ( t o p f i n a l )
273 temp = t o p f i n a l ;
274 t o p f i n a l = max ( t o p f i n a l , b u t t o m f i n a l ) ;
275 f i l l ( [ x c t r ] , [ t o p f i n a l ] , f c o l o r , ’ FaceA l pha ’ , f a l p h a )
276 f i l l ( [ p l o t l e f t x c t r p l o t r i g h t ] , [ p l o t b u t t o m temp p l o t b u t t o m ] , f c o l o r , ’
FaceA l pha ’ , f a l p h a )
277 end
278
279 f u n c t i o n [ t o p f i n , b u t t o m f i n ] = p l o t c t r ( x s p a n , y s p a n , geq , plott , left ,
r i g h t , buttom , top , t o p f i n , b u t t o m f i n , c o l o r , a l p h a )
280 i f plott
281 p l o t ( x span , y span , ’ bl a ’ )
282 i f geq
283 i f isempty ( t o p f i n ) % f i r s t c a l l
284 t o p f i n = y span ;
285 else
286 t o p f i n = max( t o p f i n , y s p a n ) ;
287 end
288 else
289 i f isempty ( b u t t o m f i n )
290 buttom fin = y span ;
291 else
292 b u t t o m f i n = min ( b u t t o m f i n , y s p a n ) ;
293 end
294 end
295 end
D.6 Auxiliary Functions 171

D.6 Auxiliary Functions

add2mat.m

1 % ADD2MAT Add/ s u b t r a c t / r e p l a c e e l e m e n t s o f two m a t r i c e s o f d i f f e r e n t s i z e s .

2 %
3 % Sy ntax :
4 % NEWMATRIX = ADD2MAT(MATRIX1, MATRIX2, INITATROW, INITATCOLUMN, ADDSTYLE)
5 %
6 % Description :
7 % A d d i t i o n o r s u b t r a c t i o n b e t w e e n o r r e p l a c e m e n t o f e l e m e n t s i n MATRIX1
8 % by MATRIX2 . MATRIX1 and MATRIX2 can be o f d i f f e r e n t s i z e s , a s l o n g a s
9 % MATRIX2 f i t s i n s i d e MATRIX1 w i t h r e s p e c t t o t h e i n i t i a l p o i n t . MATRIX2
10 % o p e r a t e s on MATRIX1 s t a r t i n g from t h e i n i t i a l p o i n t
11 % (INITATROW, INITATCOLUMN) i n MATRIX1 .
12 %
13 % ADDSTYLE = ’ add ’ : B u i l d i n g NEWMATRIX by a d d i n g MATRIX2 t o e l e m e n t s i n
14 % MATRIX1 .
15 % ADDSTYLE = ’ sub ’ : B u i l d i n g NEWMATRIX by s u b t r a c t i n g MATRIX2 from e l e m e n t s
16 % i n MATRIX1 .
17 % ADDSTYLE = ’ mul ’ : B u i l d i n g NEWMATRIX by e l e m e n t w i s e m u l t i p l i c a t i o n o f
MATRIX2
18 % and e l e m e n t s i n MATRIX1 .
19 % ADDSTYLE = ’ d i v ’ : B u i l d i n g NEWMATRIX by e l e m e n t w i s e d i v i s i o n o f MATRIX2
20 % and e l e m e n t s i n MATRIX1 .
21 % ADDSTYLE = ’ r e p ’ : B u i l d i n g NEWMATRIX by r e p l a c i n g e l e m e n t s i n MATRIX1 w i t h
22 % MATRIX2 .
23 %
24 % Example :
25 % >> A = [ 1 2 3 4 ; 5 6 7 8 ; 9 10 11 1 2 ; 13 14 15 1 6 ]
26 %
27 % A =
28 %
29 % 1 2 3 4
30 % 5 6 7 8
31 % 9 10 11 12
32 % 13 14 15 16
33 %
34 % >> b = [ 1 1 1 ]
35 %
36 % b =
37 %
38 % 1 1 1
39 %
40 % >> B = d i a g ( b )
41 %
42 % B =
43 %
44 % 1 0 0
45 % 0 1 0
46 % 0 0 1
47 %
48 % >> C = add2mat (A, B , 2 , 2 , ’ r e p ’ )
49 %
50 % C =
51 %
52 % 1 2 3 4
53 % 5 1 0 0
54 % 9 0 1 0
55 % 13 0 0 1
56 %
57 % S e e a l s o DIAG2MAT, DIAG , CAT .
58
59 % −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
60 % ADD2MAT V e r s i o n 3 . 0
61 % Made by C a r s t e n V( o e ) l c k e r , <s 9 6 1 5 7 2 @ s t u d e n t . dtu . dk>
62 % i n MATLAB V e r s i o n 6 . 5 R e l e a s e 13
63 % −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
64
65 function m a t r i x 3 = add2mat ( m a t r i x 1 , m a t r i x 2 , i n i t m , i n i t n , a d d s t y l e )
66
67 if nargin < 5
68 e r r o r ( ’ Not enough i n p u t arguments . ’ )
69 end
70 i f ˜ i s n u m e r i c ( matrix1 )
71 e r r o r ( ’MATRIX1 must be a m a t r i x . ’ )
72 end
73 i f ˜ i s n u m e r i c ( matrix2 )
74 e r r o r ( ’MATRIX2 must be a m a t r i x . ’ )
75 end
76 i f ˜ i s n u m e r i c ( i n i t m ) | | l e n g t h ( i n i t m ) ˜= 1
77 e r r o r ( ’INITATROW must be an i n t e g e r . ’ )
172 Matlab-code

78 end
79 i f ˜ i s n u m e r i c ( i n i t n ) | | l e n g t h ( i n i t n ) ˜= 1
80 e r r o r ( ’INITATCOLUMN must be an i n t e g e r . ’ )
81 end
82 i f ˜ i s s t r ( addstyle )
83 e r r o r ( ’ADDSTYLE n o t d e f i n e d . ’ )
84 end
85 [ m1, n1 ] = s i z e ( m a t r i x 1 ) ;
86 [ m2, n2 ] = s i z e ( m a t r i x 2 ) ;
87 i f m2 > m1 | | n2 > n1
88 e r r o r ( [ ’MATRIX2 w i t h d i m e n s i o n ( s ) ’ , i n t 2 s t r (m2) , ’ x ’ , i n t 2 s t r ( n2 ) , ’ d o e s n o t
f i t i n s i d e MATRIX1 w i t h d i m e n s i o n ( s ) ’ . . .
89 , i n t 2 s t r (m1) , ’ x ’ , i n t 2 s t r ( n1 ) , ’ . ’ ] )
90 end
91 i f i n i t m > m1 | | i n i t n > n1
92 e r r o r ( [ ’ I n i t i a l poi nt ( ’ , i n t 2 s t r ( initm ) , ’ , ’ , i n t 2 s t r ( i n i t n ) , ’ ) ex ceeds
d i m e n s i o n ( s ) ’ , i n t 2 s t r (m1) , ’ x ’ , i n t 2 s t r ( n1 ) , . . .
93 ’ o f MATRIX1 . ’ ] )
94 end
95 i f i n i t m+m2−1 > m1 | | i n i t n +n2−1 > n1
96 e r r o r ( [ ’ With i n i t i a l p o i n t ( ’ , i n t 2 s t r ( i n i t m ) , ’ , ’ , i n t 2 s t r ( i n i t n ) , ’ ) ,
d i m e n s i o n ( s ) ’ , i n t 2 s t r (m2) , ’ x ’ , i n t 2 s t r ( n2 ) , . . .
97 ’ o f MATRIX2 e x c e e d s d i m e n s i o n ( s ) ’ , i n t 2 s t r (m1) , ’ x ’ , i n t 2 s t r ( n1 ) , ’
o f MATRIX1 . ’ ] )
98 end
99 switch a dds ty l e
100 c a s e ’ add ’
101 m a t r i x 1 ( i n i t m : i n i t m+m2−1 , i n i t n : i n i t n+n2 −1) = m a t r i x 1 ( i n i t m : i n i t m+m2−1 ,
i n i t n : i n i t n +n2 −1)+m a t r i x 2 ;
102 matrix3 = matrix1 ;
103 c a s e ’ s ub ’
104 m a t r i x 1 ( i n i t m : i n i t m+m2−1 , i n i t n : i n i t n+n2 −1) = m a t r i x 1 ( i n i t m : i n i t m+m2−1 ,
i n i t n : i n i t n +n2 −1)−m a t r i x 2 ;
105 matrix3 = matrix1 ;
106 c a s e ’ mul ’
107 m a t r i x 1 ( i n i t m : i n i t m+m2−1 , i n i t n : i n i t n+n2 −1) = m a t r i x 1 ( i n i t m : i n i t m+m2−1 ,
i n i t n : i n i t n +n2 −1) . ∗ m a t r i x 2 ;
108 matrix3 = matrix1 ;
109 case ’ div ’
110 m a t r i x 1 ( i n i t m : i n i t m+m2−1 , i n i t n : i n i t n+n2 −1) = m a t r i x 1 ( i n i t m : i n i t m+m2−1 ,
i n i t n : i n i t n +n2 −1) . / m a t r i x 2 ;
111 matrix3 = matrix1 ;
112 cas e ’ rep ’
113 m a t r i x 1 ( i n i t m : i n i t m+m2−1 , i n i t n : i n i t n+n2 −1) = m a t r i x 2 ;
114 matrix3 = matrix1 ;
115 end

line search algorithm.m

1 f u n c t i o n [ a l p h a , x , f , g ] = l i n e s e a r c h a l g o r i t h m ( modfun , c o s t f u n , f 0 , g0 , c , x0 ,
d e l t a x , s i gm a , c1 , v a r a r g i n )
2
3 % LINE SEARCH ALGORITHM i m p l e m e n t e d a c c o r d i n g t o P o w e l l s l 1 −P e n a l t y
4 % function
5 %
6 % By : C a r s t e n V\ ¨ o l c k e r , s 9 6 1 5 7 2 & Es ben L u n d s a g e r Hansen , s 0 2 2 0 2 2 .
7 % Subject : N u m e r i c a l Methods f o r S e q u e n t i a l Q u a d r a t i c O p t i m i z a t i o n ,
8 % M a s t e r T h e s i s , IMM, DTU, DK−2800 Lyngby .
9 % S u p e r v i s o r : John B a g t e r p J ø r g e n s e n , A s s i s t a n t P r o f e s s o r & Per Grove
Thomsen , P r o f e s s o r .
10 % D ate : 08. february 2007.
11
12 n0 = s i gm a ’ ∗ a b s ( g0 ) ;
13 T0 = f 0+n0 ;
14 dT0 = c ’ ∗ d e l t a x −n0 ;
15 a l p h a 1 = 1 ;%a l p h a v a l ; % 1 ;
16
17 x = x0+a l p h a 1 ∗ d e l t a x ;
18 f = f e v a l ( costfun , x , v ara rg i n { :} ) ;
19 g = f e v a l ( modfun , x , v a r a r g i n { : } ) ;
20 T1 = f+s i gm a ’ ∗ a b s ( g ) ;
21
22 if T1 <= T0+c 1 ∗dT0
23 alpha = alpha1 ;
24 return
25 end
26
27 a l p h a m i n = dT0 / ( 2 ∗ ( T0+dT0−T1 ) ) ;
28 a l p h a 2 = max ( 0 . 1 ∗ a l p h a 1 , a l p h a m i n ) ; % s k a l 0.1 v æ re c1 i stedet f o r ??
29
D.6 Auxiliary Functions 173

30 x = x0+a l p h a 2 ∗ d e l t a x ;
31 f = f e v a l ( costfun , x , va ra rg in {: } ) ;
32 g = f e v a l ( modfun , x , v a r a r g i n { : } ) ;
33 T2 = f+s i gm a ’ ∗ a b s ( g ) ;
34
35 if T2 <= T0+c 1 ∗ a l p h a 2 ∗dT0
36 alpha = alpha2 ;
37 return
38 end
39
40 stop = 0;
41 max itr = 100;
42 it r = 0;
43 w hi l e ˜ stop
44 i t r = i t r + 1;
45 i f i t r > max itr
46 disp ( ’ l i ne s ear ch ( i t r > mat itr ) ’ ) ;
47 stop = 1 ;
48 end
49
50 ab = 1 / ( a l p h a 1 −a l p h a 2 ) ∗ [ 1 / ( a l p h a 1 ∗ a l p h a 1 ) −1/( a l p h a 2 ∗ a l p h a 2 ) ;− a l p h a 2 / (
a l p h a 1 ∗ a l p h a 1 ) a l p h a 1 / ( a l p h a 2 ∗ a l p h a 2 ) ] ∗ [ T1−dT0∗ a l p h a 1−T0 ; T2−dT0∗
a l p h a 2 −T0 ] ;
51 a = ab ( 1 ) ;
52 b = ab ( 2 ) ;
53 i f ( a b s ( a )<e p s )
54 a l p h a m i n = −dT0/b ;
55 else
56 a l p h a m i n = (−b+( s q r t ( b∗b−3∗ a ∗dT0 ) ) ) /3∗ a ;
57 end
58
59 if ( a l p h a m i n <= 0 . 1 ∗ a l p h a 2 )
60 alpha = 0.1∗ alpha2 ;
61 else
62 if ( a l p h a m i n >= 0 . 5 ∗ a l p h a 2 )
63 alpha = 0.5∗ alpha2 ;
64 else
65 alpha = alpha min ;
66 end
67 end
68
69 x = x0+a l p h a ∗ d e l t a x ;
70 f = f e v a l ( costfun , x , v ar ar gi n { : }) ;
71 g = f e v a l ( modfun , x , v a r a r g i n { : } ) ;
72 T a l p h a = f+s i gm a ’ ∗ a b s ( g ) ;
73
74 i f T a l p h a <= T0+c 1 ∗ a l p h a ∗dT0
75 return
76 end
77 alpha1 = alpha2 ;
78 alpha2 = alpha ;
79 T1 = T2 ;
80 T2 = T a l p h a ;
81 end
174 Matlab-code