Contents

1 Quadratic Programming 5
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Equality-constrained quadratic programming . . . . . . . . . . 7
1.3 Active set method for quadratic programming problems . . . 10
1.3.1 Problem formulation . . . . . . . . . . . . . . . . . . . . 10
1.3.2 The active set method . . . . . . . . . . . . . . . . . . . 13

3
 Contents

4
Chapter 1
Quadratic Programming

1.1 Introduction

A quadratic program (QP) is an optimization problem where the objective func-

tion is quadratic and the constraints are linear. Problems of this type are im-
portant in their own right, and they also arise as subproblems in methods for
general constrained optimization, such as sequential quadratic programming
(Nocedal and Wright, 1999).
A general quadratic programming problem can be stated as:

1
min f (x) = xT Qx + cT x (1.1)
x 2

subject to
Ax = b (1.2)

Dx ≤ g (1.3)

where:
x is a n × 1 vector: x = [x1 x2 . . . xn ]T
Q is a symmetric n × n matrix
c is an n × 1 vector
A is a m × n matrix
b is an m × 1 vector
D is a p × n matrix
g is an p × 1 vector

5
 Chapter 1. Quadratic Programming

The problem has n decision variables, m equality constraints and p in-

equality constraints.
If the matrix Q is positive semidefinite, the problem is a convex quadratic
program.

Example 1.1 (Nocedal and Wright, 1999) Consider the following QP problem:

min f (x) = 3x21 + 2x1 x2 + x1 x3 + 2.5x22 + 2x2 x3 + 2x23 − 8x1 − 3x2 − 3x3 (1.4)
x

subject to

x1 + x3 = 3 (1.5)
x2 + x3 = 0 (1.6)

We shall write the problem in the general form (1.1 -1.2).

In general, a quadratic form 12 xT Qx, where Q is a symmetric 3 × 3 matrix and
x is a 3 × 1 vector, can be written as:
  
q11 q12 q13 x1
1 T 1   
x Qx = [x1 x2 x3 ]  q12 q22 q23   x2 
2 2
q13 q23 q33 x3
1
= (q11 x21 + q22 x22 + q33 x23 + 2q12 x1 x2 + 2q13 x1 x3 + 2q23 x2 x3 )
2
1 1 1
= q11 x21 + q22 x22 + q33 x23 + q12 x1 x2 + q13 x1 x3 + q23 x2 x3 (1.7)
2 2 2

By simple identification of the coefficients, the objective function is written as a

quadratic form:
 
6 2 1
1  
f (x) = xT  2 5 2  x + [−8 − 3 − 3]x (1.8)
2
1 2 4

The equality constraints, in the form Ax = b, result as:

 
" # x1 " #
1 0 1   3
 x2  = (1.9)
0 1 1 0
x3

6
1.2. Equality-constrained quadratic programming

Thus, the matrices Q, A and the vectors c and b are:

   
6 2 1 " # −8 " #
  1 0 1   3
Q =  2 5 2 , A = , c =  −3  , b = (1.10)
0 1 1 0
1 2 4 −3

1.2 Equality-constrained quadratic programming

We consider first the case when the problem has only equality constraints.
The QP-problem is stated as:

1
min xT Qx + cT x (1.11)
x 2

subject to
Ax = b (1.12)

where A is an m × n matrix, b an m × 1 vector and x an n × 1 vector, with

m ≤ n.
If the number of constraints (m) is equal to the number of variables (n) the
problem is uniquely determined and the only solution is the one of the linear
system (1.12). In case the number of constraints is greater than the number of
variables, m ≥ n, the problem may have no solution.
The method of Lagrange multipliers will be applied. Let λ be the vector
of m Lagrange multipliers, λ = [λ1 λ2 . . . λm ]T . The Lagrangian of this
problem is then:

1
L(x, λ) = xT Qx + cT x + λT (Ax − b) (1.13)
2

The necessary condition for a vector x to be a solution of the QP problem

is that there is a vector λ such that the derivatives of the Lagrangian with
respect to x and λ are zero:

∂L
= Qx + c + AT λ = 0 (1.14)
∂x

and
∂L
= Ax − b = 0 (1.15)
∂λ

7
 Chapter 1. Quadratic Programming

The equations (1.14), (1.15) can be written as a system of linear equations:

Qx + AT λ = −c
Ax − 0 · λ = b (1.16)

or: " # " # " #

Q AT x −c
· = (1.17)
A 0 λ b
We shall write again the equation (1.17) emphasizing the size of the matrices
involved: " # " # " #
Qn×n ATn×m xn×1 −cn×1
· = (1.18)
Am×n 0m×m λm×1 bm×1
The problem is reduced to the solution of a linear system of n + m equations
with n + m unknowns. The sufficient conditions for minimum can be verified
using, for example, the procedure described in section ??.

Example 1.2 Solve the following QP problem:

min f (x1 , x2 ) = x21 + x22 (1.19)

(x1 ,x2 )

subject to
x1 + x2 = 5 (1.20)

The objective function and the constraint can be written in the general matrix
form as: " #
1 T 2 0
f (x1 , x2 ) = f (x) = x x + [0 0]T x (1.21)
2 0 2
" #
x1
[1 1] =5 (1.22)
x2

The necessary condition for optimum is obtained from the linear system (1.17)
given by:
     
2 0 1 x1 0
     
 0 2 1  ·  x2  =  0  (1.23)
1 1 0 λ 5

8
1.2. Equality-constrained quadratic programming

and has the solution:

x1 = 2.5, x2 = 2.5, λ = −5 (1.24)

The sufficient condition for minimum is given by the sign of the determinant (??)
which, in this example, is:
¯ ¯
¯ 2 0 1 ¯
¯ ¯
1¯ ¯
(−1) ¯ 0 2 1 ¯=4>0 (1.25)
¯ ¯
¯ 1 1 0 ¯

The point is a unique minimizer of the objective function subject to the given con-
straint, because the determinant is positive.

Figure 1.1 shows the contour plot of the objective function and the constraint.
The solution is the tangent point of one level curve and the constraint line.

4
x2

2
x +x =5
1 2

0
0 1 2 3 4 5 6
x1

Figure 1.1: The contour plot of f (x1 , x2 ) and the constraint

9
 Chapter 1. Quadratic Programming

1.3 Active set method for quadratic programming prob-

lems

1.3.1 Problem formulation

The general QP problem is stated now in a general form which emphasizes

the m equality constraints and p inequality constraints:

1
min xT Qx + cT x (1.26)
x 2

subject to

aiT x = bi , i = 1, m (1.27)
ajT x ≤ bj , j = m + 1, m + p (1.28)

where:
x is the n × 1 vector of unknowns
Q is a symmetric n × n positive semidefinite matrix
c is an n × 1 vector
bi are scalars, i = 1, m + p
ai are n × 1 vectors, i = 1, m + p
The vectors aTi are the rows of the matrices A and D from (1.2) and (1.3).

We define the active set, A(x∗ ), at an optimal point x∗ as the indices of the
constraints at which equality holds, that is (Nocedal and Wright, 1999):

A(x∗ ) = {i ∈ {1, 2, . . . , m + p} : aTi x∗ = bi } (1.29)

The Lagrangian of this problem is:

m
X m+p
X
1 T T T
L(x, λ) = x Qx + c x + λi (ai x − bi ) + λj (aTj x − bj ) (1.30)
2
i=1 j=m+1

where λ is a vector of m + p Lagrange multipliers.

10
1.3. Active set method for quadratic programming problems

We can write now the KKT conditions for this problem:

∂L
= 0 (1.31)
∂x
aTi x = bi , i = 1, m (1.32)
aTj x ≤ bj , j = m + 1, m + p (1.33)
λj (aTj x − bj ) = 0, j = m + 1, m + p (1.34)
λj ≥ 0, j = m + 1, m + p (1.35)
λi , i = 1, m unrestricted in sign (1.36)

Example 1.3 Consider the following QP problem:

min f (x1 , x2 ) = x21 + 2x22 − x1 − 2x2 (1.37)

(x1 ,x2 )

subject to

x1 + x2 = 3
x1 ≥ 1
x2 ≤ 3 (1.38)
−x1 + 3x2 ≥ −1
x1 + x2 ≤ 5

We shall illustrate the feasible region and write the problem in the general form
(1.26), (1.27), (1.28).
The inequality constraints form the convex gray region illustrated in Figure 1.2.
The minimizer of the objective function subject to the constraints must be located
in the interior of the feasible region, on the straight line x1 + x2 = 3 (the equality
constraint).
If we denote x = [x1 x2 ]T , the quadratic objective function is written in the
matrix form as:
" #
1 T 2 0
f (x) = x x + [−1 − 2]x
2 0 4
1 T
= x Qx + cT x (1.39)
2

11
 Chapter 1. Quadratic Programming

Figure 1.2: The feasible region and the contour plot of the objective function

The number of equality constraints for this problem is m = 1. It can be written

as:
x1 + x2 = [1 1] x = 3 (1.40)

thus: " #
1
a1 = and b1 = 3 (1.41)
1

Some of the inequality constraints will be multiplied by (−1) so all the left hand sides
are less or equal than the right hand side:

−x1 ≤ −1
x2 ≤ 3 (1.42)
x1 − 3x2 ≤ 1
x1 + x2 ≤ 5

The vectors aj , j = 2, 3, 4, 5 are:

" # " # " # " #
−1 0 1 1
a2 = , a3 = , a4 = , a5 = (1.43)
0 1 −3 1

12
1.3. Active set method for quadratic programming problems

and the scalars from the right hand side:

b2 = −1, b3 = 3, b4 = 1, b5 = 5 (1.44)

We shall not give a solution for this problem, but it is clear from the Figure 1.2
that the optimum is located in the interior of the feasible region, at some point where
the equality-constraint line is tangent to a level curve of the objective function. Since
at this point, all the inequality constraints are inactive (none of them is satisfied as
an equality), the optimal active set is A(x∗ ) = {1}, which is the index of the only
constraint fulfilled as equality.

1.3.2 The active set method

The active set method is one of the most popular approaches for solving small
and medium scale QP problems. The idea behind the method may be sum-
marized as follows (Nocedal and Wright, 1999; Bhatti, 2000):
• Start with a guess of the optimal active set A and calculate a feasible
initial iterate x0 .

• Use the gradient and Lagrange multiplier information to remove one

index from the current active set and to add a new one. The method
ensures the feasibility of the next iterate xk+1 calculated from:

xk+1 = xk + αk dk (1.45)

where αk is the step length and dk the direction of moving, obtained

by solving a QP subproblem. This subproblem will have a subset of
constraints imposed as equalities and referred as the working set, Wk ,
consisting of all m equality constraints and some of the active inequali-
ties. Some iterates may be located on the boundary or in the interior of
the feasible region.

• New iterates are calculated and the working set is modified until the
optimality conditions are satisfied, or all Lagrange multipliers are pos-
itive as required by the KKT conditions.
Let xk be the current iterate. At this point, some of the inequality con-
straints may be active (or satisfied as equalities). Together with the equality

13
 Chapter 1. Quadratic Programming

constraints they form the working set Wk :

Wk = {1, . . . , m} ∪ {i : aTi xk = bi , i = m + 1, . . . , m + p} (1.46)

For the current point, we check whether xk minimizes the quadratic ob-
jective function in the subspace defined by the working set, i.e. the Lagrange
multipliers corresponding to the inequality constraints are positive. This is a
direct consequence of the KKT conditions.
If the optimality conditions are not satisfied, we compute a direction, dk ,
to move to the next point xk+1 = xk + dk such that the new iterate is feasible
in Wk and the objective function is minimized at xk + dk . Since xk is known
at the current stage, it will be regarded as a constant vector and the unknown
vector is only dk . The problem is stated as:

1
min f (dk ) = (xk + dk )T Q(xk + dk ) + cT (xk + dk ) (1.47)
dk 2

subject to:
aTi (xk + dk ) = bi , i ∈ Wk (1.48)

Expanding the new objective function we have:

1 1
f (dk ) = xTk Qxk + dTk Qdk + xTk Qdk + cT xk + cT dk (1.49)
2 2

The term 12 xTk Qxk +cT xk is constant for a given xk , thus it can be removed
from the objective function without changing the solution.
We denote:
gk = Qxk + c (1.50)

and the function to be minimized becomes:

1 1
f (dk ) = dTk Qdk + (xTk Q + cT )dk = dTk Qdk + gkT dk (1.51)
2 2

Note that Q is symmetric, thus Q = QT .

Because xk is a feasible point within the working set Wk , the equality
constraint:
aTi xk = bi , i ∈ Wk (1.52)

14
1.3. Active set method for quadratic programming problems

is satisfied. From (1.52) and (1.48) we obtain the equality constraint of the
new QP subproblem. It will be formulated as:

1
min dTk Qdk + gkT dk (1.53)
dk 2

subject to:
aTi dk = 0, i ∈ Wk (1.54)

We may proceed in a manner similar to the one applied for equality-

constrained QP problems. If we denote by A the matrix having the rows
aTi , for all indices i in the working set:
£ ¤
A = aTi , i ∈ Wk (1.55)

and b the vector of all bi , i ∈ Wk , the Lagrangian is:

1
L(dk , λ) = dTk Qdk + gkT dk + λT Adk (1.56)
2

The first-order optimality condition result as:

∂L
= Qdk + gk + AT λ = 0 (1.57)
∂dk

Adk = 0 (1.58)

and can be written in a matrix form:

" #" # " #
Q AT dk −gk
= (1.59)
A 0 λ 0

The direction dk and the multipliers are the solution of the linear system
(1.59).

• If dk is nonzero we shall move in this direction. Note that for all indices
i in the working set Wk , the term aTi xk does not change as we move
along dk because:

aTi (xk + dk ) = aTi xk = bi and from (1.54): aTi dk = 0 (1.60)

15
 Chapter 1. Quadratic Programming

Since the constraints in Wk were satisfied at xk , they are satisfied at

xk + αk dk , for any value of α.
If the point xk + dk is feasible for all constraints (including the ones that
are not in Wk ), we compute the next point: xk+1 = xk + dk . Otherwise,
we set xk+1 = xk + αk dk , where the step length is chosen as the largest
value in [0, 1) to satisfy all the constraints.
Because the constraints in Wk are satisfied for any αk , a relation to com-
pute the step length will be determined such that xk + αk dk satisfies the
constraints i ∈
/ Wk .

– If aTi dk ≤ 0 for some constraints i ∈

/ Wk , the feasibility is satisfied
for any value of αk because:

aTi (xk + αk dk ) = aTi xk + αk aTi dk ≤ aTi xk ≤ bi (1.61)

– If aTi dk > 0 for some constraints i ∈/ Wk , the value of αk is calcu-

lated from:
aTi (xk + αk dk ) ≤ bi (1.62)

or:
bi − aTi xk
αk ≤ (1.63)
aTi dk

Because we want αk to be as large as possible in the interval [0; 1], it will

be calculated from (Nocedal and Wright, 1999; Bhatti, 2000):
µ ¶
bi − aTi xk
αk = min 1, (1.64)
/ k , aT
i∈W i dk >0
aTi dk

If αk < 1, the movement along dk was blocked by some constraint that

does not belong to Wk . This is a blocking constraint and will be added to
the new working set.
If αk = 1 no new constraints are active for xk + αk dk and there are no
blocking constraints at this stage.

• If dk is zero we have to check for optimality, i.e. all the Lagrange mul-
tipliers for the inequality constraints must be positive. If this is not the

16
1.3. Active set method for quadratic programming problems

case, and some of them are negative, the constraint corresponding to

the multiplier having the smallest negative value will be removed from
the working set.

The iterations are continued until we reach a point that minimizes the
objective over the current working set. In this case the directions dk will be
zero and the Lagrange multipliers are positive.
An algorithm for the active set method is described in Algorithm 1.

Example 1.4 Solve the following QP problem using the active set method:

min f (x1 , x2 ) = x21 + x22 − 4x1 − 4x2 (1.69)

x1 ,x2

subject to

(1) : x1 + x2 ≤ 2
(2) : x1 − 2x2 ≤ 2
(3) : −x1 − x2 ≤ 1 (1.70)
(4) : −2x1 + x2 ≤ 2

The problem is written in the quadratic matrix form:

" #
1 T 2 0
min f (x) = x x + [−4 − 4]x
x 2 0 2
1 T
= x Qx + cT x (1.71)
2

subject to h i
(1) : 1 1 x ≤2
h i
(2) : 1 −2 x ≤2
h i (1.72)
(3) : −1 −1 x ≤ 1
h i
(4) : −2 1 x ≤ 2

17
 Chapter 1. Quadratic Programming

Algorithm 1 Active set method

Define the quadratic objective function: matrix Q and vector c
Define the constraints: vectors aTi and scalars bi , i = 1, p
Select an initial feasible point x0
Find the initial working set W0
Compute the gradient of the objective function at the current point: g0 =
Qx0 + c
Compute the matrix A having the rows ai , i ∈ Wk
Solve the linear system
· ¸· ¸ · ¸
Q AT d0 −g0
= (1.65)
A 0 λ 0

Set k = 0
while not all λi ≥ 0, i ∈ Wk ∩ {1, 2, ..., m} or dk 6= 0 do
if dk = 0 then
Check optimality:
if λi ≥ 0, i ∈ Wk ∩ {1, 2, ..., m} then
Stop and return the current point xk
else
Find the most negative λj
Remove constraint j from the working set Wk
Keep the same point for the next step: xk+1 = xk
end if
else
Compute the step length αk from:
µ ¶
bi − aTi xk
αk = min 1, (1.66)
/ k , aT
i∈W i dk >0
aTi dk

Compute a new point: xk+1 = xk + αk dk

If αk < 1, find the blocking constraint i and add it to the working set
Wk
end if
Set k ← k + 1
Compute the gradient of the objective function at the current point:

gk = Qxk + c (1.67)

Compute the matrix A having the rows ai , i ∈ Wk

Solve the linear system
· ¸· ¸ · ¸
Q AT dk −gk
= (1.68)
A 0 λ 0
18
end while
1.3. Active set method for quadratic programming problems

The vectors aTi , i = 1, 4 are:

h i
aT1 = 1 1 (1.73)
h i
aT2 = 1 −2 (1.74)
h i
aT3 = −1 −1 (1.75)
h i
aT4 = −2 1 (1.76)

• We select as the starting point x0 = [0 − 1]T . The objective function, the

constraints and the iterates are illustrated in Figure 1.3. The feasible region is
the interior and the sides of the polygon bordered by the constraints numbered
as in the relations (1.70).

1.5 (1)

(4)
1 STOP
x2

0.5

0
(3)
(2)
−0.5

START
−1
−1 −0.5 0 0.5 1 1.5 2
x1

Figure 1.3: Active set method with x0 = [0 − 1]T and W0 = {2, 3}

For the initial point selected as the vertex (0, -1), (or as the intersection of the
lines x1 = 2x2 = 2 and −x1 − x2 = 1), the constraints (2) and (3) are active.
The initial working set is then W0 = {2, 3}. We may choose W0 = {2}, or
{3}, or an empty set, and the algorithm will return the same result, but the
iterates will follow another trajectory.

19
 Chapter 1. Quadratic Programming

The rows of matrix A are aTi , i = {2, 3}:

" #
1 −2
A= (1.77)
−1 −1

The gradient g0 is obtained from Q, c and x0 and the solution of the linear
system (1.65) is: " # " #
0 −0.6667
d0 = , λ= (1.78)
0 −4.6667

Step 1. The vector d0 is zero, thus we check optimality: all λ-multipliers are
negative. We remove the constraint (3) from the working set since it has
the most negative multiplier and obtain W1 = {2}. The matrix A is
now: A = [1 − 2] and the linear system (1.68) has the solution:
" #
2.8
d1 = , λ = −1.6 (1.79)
1.4

Step 2. The vector d1 is nonzero. The step length formula (1.66) yields: α =
0.7143. The only positive product aTi d1 where i ∈/ Wk is obtained for
i = 1. The first constraint will be added to the working set and W2 =
{2, 1}.
The new iterate is: " #
2
x2 = x1 + αd1 = (1.80)
0
and the solution of (1.68) result as:
" # " #
0 −1.3333
d2 = , λ= (1.81)
0 1.3333

Step 3. The vector d2 is zero, thus we check optimality: the first multiplier
is negative and the constraint (2) will be removed from the working set
which becomes: W3 = {1}. The iterate x3 is the same as in the previous
step, the matrix A = [1 1] and the linear system (1.68) has the solution:
" #
−1
d3 = , λ=2 (1.82)
1

20
1.3. Active set method for quadratic programming problems

Step 4. The vector d1 is nonzero although the multiplier is positive. The step
length formula (1.66) yields: α = 1. The working set in the same as
before W4 = {1} and the new iterate is:
" #
1
x4 = x3 + d3 = (1.83)
1

The linear system (1.68) has the solution:

" #
0
d4 = , λ=2 (1.84)
0

Step 5. The vector d4 is zero and the optimality condition is satisfied (λ =

2 > 0). The minimizer of the objective function subject to the given
constraints is x∗ = [1 1]T .

The steps taken in this procedure are shown in Figure 1.3. The same result
is obtained if the initial point is changed to another feasible location, or if we
choose another initial working set, as shown in the following cases:

• We select as the starting point x0 = [−0.2 − 0.8]T and the initial working
set W0 = {3}. The objective function, the constraints and the iterates are
illustrated in Figure 1.4.

• If the initial point is x0 = [0 − 1]T and the initial working set W0 = {3},
the result is the same as in previous cases, as shown in Figure 1.5.

• Another test was performed for an initial point x0 = [0 − 1]T and an empty
working set W0 = {∅}. The trajectory of the iterates is different from previous
cases, but the final point is the same as shown in Figure 1.6.

21
 Chapter 1. Quadratic Programming

1.5
(1)

(4)
1 STOP

x2
0.5

0
(3)

−0.5
(2)

START
−1
−1 −0.5 0 0.5 1 1.5 2
x1

Figure 1.4: Active set method with x0 = [−0.2 − 0.8]T and W0 = {3}

1.5 (1)

(4)
1 STOP
x2

0.5

0
(3)
(2)
−0.5

START
−1
−1 −0.5 0 0.5 1 1.5 2
x1

Figure 1.5: Active set method with x0 = [0 − 1]T and W0 = {3}

22
1.3. Active set method for quadratic programming problems

1.5 (1)

(4)
1 STOP
x2

0.5

0
(3)
(2)
−0.5

START
−1
−1 −0.5 0 0.5 1 1.5 2
x1

Figure 1.6: Active set method with x0 = [0 − 1]T and W0 = {∅}

23
 Chapter 1. Quadratic Programming

24
Bibliography

Bhatti, M. A. (2000). Practical Optimization Methods: With Mathematica Appli-

cations. Springer.

Nocedal, J. and Wright, S. J. (1999). Numerical Optimization. Springer series in

operation research. Springer-Verlag.

25