book_tem

i i
2010/7/27
page 91
i i

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Chapter 5

Newton Methods for Equality

Constrained Optimization

This chapter extends the Newton-based algorithms in Chapter 3 to deal with equality
constrained optimization. It generalizes the concepts of Newton iterations and associ-
ated globalization strategies to the optimality conditions for this constrained optimiza-
tion problem. As with unconstrained optimization we focus on two important aspects:
solving for the Newton step and ensuring convergence from poor starting points. For
the first aspect, we focus on properties of the KKT system and introduce both full- and
reduced-space approaches to deal with equality constraint satisfaction and constrained
minimization of the objective function. For the second aspect, the important properties
of penalty-based merit functions and filter methods will be explored, both for line search
and trust region methods. Several examples are provided to illustrate the concepts devel-
oped in this chapter and to set the stage for the nonlinear programming codes discussed in
Chapter 6.

5.1 Introduction to Equality Constrained Optimization

The previous chapter deals with optimality conditions for the nonlinear program (4.1). This
chapter considers solution strategies for a particular case of this problem. Here we consider
the equality constrained NLP problem of (5.1) given by

min f (x) (5.1)

s.t. h(x) = 0,

and we assume that the functions f (x) : Rn → R and h(x) : Rn → Rm have continuous
first and second derivatives. First, we note that if the constraints h(x) = 0 are linear, then
(5.1) is a convex problem if and only if f (x) is convex. On the other hand, as discussed in
Chapter 4, nonlinear equality constraints imply nonconvex problems even if f (x) is convex.
As a result, the presence of nonlinear equalities leads to violation of convexity properties
and will not guarantee that local solutions to (5.1) are global minima. Therefore, unless
additional information is known about (5.1), we will be content in this chapter to determine
only local minima.

91

i i

i i
 book_tem
i i
2010/7/27
page 92
i i

92 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

In analogy to Chapter 3, we need to consider the following questions:

• When is the Newton iteration well defined, and can we maintain a fast (local) con-
vergence rate to the solution of (5.1)?
• Can we converge to local solutions from distant starting points?
The next section analyzes the Newton iteration through a characterization of the so-
called KKT matrix. Here we consider properties that lead to a nonsingular matrix and are also
compatible with the first and second order optimality conditions. Moreover, some important
remedies are discussed when these properties are not satisfied. Related to these properties,
Section 5.3 discusses the solution of the Newton step for full- and reduced-space approaches.
Section 5.4 then deals with the extension of these methods if second derivatives are not avail-
able. As in Chapter 3, we apply quasi-Newton updates that provide curvature information
based on differences in gradient vectors and adapt these updates to constrained optimization.
We then consider convergence from poor starting points using globalization strategies
that extend from the line search and trust region strategies discussed in Chapter 3. In contrast
to that chapter, progress in solving (5.1) requires satisfaction of the equality constraints
along with constrained minimization of f (x). Consequently, Section 5.5 discusses concepts
of merit functions and filter methods which measure satisfaction of both goals. Sections 5.6
and 5.7 discuss the application of line search and trust region methods, respectively, that
enforce these measures and lead to convergence to local solutions. Finally, reconciliation of
these globalization methods with fast Newton steps requires additional analysis. Section 5.8
discusses the Maratos effect which can lead to slow convergence, along with the remedy
of second order corrections, which is essential to prevent premature termination of these
methods. The treatment of these concepts leads to a family of powerful methods to solve
(5.1) and also lays the framework to deal with more general NLP methods in Chapter 6.

5.2 Newton’s Method with the KKT Matrix

The first order KKT conditions for local optimality can be simplified for equality con-
straints. Defining the Lagrange function as L(x, v) = f (x) + v T h(x), we can write the KKT
conditions as
∇x L(x ∗ , v ∗ ) = ∇f (x ∗ ) + ∇h(x ∗ )T v ∗ = 0, (5.2)
h(x ∗ ) = 0.
As noted in the previous chapter, because linearized information is used to characterize the
optimum of a nonlinearly constrained problem, a constraint qualification must be satisfied
at x ∗ so that (5.2) are necessary conditions for a local optimum for (5.1). For this problem we
again require the LICQ: equality constraint gradients at x ∗ must be linearly independent,
i.e., the matrix ∇h(x ∗ ) has full column rank. With this constraint qualification, the KKT
multipliers (v ∗ ) are unique at the optimal solution.
The sufficient (necessary) second order conditions state that for x ∗ and v ∗ satisfy-
ing first order conditions, positive (nonnegative) curvature holds for all constrained direc-
tions, p. For NLP (5.1) the sufficient conditions require
pT ∇xx L(x ∗ , v ∗ )p > 0 for all p  = 0, ∇h(x ∗ )T p = 0. (5.3)
The corresponding necessary condition requires that the inequality in (5.3) be nonstrict.

i i

i i
 book_tem
i i
2010/7/27
page 93
i i

5.2. Newton’s Method with the KKT Matrix 93

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Finding a local solution of (5.1) can therefore be realized by solving (5.2) and then
checking the second order conditions (5.3). In this chapter we develop Newton-based strate-
gies for this task. As with the Newton methods in Chapter 3, a number of important concepts
need to be developed.
Solution of (5.2) with Newton’s method relies on the generation of Newton steps from
the following linear system:
    
Wk Ak dx ∇L(x k , v k )
= − , (5.4)
(Ak )T 0 dv h(x k )

where W k = ∇xx L(x k , v k ) and Ak = ∇h(x k ). Defining the new multiplier estimate as v̄ =
v k + dv and substituting into (5.4) leads to the equivalent system:
    
Wk Ak dx ∇f (x k )
=− . (5.5)
(Ak )T 0 v̄ h(x k )

Note that (5.5) are also the first order KKT conditions of the following quadratic program-
ming problem:
1
min ∇f (x k )T dx + dxT W k dx (5.6)
dx 2
s.t. h(x k ) + (Ak )T dx = 0.

Using either (5.4) or (5.5), the basic Newton method can then be stated as follows.

Algorithm 5.1.
Choose a starting point (x 0 , v 0 ).

For k ≥ 0 while dxk , dvk  > 1 and ∇L(x k , v k ) > 2 :

1. Evaluate ∇f (x k ), Ak , and W k at x k and v k . If the matrix in (5.4), the so-called KKT
matrix, is singular, STOP.
2. Solve the linear system (5.4) (or equivalently, (5.5)).
3. Set x k+1 = x k + dx , v k+1 = v k + dv (or equivalently, v k+1 = v̄), and k = k + 1.
As with unconstrained optimization, we would expect this method to have the follow-
ing convergence property near the optimal solution.

Theorem 5.1 Consider a solution x ∗ , v ∗ , which satisfies the sufficient second order con-
ditions and ∇h(x ∗ ) is full column rank (LICQ). Moreover, assume that f (x) and h(x) are
twice differentiable and ∇ 2 f (x) and ∇ 2 h(x) are Lipschitz continuous in a neighborhood of
this solution. Then, by applying Algorithm 5.1, and with x 0 and v 0 sufficiently close to x ∗
and v ∗ , there exists a constant Ĉ > 0 such that
 k+1   k 
 x − x∗   ∗ 2
  ≤ Ĉ  x − x 
 v k+1 − v ∗   vk − v∗  , (5.7)

i.e., the convergence rate for {x k , v k } is quadratic.

i i

i i
 book_tem
i i
2010/7/27
page 94
i i

94 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

The proof of this theorem follows the proof of Theorem 2.20 as long as the
KKT matrix in (5.4) is nonsingular at the solution (see Exercise 1). In the remainder of
this section we consider properties of the KKT matrix and its application in solving (5.1).

5.2.1 Nonsingularity of KKT Matrix

We begin with the following properties.
Theorem 5.2 (Sylvester’s Law of Inertia [169]).
• The inertia of a symmetric matrix B is defined as the triple that contains the numbers
of positive, negative, and zero eigenvalues of B, i.e., I n(B) = (n+ , n− , n0 ).
• Sylvester’s law of inertia states that the inertia of a symmetric matrix is invariant
under the so-called congruence transformations. That is, given a square nonsingular
matrix S, the inertia of S T BS is the same as the inertia of B.
• It follows that if S T BS is nonsingular, then so is B.
We now consider a local solution of (5.1), x ∗ , v ∗ , that satisfies first and sufficient second
order KKT conditions as well as the LICQ. Since A∗ = ∇h(x ∗ ) is full column rank, we
can use Definition 4.19 to obtain, Z ∗ ∈ Rn×(n−m) , a full rank, null space basis matrix, with
(A∗ )T Z ∗ = 0. Moreover, at iteration k we will also assume that Ak is full column rank and
similarly define Z k with (Ak )T Z k = 0. Finally, we define other basis matrices Y ∗ , Y k ∈ Rn×m
so that the n × n matrices [Y ∗ Z ∗ ] and [Y k Z k ] are nonsingular.
Partitioning the search direction into two components, dx = Y ∗ pY + Z ∗ pZ , we can
apply the transformation to (5.4) at the solution of (5.1) to obtain
   ∗T 
 ∗ ∗T  ∗ ∗
 ∗ ∗  pY (Y ) ∇L(x ∗ , v ∗ )
[Y Z ] 0 W A [Y Z ] 0 
pZ  = −  (Z ∗ )T ∇L(x ∗ , v ∗ )  ,
0 I (A∗ )T 0 0 I
dv h(x ∗ )
(5.8)

or simply
 ∗T ∗ ∗    ∗T 
(Y ) W Y (Y ∗ )T W ∗ Z ∗ (Y ∗ )T A∗ pY (Y ) ∇L(x ∗ , v ∗ )
 (Z ∗ )T W ∗ Y ∗ (Z ∗ )T W ∗ Z ∗ 0   pZ  = −  (Z ∗ )T ∇L(x ∗ , v ∗ )  .
(A∗ )T Y ∗ 0 0 dv h(x ∗ )
(5.9)

This linear system can be characterized by the following properties.

• Because [Y ∗ Z ∗ ] is nonsingular and A∗ is full column rank, (A∗ )T [Y ∗ Z ∗ ] =
[(A∗ )T Y ∗ | 0] is full row rank, i.e., (A∗ )T Y ∗ is nonsingular.
• From the sufficient second order conditions (5.3), we can define

p = Z ∗ p̄ and p̄T (Z ∗ )T ∇xx L(x ∗ , v ∗ )Z ∗ p̄ > 0

for all p̄ ∈ Rn−m . This is equivalent to positive definiteness of (Z ∗ )T W ∗ Z ∗ .

i i

i i
 book_tem
i i
2010/7/27
page 95
i i

5.2. Newton’s Method with the KKT Matrix 95

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

• Because the KKT conditions (5.2) are satisfied at x ∗ , v ∗ , the right-hand side of (5.9)
equals zero.
As a result, we can use the bottom row of (5.9) to solve uniquely for pY = 0. Then, from
the second row of (5.9), we can solve uniquely for pZ = 0. Finally, from the first row of
(5.9), we solve uniquely for dv = 0. This unique solution implies that the matrix in (5.9)
is nonsingular, and from Sylvester’s law of inertia we have that the KKT matrix in (5.4) is
nonsingular as well at x ∗ and v ∗ . Note that nonsingularity is a key property required for the
proof of Theorem 5.1.

5.2.2 Inertia of KKT Matrix

We now consider additional information regarding the KKT matrix at the solution of (5.1).
If we assume that first order and sufficient second order KKT conditions are satisfied at x ∗
and v ∗ , then using the definition in (5.8), we know that the matrices
 
 ∗ ∗
 (Y ∗ )T W ∗ Y ∗ (Y ∗ )T W ∗ Z ∗ (Y ∗ )T A∗
W A
and  (Z ∗ )T W ∗ Y ∗ (Z ∗ )T W ∗ Z ∗ 0 
(A∗ )T 0 ∗ T ∗
(A ) Y 0 0
have the same inertia. Moreover, by defining the matrix
 
I 0 − 12 (Y ∗ )T W ∗ Y ∗ [(A∗ )T Y ∗ ]−1
V =  0 I −(Z ∗ )T W ∗ Y ∗ [(A∗ )T Y ∗ ]−1  ,
0 0 [(A∗ )T Y ∗ ]−1
we have
   
(Y ∗ )T W ∗ Y ∗ (Y ∗ )T W ∗ Z ∗ (Y ∗ )T A∗ 0 0 I
V  (Z ∗ )T W ∗ Y ∗ (Z ∗ )T W ∗ Z ∗ 0 V T =  0 (Z ∗ )T W ∗ Z ∗ 0  (5.10)
(A∗ )T Y ∗ 0 0 I 0 0
and from Theorem 5.2, it is clear that the right-hand matrix in (5.10) has the same inertia
as the KKT matrix in (5.4). A simple permutation of the rows and columns of the matrix on
the right-hand side (which leaves the inertia unchanged) leads to
 
T 0
,
0 (Z ∗ )T W ∗ Z ∗
 
where T is a block diagonal matrix with m blocks of the form 01 10 . Because each block
has eigenvalues of +1 and −1, and because (Z ∗ )T W ∗ Z ∗ ∈ R(n−m)×(n−m) and is positive
definite (it has n − m positive eigenvalues), we see that the KKT matrix at x ∗ and v ∗
is nonsingular with an inertia of (n, m, 0). Moreover, by Lipschitz continuity of A(x) and
W (x, v), the KKT matrix maintains this inertia in a nonzero neighborhood around the optimal
solution.
Example 5.3 Consider the following equality constrained quadratic programming problem:
1 2
min (x + x22 ) (5.11)
2 1
s.t. x1 + x2 = 1.

i i

i i
 book_tem
i i
2010/7/27
page 96
i i

96 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

The solution can be found by solving the first order KKT conditions:
x1 + v = 0,
x2 + v = 0,
x1 + x2 = 1
with the solution x1∗ = 12 , x2∗ = 12 , and v1∗ = − 12 . We can define a null space basis as Z ∗ =
[1 −1]T , so that AT Z ∗ = 0. The reduced Hessian at the optimum is (Z ∗ )T ∇xx L(x ∗ , v ∗ )Z ∗ =
2 > 0 and the sufficient second order conditions are satisfied. Moreover, the KKT matrix at
the solution is given by
 
  1 0 1
W∗ A∗
K= = 0 1 1 
(A∗ )T 0
1 1 0
with eigenvalues, −1, 1, and 2. The inertia of this system is therefore I n(K) = (2, 1, 0) =
(n, m, 0) as required.

5.3 Taking Newton Steps

We now consider the solution of the equality constrained problems with the basic Newton
method. As seen in Algorithm 5.1, the key step is the solution of the linear system (5.4).
Especially for large systems, this linear system can be solved with linear solvers that use
one of the following three approaches:
• A full-space solution is particularly useful if the linear system (5.4) is sparse and can
be represented all at once with no particular decomposition related to structure.
• Reduced-space decomposition is particularly useful for (5.4) if n − m is small (i.e.,
few degrees of freedom) and second derivative information is hard to obtain and must
be approximated.
• Special purpose decompositions can be applied that exploit other specialized struc-
tures, e.g., if the KKT matrix is banded, bordered block diagonal, or preconditioned
to solve with iterative linear solvers. Depending on problem size and available archi-
tectures, these problems can be treated with either specialized, in-core linear algebra
or out of core with a specialized decomposition.
In this section we focus on the first two general approaches and discuss the determi-
nation of well-defined Newton steps.

5.3.1 Full-Space Newton Steps

Full-space Newton steps are determined by solution of (5.4), or equivalently, (5.5). If the
solution of (5.1) satisfies the LICQ and sufficient second order conditions, we know that the
KKT matrix at the solution is nonsingular and has the correct inertia. However, at a given
iteration k, particularly far from the solution, there is no expectation that these properties
would hold. As a result, some corrections to (5.4) may be required to ensure well-defined
Newton steps.

i i

i i
 book_tem
i i
2010/7/27
page 97
i i

5.3. Taking Newton Steps 97

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

A number of direct solvers is available for symmetric linear systems. In particular, for
positive definite symmetric matrices, efficient Cholesky factorizations can be applied. These
can be represented as LDLT , with L a lower triangular matrix and D diagonal. On the other
hand, because the KKT matrix in (5.4) is indefinite, possibly singular, and frequently sparse,
another symmetric factorization (such as the Bunch–Kaufman [74] factorization) needs to
be applied. Denoting the KKT matrix as K, and defining P as an orthonormal permutation
matrix, the indefinite factorization allows us to represent K as P T KP = LBLT , where the
block diagonal matrix B is determined to have 1 × 1 or 2 × 2 blocks. From Sylvester’s law of
inertia we can obtain the inertia of K cheaply by examining the blocks in B and evaluating
the signs of their eigenvalues.
To ensure that the iteration matrix is nonsingular, we first check whether the KKT
matrix at iteration k has the correct inertia, (n, m, 0), i.e., n positive, m negative, and no
zero eigenvalues. If the inertia of this matrix is incorrect, we can modify (5.4) to form the
following linear system:
 k    
W + δW I Ak dx ∇L(x k , v k )
= − . (5.12)
(Ak )T −δA I dv h(x k )

Here, different trial values can be selected for the scalars δW , δA ≥ 0 until the inertia is
correct. To see how this works, we first prove that such values of δW and δA can be found,
and then sketch a prototype algorithm.

Theorem 5.4 Assume that matrices W k and Ak are bounded in norm; then for any δA > 0
there exist suitable values of δW such that the matrix in (5.12) has the inertia (n, m, 0).

Proof: In (5.12) Ak has column rank r ≤ m. Because Ak may be rank deficient, we represent
it by the LU factorization Ak LT = [U T | 0], with upper triangular U ∈ Rr×n of full rank
and nonsingular lower triangular L ∈ Rm×m . This leads to the following factorization of the
KKT matrix:
 k 
 k  W + δW I [U T | 0]
W + δW I Ak 
V̄ V̄ T =  U , (5.13)
(Ak )T −δA I −δA LLT
0

where  
I0
V̄ = .
0L
From Theorem 5.2, it is clear that the right-hand matrix has the same inertia as the KKT
matrix in (5.12). Moreover, defining the nonsingular matrix
 −1

I δA [U T | 0]L−T L−1
Ṽ =
0 I

allows us to write the factorization

 k 
W + δW  
 I [U | 0]
T
  Ŵ 0
Ṽ U Ṽ =
T
, (5.14)
−δA LLT 0 −δA LLT
0

i i

i i
 book_tem
i i
2010/7/27
page 98
i i

98 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

where
 
−1 −T −1 U −1 k
Ŵ = W + δW I
k
+ δA [U T | 0]L L = W k + δW I + δA A (Ak )T .
0

We note that if δA > 0, the matrix −δA LLT has inertia (0, m, 0) and we see that the right-
hand matrix of (5.14) has an inertia given by I n(Ŵ ) + (0, m, 0). Again, we note that this
matrix has the same inertia as the KKT matrix in (5.12).
To obtain the desired inertia, we now examine Ŵ more closely. Similar to the decom-
position in (5.9), we define full rank range and null-space basis matrices, Y ∈ Rn×r and Z ∈
Rn×(n−r) , respectively, with Y T Z = 0. From Z T Ak = 0, it is clear that Z T [U T | 0]L−T = 0
and also that (Y )T Ak ∈ Rr×m has full column rank. For any vector p ∈ Rn with p =
YpY + ZpZ , we can choose positive constants ã and c̃ that satisfy
c̃pY pZ  ≥ −pZT Z T (W k + δW I )YpY = −pZT Z T W k YpY ,
pY Y T Ak (Ak )T YpY ≥ ãpY 2 .
For a given δA > 0, we can choose δW large enough so that for all pZ ∈ Rn−r and pY ∈ Rr
with
pZT Z T (W k + δW I )ZpZ ≥ m̃pZ 2 for m̃ > 0,
pYT Y T (W k + δW I )YpY ≥ −w̃pY  2
for w̃ ≥ 0,

−1 c̃2
we have ãδA > w̃ and m̃ ≥ −1 . With these quantities, we obtain the following
(ãδA −w̃)
relations:
  −1 k
 
Y T (W k + δW I + δA A (Ak )T )Y Y T W kZ pY
p Ŵ p =
T
pYT | pZT T
Z W Y k Z (W k + δW I )Z
T pZ
= pYT Y T (Ŵ )YpY + 2pYT Y T W k ZpZ + pZT Z T (W k + δW I )ZpZ
−1
≥ (ãδA − w̃)pY 2 − 2c̃pY pZ  + m̃pZ 2
 2
−1 c̃
= (ãδA − w̃) pY  −
1/2
−1
pz 
(ãδA − w̃)1/2

c̃2
+ m̃ − −1
pz 2 > 0
(ãδA − w̃)

for all values of p. As a result, Ŵ is positive definite, the inertia of the KKT matrix in (5.12)
is (n, m, 0), and we have the desired result.

These observations motivate the following algorithm (simplified from [404]) for
choosing δA and δW for a particular Newton iteration k.

Algorithm 5.2.
Given constants 0 < δ̄Wmin < δ̄ 0 < δ̄ max ; δ̄ > 0; and 0 < κ < 1 < κ . (From [404], rec-
W W A l u
ommended values of the scalar parameters are δ̄W min = 10−20 , δ̄ 0 = 10−4 , δ̄ max = 1040 ,
W W
δ̄A = 10−8 , κu = 8, and κl = 13 .) Also, for the first iteration, set δW
last := 0.

i i

i i
 book_tem
i i
2010/7/27
page 99
i i

5.3. Taking Newton Steps 99

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

At each iteration k:
1. Attempt to factorize the matrix in (5.12) with δW = δA = 0. For instance, this can be
done with an LBLT factorization; the diagonal (1 × 1 and 2 × 2) blocks of B are then
used to determine the inertia. If the matrix has correct inertia, then use the resulting
search direction as the Newton step. Otherwise, continue with step 2.
2. If the inertia calculation reveals zero eigenvalues, set δA := δ̄A . Otherwise, set δA := 0.
last = 0, set δ := δ̄ 0 , else set δ := max{δ̄ min , κ δ last }.
3. If δW W W W W l W

4. With updated values of δW and δA , attempt to factorize the matrix in (5.12). If the
last := δ and use the resulting search direction in the line
inertia is now correct, set δW W
search. Otherwise, continue with step 5.
5. Continue to increase δW and set δW := κu δW .
max , abort the search direction computation. The matrix is severely ill-
6. If δW > δ̄W
conditioned.

This correction algorithm provides a systematic way to choose δW and δA to obtain

the required inertia. Step 1 checks to see if (5.4) has the correct inertia so that an uncor-
rected Newton search direction can be used whenever possible. Otherwise, only a positive
(and usually small) δA is chosen if the unmodified iteration matrix has a zero eigenvalue,
as we assume that the singularity is caused by a rank-deficient constraint Jacobian. This
correction helps to maintain the inertia for the negative eigenvalues. On the other hand, if
the factorization is still unsuccessful, larger values of δW are chosen, but only to find the
smallest correction necessary to yield a successful factorization. The algorithm can also be
extended (see [404]) to prevent cycling of δW when successive iterations and successive
iterates are close together.

5.3.2 Reduced-Space Newton Steps

The reduced-space option is well suited to problems where n − m is small and especially
when second order information needs to be approximated, as shown in Section 5.4. In
contrast to the full-space step (5.12), the reduced-space approach is motivated by the matrix
decomposition for the linear system in (5.9) along with the Newton step (5.5). To compute
the search direction at iteration k, we consider components of this direction in two separate
spaces (see, e.g., [162]). We write the solution dx of (5.5) as
dx = Y k pY + Z k pZ , (5.15)
where Z k is an n × (n − m) matrix spanning the null space of (Ak )T , and Y k is any n × m
matrix that allows [Y k Z k ] to be nonsingular. We also refer to the two components Y k pY
and Z k pZ as normal and tangential steps, respectively, as shown in Figure 5.1. Applying
the decomposition (5.8)–(5.9) with the analogous terms at iteration k leads to
    
(Y k )T W k Y k (Y k )T W k Z k (Y k )T Ak pY (Y k )T ∇f (x k )
 (Z k )T W k Y k (Z k )T W k Z k 0   pZ  = −  (Z k )T ∇f (x k )  .
k
(A ) YT k 0 0 v̄ h(x k )
(5.16)

i i

i i
 book_tem
i i
2010/7/27
page 100
i i

100 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Figure 5.1. Normal and tangential steps for the solution dx of (5.4) or (5.6). Note
also that if a coordinate basis (Yc ) is chosen, the normal and tangential steps may not be
orthogonal and the steps Yc pY c are longer than YpY .

In contrast to the full-space method, this decomposition requires that Ak have full column
rank for all k. From the bottom row in (5.16) we obtain

h(x k ) + (Ak )T Y k pY = 0.

Therefore the normal component, pY , is given by

pY = −[(Ak )T Y k ]−1 h(x k ), (5.17)

so that dxk has the form

dx = −Y k [(Ak )T Y k ]−1 h(x k ) + Z k pZ . (5.18)

To determine the tangential component, pZ , we substitute (5.17) in the second row of (5.16).
The following linear system:

(Z k )T W k Z k pZ = −[(Z k )T ∇f (x k ) + (Z k )T W k Y k pY ] (5.19)

can then be solved if (Z k )T W k Z k is positive definite, and this property can be verified
through a successful Cholesky factorization. Otherwise, if (Z k )T W k Z k is not positive def-
inite, this matrix can be modified either by adding a sufficiently large diagonal term, say
δR I , or by using a modified Cholesky factorization, as discussed in Section 3.2.
Once pZ is calculated from (5.19) we use (5.18) to obtain dx . Finally, the top row in
(5.16) can then be used to update the multipliers:

v̄ = −[(Y k )T Ak ]−1 [(Y k )T ∇f (x k ) + (Y k )T W k Z k pZ + (Y k )T W k Y k pY ]

= −[(Y k )T Ak ]−1 [(Y k )T ∇f (x k ) + (Y k )T W k dx ]. (5.20)

i i

i i
 book_tem
i i
2010/7/27
page 101
i i

5.3. Taking Newton Steps 101

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Because dxk → 0 as the algorithm converges, a first order multiplier calculation may be used
instead, i.e.,
v̄ = −[(Y k )T Ak ]−1 (Y k )T ∇f (x k ), (5.21)
and we can avoid the calculation of (Y k )T W k Y k and (Y k )T W k Z k . Note that the multipliers
from (5.21) are still asymptotically correct.
A dominant part of the Newton step is the computation of the null-space and range-
space basis matrices, Z and Y , respectively. There are many possible choices for these basis
matrices, including the following three options.
• By computing a QR factorization of A, we can define Z and Y so that they have
orthonormal columns, i.e., Z T Z = In−m , Y T Y = Im , and Z T Y = 0. This gives a
well-conditioned representation of the null space and range space of A, However,
Y and Z are dense matrices, and this can lead to expensive computation when the
number of variables (n) is large.
• A more economical alternative for Z and Y can be found through a simple elimination
of dependent variables [134, 162, 294]. Here we permute the components of x into m
dependent or basic variables (without loss of generality, we select the first m variables)
and n − m independent or superbasic variables. Similarly, the columns of (Ak )T are
permuted and partitioned accordingly to yield
(Ak )T = [AkB | AkS ]. (5.22)
We assume that the m×m basis matrix AkB is nonsingular and we define the coordinate
bases    
−(AkB )−1 AkS I
Z =
k
and Y =k
. (5.23)
I 0
Note that in practice Z k is not formed explicitly; instead we can compute and store
the sparse LU factors of AkB . Due to the choice (5.23) of Y k , the normal component
determined in (5.17) and multipliers determined in (5.21) simplify to
pY = −(AkB )−1 h(x k ), (5.24)
v̄ = −(AkB )−T (Y k )T ∇f (x k ). (5.25)
• Numerical accuracy and algorithmic performance are often improved if the tangential
and normal directions, ZpZ and YpY , can be maintained orthogonal and the length
of the normal step is minimized. This can, of course, be obtained through a QR
factorization, but the directions can be obtained more economically by modifying the
coordinate basis decomposition and defining the orthogonal bases
   
−(AkB )−1 AkS I
Z =
k
and Y = k
. (5.26)
I (AkS )T (AkB )−T
Note that (Z k )T Y k = 0, and from the choice (5.26) of Y k , the calculation in (5.17)
can be written as
pY = −[I + (AkB )−1 AkS (AkS )T (AkB )−T ]−1 (AkB )−1 h(x k )
 
= − I − (AkB )−1 AkS [I + (AkS )T (AkB )−T (AkB )−1 AkS ]−1 (AkS )T (AkB )−T
×(AkB )−1 h(x k ), (5.27)

i i

i i
 book_tem
i i
2010/7/27
page 102
i i

102 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

where the second equation follows from the application of the Sherman–Morison–
Woodbury formula [294]. An analogous expression can be derived for v̄. As with
coordinate bases, this calculation requires a factorization of AkB , but it also requires a
factorization of

[I + (AkS )T (AkB )−T (AkB )−1 AkS ]

with an additional cost that is O((n − m)3 ).

• The normal and tangential search directions can also be determined implicitly through
an appropriate modification of the full-space KKT system (5.4). As shown in Exer-
cise 3, the tangential step dt = Z k pZ can be found from the following linear system:
    
Wk Ak dt ∇f (x k )
= − . (5.28)
(Ak )T 0 v 0

Implicit calculation of the normal step, dn = Y k pY , requires a substitution of the

Hessian matrix by an alternative matrix W̃ . Here, the normal step is determined by
the following system:
    
W̃ Ak dn 0
=− , (5.29)
(Ak )T 0 v h(x k )

where the condition

 (Z k )T W̃ Y k = 0 must hold. Popular choices for the Hessian term
are W̃ = 0 I and W̃ = I for the coordinate and orthogonal bases, respectively.
0 0

5.4 Quasi-Newton Methods

If exact second derivative information is unavailable or difficult to evaluate, one can apply a
quasi-Newton update matrix B k to substitute for W k or the reduced Hessian. In either case,
the BFGS and SR1 updates developed in Section 3.3 can be applied as before, but with a
slight modification for the constrained system. In particular, for the full-space method, we
choose the matrix B k as a quasi-Newton approximation to the Hessian of the Lagrangian,
W (x k , v k ). As in Chapter 3, we apply the secant relation (3.9):

B k+1 s = y,

but with

s = x k+1 − x k , y = ∇L(x k+1 , v k+1 ) − ∇L(x k , v k+1 ). (5.30)

Note that because we approximate the Hessian with respect to x, both terms in the definition
of y require the multiplier evaluated at v k+1 .
With these definitions of s and y, we can directly apply the BFGS update (3.18):

yy T B k ss T B k
B k+1 = B k + − T k (5.31)
sT y s B s

i i

i i
 book_tem
i i
2010/7/27
page 103
i i

5.4. Quasi-Newton Methods 103

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

or the SR1 update (3.10):

(y − B k s)(y − B k s)T
B k+1 = B k + . (5.32)
(y − B k s)T s

However, unlike the Hessian matrix for unconstrained optimization, discussed in Chap-
ter 3, W (x, v) is not required to be positive definite at the optimum. Only its projection,
i.e., Z(x ∗ )T W (x ∗ , v ∗ )Z(x ∗ ), needs to be positive definite to satisfy sufficient second order
conditions. As a result, some care is needed in applying the update matrices using (5.31) or
(5.32).
As discussed in Chapter 3, these updates are well defined when s T y is sufficiently
positive for BFGS and (y − B k s)T s  = 0 for SR1. Otherwise, the updates are skipped for a
given iteration, or Powell damping (see Section 3.3) can be applied for BFGS.

5.4.1 A Quasi-Newton Full-Space Method

In the full-space method, we substitute B k for W k in (5.4), i.e.,
    
Bk Ak dx ∇L(x k , v k )
=− . (5.33)
(Ak )T 0 dv h(x k )

However, because the updated quasi-Newton matrix B k is dense, handling this matrix di-
rectly leads to factorizations of the Newton step that are O(n3 ). For large problems, this
calculation can be prohibitively expensive. Instead we would like to exploit the fact that
Ak is sparse and B k has the structure given by (5.31) or (5.32). We can therefore store the
updates y and s for successive iterations and incorporate these into the solution of (5.33).
For this update, we store only the last q updates and apply the compact limited memory
representation from [83] (see Section 3.3), along with a sparse factorization of the initial
quasi-Newton matrix, B 0 , in (5.33). Also, we assume that B 0 is itself sparse; often it is
chosen to be diagonal. For the BFGS update (5.31), this compact representation is given by
 −1  
SkT B 0 Sk Lk SkT B 0
B k+1 = B 0 − [B 0 Sk Y k ] , (5.34)
LTk −Dk (Y k )T

where

Dk = diag[(s k−q+1 )T (y k−q+1 ), . . . , (s k )T y k ], Sk = [s k−q+1, . . . , s k ], Y k = [y k−q+1, . . . , y k ],

and

(s k−q+i )T (y k−q+j ), i > j,
(Lk ) =
0 otherwise.

With this compact form, we represent the quasi-Newton matrix as

B k+1 = B 0 + Vk VkT − Uk UkT . (5.35)

i i

i i
 book_tem
i i
2010/7/27
page 104
i i

104 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

By writing (5.34) as
 1/2 1/2 −1 1/2 −1  
−Dk Dk LTk Dk 0 (Y k )T
B k+1 = B 0 − [Y k B 0 Sk ] −1/2 ,
0 JkT −LTk Dk Jk SkT B 0
(5.36)
where Jk is a lower triangular factorization constructed from the Cholesky factorization that
satisfies
Jk JkT = SkT B 0 Sk + Lk Dk−1 LTk , (5.37)
−1/2
we now define Vk = Y k Dk , Uk = (B 0 Sk + Y k Dk−1 LTk )Jk−T , Ũ T = [UkT 0], and Ṽ T =
[VkT 0] and consider the matrices
   
Bk Ak (B 0 + Vk VkT − Uk UkT ) Ak
K= = ,
(Ak )T 0 (Ak )T 0
 
B0 Ak
K0 = .
(Ak )T 0
Moreover, we assume that K0 is sparse and can be factorized cheaply using a sparse or
structured matrix decomposition. (Usually such a factorization requires O(nβ ) operations,
with the exponent β ∈ [1, 2].) Factorization of K can then be made by two applications of
the Sherman–Morison–Woodbury formula, yielding
K1−1 = K0−1 − K0−1 Ṽ [I + Ṽ T K0−1 Ṽ ]−1 Ṽ T K0−1 , (5.38)
K −1
= K1−1 − K1−1 Ũ [I + Ũ T
K1−1 Ũ ]−1 Ũ T K1−1 . (5.39)

By carefully structuring these calculations so that K0−1 , K1−1 , and K −1 are factorized and
incorporated into backsolves, these matrices are never explicitly created, and we can obtain
the solution to (5.33), i.e.,
   
dx −1 ∇L(x k , v k )
= −K (5.40)
dv h(x k )

in only O(nβ + q 3 + q 2 n) operations. Since only a few updates are stored (q is typically less
than 20), this limited memory approach leads to a much more efficient implementation of
quasi-Newton methods. An analogous approach can be applied to the SR1 method as well.
As seen in Chapter 3, quasi-Newton methods are slower to converge than Newton’s
method, and the best that can be expected is a superlinear convergence rate. For equality
constrained problems, the full-space quasi-Newton implementation has a local convergence
property that is similar to Theorem 3.1, but with additional restrictions.
Theorem 5.5 [63] Assume that f (x) and h(x) are twice differentiable, their second deriva-
tives are Lipschitz continuous, and Algorithm 5.1 modified by a quasi-Newton approxi-
mation converges to a KKT point that satisfies the LICQ and the sufficient second order
optimality conditions. Then x k converges to x ∗ at a superlinear rate, i.e.,
x k + dx − x ∗ 
lim =0 (5.41)
k→∞ x k − x ∗ 

i i

i i
 book_tem
i i
2010/7/27
page 105
i i

5.4. Quasi-Newton Methods 105

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

if and only if

Z(x k )T (B k − W (x ∗ , v ∗ ))dx 
lim = 0, (5.42)
k→∞ dx 

where Z(x) is a representation of the null space of A(x)T that is Lipschitz continuous in a
neighborhood of x ∗ .

Note that the theorem does not state whether a specific quasi-Newton update, e.g.,
BFGS or SR1, satisfies (5.42), as this depends on whether B k remains bounded and well-
conditioned, or whether updates need to be modified through damping or skipping. On the
other hand, under the more restrictive assumption where W (x ∗ , v ∗ ) is positive definite, the
following property can be shown.

Theorem 5.6 [63] Assume that f (x) and h(x) are twice differentiable, their second deriva-
tives are Lipschitz continuous, and Algorithm 5.1 modified by a BFGS update converges to
a KKT point that satisfies the LICQ and the sufficient second order optimality conditions.
If, in addition, W (x ∗ , v ∗ ) and B 0 are positive definite, and B 0 − W (x ∗ , v ∗ ) and x 0 − x ∗ 
are sufficiently small, then x k converges to x ∗ at a superlinear rate, i.e.,

x k + dx − x ∗ 
lim = 0. (5.43)
k→∞ x k − x ∗ 

5.4.2 A Quasi-Newton Reduced-Space Method

When n is large and n − m (the number of degrees of freedom) is small, it is attractive to
simplify the calculation of the Newton step. Using (5.16) and the first order approximation
to the Lagrange multipliers, (5.21), the resulting linear system becomes
    
0 0 (Y k )T Ak pY (Y k )T ∇f (x k )
 (Z k )T W k Y k (Z k )T W k Z k 0   pZ  = −  (Z k )T ∇f (x k )  .
k
(A ) YT k 0 0 v̄ h(x k )
(5.44)

This linear system is solved by solving two related subsystems of order m for pY and v̄,
respectively, and a third subsystem of order n − m for pZ . For this last subsystem, we can
approximate the reduced Hessian, i.e., B̄ k ≈ (Z k )T W k Z k , by using the BFGS update (5.31).
By noting that

d x = Y k pY + Z k pZ

and assuming that a full Newton step is taken (i.e., x k+1 = x k + dx ), we can apply the secant
condition based on the following first order approximation to the reduced Hessian:

(Z k )T W k Z k pZ ≈ (Z k )T ∇L(x k + Z k pZ , v k ) − (Z k )T ∇L(x k , v k ) (5.45)

= (Z k )T ∇L(x k+1 − Y k pY , v k ) − (Z k )T ∇L(x k , v k ) (5.46)
≈ (Z k )T ∇L(x k+1 , v k ) − (Z k )T ∇L(x k , v k ) − wk , (5.47)

i i

i i
 book_tem
i i
2010/7/27
page 106
i i

106 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

where w k = Z k W k Y k pY or its approximation. Defining

sk = pZ and yk = (Z k )T ∇L(x k+1 , v k ) − (Z k )T ∇L(x k , v k ) − wk , (5.48)

we can write the secant formula B̄ k+1 sk = yk and use this definition for the BFGS update
(5.31). However, note that for this update, some care is needed in the choice of Z(x k )
so that it remains continuous with respect to x. Discontinuous changes in Z(x) (due to
different variable partitions or QR rotations) could lead to serious problems in the Hessian
approximation [162].
In addition, we modify the Newton step from (5.44). Using the normal step pY cal-
culated from (5.17), we can write the tangential step as follows:

B̄ k pZ = −[(Z k )T ∇f (x k ) + (Z k )T W k Y k pY ] = −[(Z k )T ∇f (x k ) + wk ], (5.49)

where the second part follows from writing (or approximating) (Z k )T W k Y k pY as w k . Note
that the BFGS update eliminates the need to evaluate W k and to form (Z k )T W k Z k explicitly.
In addition, to avoid the formation of (Z k )T W k Y k pY , it is appealing to approximate the
vectors w k in (5.49) and (5.47). This can be done in a number of ways.
• Many reduced Hessian methods [294, 100, 134] typically approximate Z k W k Y k pY
by w k = 0. Neglecting this term often works well, especially when normal steps are
much smaller than tangential steps. This approximation is particularly useful when the
constraints are linear and pY remains zero once a feasible iterate is encountered. More-
over, when Z and Y have orthogonal representations, the length of the normal step
Y k pY is minimized. Consequently, neglecting this term in (5.49) and (5.47) still can
lead to good performance. On the other hand, coordinate representations (5.23) of Y
can lead to larger normal steps (see Figure 5.1), a nonnegligible value for Z k W k Y k pY ,
and possibly erratic convergence behavior of the method.
• To ensure that good search directions are generated in (5.44) that are independent of
the representations of Z and Y , we can compute a finite difference approximation of
(Z k )T W k Y k along pY , for example

w k+1 = (Z k )T ∇L(x k+1 , v k ) − (Z k )T ∇L(x k + Z k pZ , v k ). (5.50)

However, this term requires additional evaluation of reduced gradient (Z k )T ∇L(x k +

Z k pZ , v k ) and this may double the work per iteration.
• Another approximation, which requires no additional gradient information, uses a
quasi-Newton approximation S k for the rectangular matrix (Z k )T W k . This can be
obtained, for example, with the Broyden update (see (3.15) and the discussion in
Chapter 3) and can be used to define w k = S k Y k pY . This quasi-Newton approxima-
tion works well when either the normal steps or the tangential steps dominate the
iterations.
• To preserve the economy of computation, we can define a hybrid approach, where
wk is defined via the quasi-Newton approximation S k whenever conditions on pY
and pZ indicate that either step is dominant in iteration k. To develop these condi-
tions, we define a positive constant κ > 0, a sequence of positive numbers {γk } with

i i

i i
 book_tem
i i
2010/7/27
page 107
i i

5.4. Quasi-Newton Methods 107

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Figure 5.2. Regions that determine criteria for choosing wk , the approximation
to(Z k )T W k Y k p
Y . Quasi-Newton approximation can be in R1 or R3 , while R2 requires a
finite difference approximation.

∞
k=1 γk < ∞, and σk = (Z k )T ∇f (x k ) + h(x k ) which is related directly to the
distance to the solution, x k − x ∗ . In the space of pY  and pZ , we now define
three regions given by
R1 : 0 ≤ pY  ≤ γk2 pZ ,
1/2
R2 : γk2 pZ  < pY  ≤ κpZ /σk ,
1/2
R3 : pY  > κpZ /σk
and shown in Figure 5.2. In order to obtain a fast convergence rate, we only need
to resort to the finite difference approximation (5.50) when the calculated step lies
in region R2 . Otherwise, in regions R1 and R3 , the tangential and the normal steps,
respectively, are dominant and the less expensive quasi-Newton approximation can
be used for w k . In [54], these hybrid concepts were incorporated into an algorithm
with a line search along with safeguards to ensure that the updates S k and B̄ k remain
uniformly bounded. A detailed presentation of this hybrid algorithm along with anal-
ysis of global and local convergence properties and numerical performance is given
in [54].
To summarize, the reduced-space quasi-Newton algorithm does not require the com-
putation of the Hessian of the Lagrangian W k and only makes use of first derivatives of f (x)
and h(x). The reduced Hessian matrix (Z k )T W k Z k is approximated by a positive definite

i i

i i
 book_tem
i i
2010/7/27
page 108
i i

108 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

quasi-Newton matrix B̄ k , using the BFGS formula, and the Z k W k Y k pY term is approxi-
mated by a vector wk , which is computed either by means of a finite difference formula or
via a quasi-Newton approximation. The method is therefore well suited for large problems
with relatively few degrees of freedom.
The local convergence properties for this reduced-space method are analogous to The-
orems 5.5 and 5.6, but they are modified because B̄ k is now a quasi-Newton approximation
to the reduced Hessian and also because of the choice of approximation wk . First, we note
from (5.15), (5.47), and the definitions for B̄ k and w k that the condition

(B̄ k − (Z ∗ )T W (x ∗ , v ∗ )Z ∗ )pZ  + wk − (Z ∗ )T W (x ∗ , v ∗ )Y ∗ pY 

lim =0 (5.51)
k→∞ dx 

implies (5.42). Therefore, Theorem 5.7 holds and the algorithm converges superlinearly in x.
On the other hand, (5.51) does not hold for all choices of w k or quasi-Newton updates. For
instance, if wk is a poor approximation to (Z k )T W k Y k pY , the convergence rate is modified
as follows.

Theorem 5.7 [63] Assume that f (x) and h(x) are twice differentiable, their second deriva-
tives are Lipschitz continuous, and the reduced-space quasi-Newton algorithm converges to
a KKT point that satisfies the LICQ and the sufficient second order optimality conditions.
If Z(x) and v(x) (obtained from (5.21)) are Lipschitz continuous in a neighborhood of x ∗ ,
Y (x) and Z(x) are bounded, and [Y (x) | Z(x)] has a bounded inverse, then x k converges to
x ∗ at a 2-step superlinear rate, i.e.,

x k+2 − x ∗ 
lim =0 (5.52)
k→∞ x k − x ∗ 

if and only if

(B̄ k − (Z ∗ )T W (x ∗ , v ∗ )Z ∗ )pZ 
lim = 0. (5.53)
k→∞ dx 

Finally, we consider the hybrid approach that monitors and evaluates an accurate ap-
proximation of w k based on the regions in Figure 5.2. Moreover, since (Z ∗ )T W (x ∗ , v ∗ )Z ∗ is
positive definite, the BFGS update to the reduced Hessian provides a suitable approximation.
As a result, the following, stronger property can be proved.

Theorem 5.8 [54] Assume that f (x) and h(x) are twice differentiable, their second deriva-
tives are Lipschitz continuous, and we apply the hybrid reduced-space algorithm in [54]
with the BFGS update (5.31), (5.48). If

• the algorithm converges to a KKT point that satisfies the LICQ and the sufficient
second order optimality conditions,

• Z(x) and v(x) (obtained from (5.21)) are Lipschitz continuous,

• Y (x) and Z(x) are bounded and [Y (x) | Z(x)] has a bounded inverse in a neighborhood
of x ∗ ,

i i

i i
 book_tem
i i
2010/7/27
page 109
i i

5.5. Globalization for Constrained Optimization 109

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

then x k converges to x ∗ at a 1-step superlinear rate, i.e.,

x k + dx − x ∗ 
lim = 0. (5.54)
k→∞ x k − x ∗ 

5.5 Globalization for Constrained Optimization

The previous section discusses the generation of Newton steps that ensure fast convergence
to a local solution of problem (5.1). Of course, this requires an initial point that is sufficiently
close to the solution. To ensure convergence from poor starting points, the Newton method
must be modified to ensure that the iterations make sufficient progress to the solution. This
globalization issue was explored in Chapter 3 in the context of unconstrained minimization,
where both line search and trust region methods were developed and analyzed. In this chapter,
we extend these globalization methods to problem (5.1), but here we need to consider two
competing criteria in order to achieve convergence: the reduction of the objective function
and satisfaction of the equality constraints.
To balance the reduction in f (x) and h(x), we explore two popular approaches to
deal with this task. First, we consider a composite merit function, composed of both f (x)
and h(x), which can be used to determine progress toward the solution to (5.1). Con-
sequently, a step is allowed when it leads to a sufficient reduction in this merit function.
There are many ways to pose this function and our goal is to apply a merit function whose
minimum corresponds exactly to the solution of (5.1). The second, more recent, approach
uses a bicriterion strategy to trade off reductions in the objective function and constraint
infeasibility directly. This so-called filter approach stores information from previous iter-
ates (f (x l ), h(x l ), l = 1, . . . , k) with the noninferior or nondominated pairs removed; this
represents the filter. The filter approach then allows a move to be made to x k+1 only if
either f (x k+1 ) or h(x k+1 ) improves on the points in the current filter. The filter is then
updated and the iterations continue. Both approaches lead to robust and efficient extensions
to the local Newton-type methods and allow convergence from poor starting points under
mild conditions on (5.1).

5.5.1 Concepts of Merit Functions

Merit functions are motivated by a broad literature on methods for the minimization of
penalty functions. These methods have the appeal of solving constrained nonlinear prob-
lems using unconstrained methods, such as the ones developed in Chapter 3. On the
other hand, the direct application of unconstrained minimization to penalty functions is
not straightforward and this has implications on the choice of penalty functions as merit
functions.

Quadratic Penalty Functions

To explore the use of penalty functions, we first consider the classical quadratic penalty
function given by
Q(x; ρ) = f (x) + (ρ/2)h(x)T h(x). (5.55)
Minimization of Q(x; ρ) for a fixed value of ρ is typically performed with the unconstrained
methods of Chapter 3; if successful, this leads to the local minimizer, x(ρ). Moreover,

i i

i i
 book_tem
i i
2010/7/27
page 110
i i

110 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

solving a sequence of these unconstrained minimizations with limk→∞ ρ k → ∞ leads to a

sequence x(ρ k ) that is expected to converge to the solution of (5.1). The quadratic penalty
can be characterized by the following properties:
• If the limit point, x̄, of the sequence x(ρ k ) is feasible and ∇h(x̄) has linearly inde-
pendent columns, then x̄ is a KKT point of (5.1) with multipliers obtained from
v = lim ρ k h(x(ρ k )).
k→∞
• If the limit point, x̄, of the sequence x(ρ k ) is infeasible, then it is a stationary point
of h(x)T h(x).
• As ρ increases, care must be taken to deal with ill-conditioning of the Hessian,
∇xx Q(x; ρ) (or its approximation) when applying an unconstrained method.
• If ρ is not large enough, the quadratic penalty Q(x, ρ) may become unbounded even
if a solution exists for (5.1).
Because of these properties, it is clear that the quadratic penalty is not suitable as a merit
function, as the constrained solution is obtained only as ρ becomes unbounded. Moreover,
for a finite value of ρ, the quadratic penalty is not exact; i.e., the minimizer of Q(x; ρ) does
not correspond to the solution of (5.1). As a result, Q(x; ρ) can be reduced further by moving
away from the solution of (5.1). In addition, as discussed in [294], this nonexactness is not
limited to quadratic penalties but extends to all smooth penalty functions of the form


m
φ(x; ρ) = f (x) + ρ ϕ(hj (x)) where ϕ(ξ ) ≥ 0 for all ξ ∈ R, ϕ(0) = 0. (5.56)
j =1

Augmented Lagrangian Functions

The inexact penalty Q(x; ρ) can be modified to form the augmented Lagrange function
LA (x, v; ρ) = f (x) + h(x)T v + (ρ/2)h(x)T h(x). (5.57)
Because additional variables v of indeterminate sign are included in this function, this
function is more general than (5.56). As a result, an unconstrained minimization problem
can be formulated that leads to a solution of (5.1) for a finite value of ρ. Some properties of
the augmented Lagrange function can be summarized as follows:
• Given vectors x ∗ , v ∗ that correspond to a local solution of (5.1) that satisfies the LICQ
and sufficient second order conditions, for ρ greater than a threshold value, x ∗ is a
strict local minimizer for LA (x, v ∗ ; ρ).
• LA (x, v; ρ) has a local minimum x̄ that is close to x ∗ if v is sufficiently close to v ∗ ,
or ρ is sufficiently large.
• If the multiplier approximation is defined by the least squares estimate
v(x) = −[∇h(x)T ∇h(x)]−1 ∇h(x)T ∇f (x) (5.58)
in a manner similar to (5.21), then a local solution of (5.1) that satisfies the LICQ and
sufficient second order conditions is also a strict local minimizer of LA (x, v(x); ρ) for
ρ greater than a threshold value.

i i

i i
 book_tem
i i
2010/7/27
page 111
i i

5.5. Globalization for Constrained Optimization 111

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Because the augmented Lagrange function is exact for finite values of ρ and suitable values
of v, it forms the basis of a number of NLP methods (see [294, 100, 134]). On the other
hand, the suitability of LA as a merit function is complicated by the following:
• Finding a sufficiently large value of ρ is not straightforward. A suitably large value
of ρ needs to be determined iteratively,

• Suitable values for v need to be determined through iterative updates as the algorithm
proceeds, or by direct calculation using (5.58). Either approach can be expensive and
may lead to ill-conditioning, especially if ∇h(x) is rank deficient.
Nonsmooth Exact Penalty Functions
Because a smooth function of the form (5.56) cannot be exact, we finally turn to nonsmooth
penalty functions. The most common choice for merit functions is the class of p penalty
functions, given by

φp (x; ρ) = f (x) + ρ h(x)p , (5.59)


where, from Section 2.2, we recall the definition yp = ( i |y(i) |p )1/p with p ≥ 1.
For sufficiently large values of ρ, the minimizer of (5.59), if feasible, corresponds to
a local solution of (5.1). On the other hand, because these functions are nonsmooth, their
minimizers cannot be found using the unconstrained methods in Chapter 3. Nevertheless,
nonsmoothness does not preclude the use of φp (x; ρ) as a merit function. These exact penalty
functions are characterized by the following properties:
• Let x ∗ and v ∗ correspond to a local solution of (5.1) that satisfies the first order KKT
conditions and the LICQ. Then for ρ > ρ ∗ = v ∗ q and p1 + q1 = 1, x ∗ is a strict local
minimizer for φp (x, ρ) (see [185]). Note the use of a dual norm  · q to  · p (see
Section 2.2).

• Let x̄ be a stationary point of φp (x; ρ) for all values of ρ above a positive threshold;
then if h(x̄) = 0, x̄ is a KKT point for (5.1) (see [294]).

• Let x̄ be a stationary point of φp (x; ρ) for all values of ρ above a positive threshold,
then if h(x̄)  = 0, x̄ is an infeasible stationary point for the penalty function (see [294]).
Compared to (5.55) and (5.57) the nonsmooth exact penalty is easy to apply as a merit
function. Moreover, under the threshold and feasibility conditions, the above properties
show an equivalence between local minimizers of φp (x, ρ) and local solutions of (5.1). As
a result, further improvement of φp (x; ρ) cannot be made from a local optimum of (5.1).
Moreover, even if a feasible solution is not found, the minimizer of φp (x; ρ) leads to a
“graceful failure” at a local minimum of the infeasibility. This point may be useful to the
practitioner to flag incorrect constraint specifications in (5.1) or to reinitialize the algorithm.
As a result, φp (x; ρ), with p = 1 or 2, is widely used in globalization strategies discussed
in the remainder of the chapter and implemented in the algorithms in Chapter 6.
The main concern with this penalty function is to find a reasonable value for ρ. While
ρ has a well-defined threshold value, it is unknown a priori, and the resulting algorithm
can suffer from poor performance and ill-conditioning if ρ is set too high. Careful attention
is needed in updating ρ, which is generally chosen after a step is computed, in order to

i i

i i
 book_tem
i i
2010/7/27
page 112
i i

112 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Figure 5.3. Illustration of filter concepts.

promote acceptance. For instance, the value of ρ can be adjusted by monitoring the multiplier
estimates v k and ensuring that ρ > v k q as the algorithm proceeds. Recently, more efficient
updates for ρ have been developed [78, 79, 294] that are based on allowing ρ to lie within a
bounded interval and using the smallest value of ρ in that interval to determine an acceptable
step. As described below, this approach provides considerable flexibility in choosing larger
steps. Concepts of exact penalty functions will be applied to both the line search and trust
region strategies developed in the next section.

5.5.2 Filter Method Concepts

The concept of the filter method was developed by Fletcher and Leyffer [137] as an alter-
native to merit functions, in order to avoid the determination of an appropriate value of the
penalty parameter ρ. Instead, this approach simply requires improvement of either f (x) or
h(x). The underlying concept of the filter is to interpret the nonlinear program (5.1) as
a biobjective optimization problem with two goals: minimizing the constraint infeasibility
θ (x) = h(x) and minimizing the objective function f (x).3 Also, more emphasis must
be placed on the first measure, since a point has to be feasible in order to be an optimal
solution of the nonlinear program. The filter method is illustrated in Figure 5.3. The fil-
ter is defined by a set of dominant pairs (f (x l ), θ(x l )) drawn from previous iterates. As
seen in Figure 5.3, a pair (f (x l ), θ(x l )) is considered dominant if there is no other pair
(f (x j ), θ (x j )), j ∈ {1, . . . , k}, that satisfies f (x j ) ≤ f (x l ) and θ (x j ) ≤ θ (x l ). Thus, at it-
eration k, we would like to improve either f (x) or θ(x) relative to a filter, defined as a
two-dimensional forbidden region Fk above and to the right of the solid lines in Figure 5.3,
defined from the filter pairs.4
3A scaled norm for h(x) can be used as well.
4 In [137, 135] Fk refers to the set of filter pairs, rather than a forbidden region.

i i

i i
 book_tem
i i
2010/7/27
page 113
i i

5.5. Globalization for Constrained Optimization 113

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

A trial point x̂ (e.g., generated by a Newton-like step coupled to the line search or
trust region approach) is acceptable to the filter if it is not in the forbidden region, Fk ,
and a sufficient improvement is realized corresponding to a small fraction of the current
infeasibility. In other words, a trial point is accepted if

θ (x̂) ≤ (1 − γθ )θ (x k ) or f (x̂) ≤ f (x k ) − γf θ(x k ) and (f (x̂), θ (x̂)) ∈

/ Fk (5.60)

for some small positive values γf , γθ . These inequalities correspond to the trial point
(f (x̂), θ (x̂)) that lies below the dotted lines in Figure 5.3.
The filter strategy also requires two further enhancements in order to guarantee con-
vergence.
1. Relying solely on criterion (5.60) allows the acceptance of a sequence {x k } that only
provides sufficient reduction of the constraint violation, but not the objective function.
For instance, this can occur if a filter pair is placed at a feasible point with θ(x l ) = 0,
and acceptance through (5.60) may lead to convergence to a feasible, but nonoptimal,
point. In order to prevent this, we monitor a switching criterion to test whenever the
constraint infeasibility is too small to promote a sufficient decrease in the objective
function. If so, we require that the trial point satisfy a sufficient decrease of the
objective function alone.
2. Ill-conditioning at infeasible points can lead to new steps that may be too small to
satisfy the filter criteria; i.e., trial points could be “caught” between the dotted and
solid lines in Figure 5.3. This problem can be corrected with a feasibility restoration
phase which attempts to find a less infeasible point at which to restart the algorithm.
With these elements, the filter strategy, coupled either with a line search or trust region
approach, can be shown to converge to either a local solution of (5.1) or a stationary point
of the infeasibility, θ (x). To summarize, the filter strategy is outlined as follows.
• Generate a new trial point, x̂. Continue if the trial point is not within the forbidden
region, Fk .
• If a switching condition (see (5.85)) is triggered, then accept x̂ as x k+1 if a sufficient
reduction is found in the objective function.
• Else, accept x̂ as x k+1 if the trial point is acceptable to filter with a margin given by
θ (x k ). Update the filter.
• If the trial point is not acceptable, find another trial point that is closer to x k .
• If there is insufficient progress in finding trial points, evoke the restoration phase to
find a new point, x R , with a smaller θ(x R ).

5.5.3 Filter versus Merit Function Strategies

Unlike using the p merit function with a fixed penalty parameter, the filter method requires
no fixed trade-off function between the objective function and infeasibility. As a result it
allows considerable freedom in choosing the next step. A similar advantage can be employed
for merit functions as well through the use of flexible penalty parameters, as detailed in

i i

i i
 book_tem
i i
2010/7/27
page 114
i i

114 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Figure 5.4. Illustration of merit function concepts with a flexible penalty parameter.

Curtis and Nocedal [105], and these can be interpreted and compared in the context of filter
methods.
Figure 5.4 shows how merit functions can mimic the performance of filter methods
in the space of f (x) and h(x) if we allow the penalty parameter to vary between upper
and lower bounds. To see this, consider the necessary condition for acceptance of x k+1 ,
f (x k+1 ) + ρh(x k+1 ) < f (x k ) + ρh(x k ), (5.61)
f (x k ) − f (x k+1 )
or if h(x k )  = h(x k+1 ), ρ > ρv = .
h(x k+1 ) − h(x k )
Note that the bound on ρ defines the slopes of the lines in Figure 5.4. Clearly, R1 does not
satisfy (5.61), and x k+1 is not acceptable in this region. Instead, acceptable regions for x k+1
are R2 and R3 for ρ = ρu , while R3 and R4 are acceptable regions for ρ = ρl . If we now
allow ρ ∈ [ρl , ρu ], then all three regions R2 , R3 , and R4 are acceptable regions for x k+1 and
these regions behave like one element of the filter method. This is especially true if ρu can
be kept arbitrarily large and ρl can be kept close to zero. Of course, the limits on ρ cannot
be set arbitrarily; they are dictated by global convergence properties. As will be seen later,
it is necessary to update ρu based on a predicted descent property. This property ensures
decrease of the exact penalty function as the step size or trust region becomes small. On the
other hand, due to nonlinearity, smaller values of ρ may be allowed for satisfaction of (5.61)
for larger steps. This relation will guide the choice of ρl . More detail on this selection and
the resulting convergence properties is given in the next sections. We now consider how
both merit function and filter concepts are incorporated within globalization strategies.

5.6 Line Search Methods

Line search strategies were considered in Chapter 3 for unconstrained optimization. In
the development of these methods, we were concerned with providing sufficient descent
properties as well as step sizes that remain bounded away from zero. These concerns will

i i

i i
 book_tem
i i
2010/7/27
page 115
i i

5.6. Line Search Methods 115

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

also be addressed in this section in the context of constrained optimization, for both merit
function and filter approaches.

5.6.1 Line Search with Merit Functions

Motivated by the line search algorithm in Chapter 3, this method has the possibility of a
straightforward extension, simply through a direct substitution of the merit function for the
objective function. As in Section 3.4, we revisit a line search method based on the Armijo
criterion (3.31) with backtracking to provide a sufficient decrease in the merit function. How-
ever, some care is needed as the exact penalty merit function, φ(x; ρ) = f (x) + ρh(x)p ,
is nonsmooth. Typically the 1 penalty (i.e., p = 1) is used with line search methods.
Translated into this merit function, the Armijo criterion requires satisfaction of

φp (x k + α k dx ; ρ) ≤ φp (x k ; ρ) + ηα k Ddx φp (x k ; ρ), (5.62)

where x k+1 = x k + α k dx and Ddx φp (x k ; ρ) is the directional derivative of the merit function
along dx . To evaluate the directional derivative we first consider its definition:

φp (x k + αdx ; ρ) − φp (x k ; ρ)
Ddx φp (x k ; ρ) = lim . (5.63)
α→0 α
For α ≥ 0 we can apply Taylor’s theorem and bound the difference in merit functions by

φp (x k + αdx ; ρ) − φp (x k ; ρ) = (f (x k + αdx ) − f (x k ))
+ρ(h(x k + αdx )p − h(x k )p )
≤ α∇f (x k )T dx + ρ(h(x k ) + α∇h(x k )T dx p − h(x k )p )
+ b1 α 2 dx 2 for some b1 > 0. (5.64)

Because the Newton step satisfies ∇h(x k )T dx = −h(x k ), we have

h(x k ) + α∇h(x k )T dx p = (1 − α)h(x k )p ,

thus allowing (5.64) to be simplified to

φp (x k + αdx ; ρ) − φp (x k ; ρ) ≤ α∇f (x k )T dx − ραh(x k )p + b1 α 2 dx 2 . (5.65)

Using the quadratic term from Taylor’s theorem, one can also derive a corresponding lower
bound, leading to

α∇f (x k )T dx − ραh(x k )p − b1 α 2 dx 2 ≤ φp (x k + αdx ; ρ) − φp (x k ; ρ). (5.66)

Dividing (5.65) and (5.66) by α and applying the definition (5.63) as α goes to zero leads
to the following expression for the directional derivative:

Ddx φp (x k ; ρ) = ∇f (x k )T dx − ρh(x k )p . (5.67)

The next step is to tie the directional derivative to the line search strategy to enforce
the optimality conditions. In particular, the choices that determine dx and ρ play important

i i

i i
 book_tem
i i
2010/7/27
page 116
i i

116 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

roles. In Sections 5.3 and 5.4 we considered full- and reduced-space representations as well
as the use of quasi-Newton approximations to generate dx . These features also extend to the
implementation of the line search strategy, as discussed next.
We start by noting that the Newton steps dx from (5.5) and (5.16) are equivalent.
Substituting (5.15) into the directional derivative (5.67) leads to
Ddx φp (x k ; ρ) = ∇f (x k )T (Z k pZ + Y k pY ) − ρh(x k )p , (5.68)
and from (5.17) we have
Ddx φp (x k ; ρ) = ∇f (x k )T Z k pZ − ∇f (x k )T Y k [(Ak )T Y k ]−1 h(x k ) − ρh(x k )p . (5.69)

We now choose w k = (Z k )T W k Y k pY (or an approximation) and we define H̄ as either

(Z k )T (W k + δW I )Z k or the quasi-Newton approximation B̄ k . We assume that these matrix
choices are bounded, positive definite and that they satisfy
s T H̄ −1 s ≥ c1 s2 for all s ∈ Rn−m .
Using (5.19) and (5.69) leads to
Ddx φp (x k ; ρ) = −∇f (x k )T Z k H̄ −1 (Z k f (x k ) + wk )
−∇f (x k )T Y k [(Ak )T Y k ]−1 h(x k ) − ρh(x k )p
≤ −c1 (Z k )T ∇f (x k )2 − ∇f (x k )T Z k H̄ −1 w k
−∇f (x k )T Y k [(Ak )T Y k ]−1 h(x k ) − ρh(x k )p . (5.70)

Finally, using the penalty parameter ρ k given by

 !
∇f (x k )T (Y k pY − Z k H̄ −1 w k )
ρ k = max ρ k−1 , +  (5.71)
h(x k )p
for some  > 0 leads to
Ddx φp (x k ; ρ k ) ≤ −c2 ((Z k )T ∇f (x k )2 + h(x k )p ), (5.72)
where c2 = min(c1 , ).
We now consider the following cases:
• In (5.70), the effect of w k can be modified by multiplying it by a damping factor, ζ ,
so that
∇f (x k )T Z k H̄ −1 (Z k f (x k ) + ζ wk ) ≥ c̃1 (Z k )T ∇f (x k )2
with c1 ≥ c̃1 > 0. Using
 !
∇f (x k )T Y k pY
ρ = max ρ
k k−1
, + (5.73)
h(x k )p

allows ρ k to satisfy (5.72).

• If we set wk = 0 and the multipliers v̄ are estimated from (5.21), we have
−∇f (x k )T Y k pY = v̄ T h(x k ) ≤ v̄q h(x k )p , where 1/p + 1/q = 1,

i i

i i
 book_tem
i i
2010/7/27
page 117
i i

5.6. Line Search Methods 117

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

and this leads to another bound on ρ k that satisfies (5.72):

ρ k = max{ρ k−1 , v̄q + }. (5.74)

• For the case where Ĥ = W k is positive definite for all k or Ĥ = B k , the full-space
BFGS update, we can also choose (5.74) as the update for ρ. From (5.67) and (5.5)
we have

Ddx φp (x k ; ρ) ≤ ∇f (x k )T dx − ρh(x k )p

≤ −dxT Ĥ dx − dxT Ak v̄ − ρh(x k )p
≤ −dxT Ĥ dx + h(x k )p v̄q − ρh(x k )p
≤ −dxT Ĥ dx − h(x k )p . (5.75)

Moreover, from (5.15), nonsingularity of [Y k | Z k ], and linear independence of the

columns of Ak (leading to nonsingularity of the KKT matrix in (5.4)), we have for
some c3 , c4 > 0,

dxT Ĥ dx ≥ c3 (pZ 2 + pY 2 ) ≥ c4 (h(x k )2 + Z(x k )T ∇f (x k )2 ). (5.76)

Substituting this expression into (5.75) and choosing the constants appropriately leads
directly to (5.72).
• Finally, for flexible choices of ρ ∈ [ρl , ρu ] as shown in Figure 5.4, one can choose ρu
to satisfy (5.72), based on (5.71), (5.73), or (5.74). This approach finds a step size α k
that satisfies the modified Armijo criterion:

φp (x k + α k dx ; ρ) ≤ φp (x k ; ρ) + ηα k Ddx φp (x k ; ρm
k
) (5.77)

for some ρ ∈ [ρlk , ρuk ]. For this test, ρm

k should be chosen large enough to satisfy (5.72).

On the other hand, ρlk is chosen as a slowly increasing lower bound given by

ρlk+1 = min{ρuk , ρlk + max{, 0.1(ρv − ρlk )}}, (5.78)

where ρv is as determined in (5.61).

Using any of the above update cases for ρ, we now state the following line search algorithm.

Algorithm 5.3.
Choose constants η ∈ (0, 1/2), 1 , 2 > 0, and τ , τ with 0 < τ < τ < 1. Set k := 0 and
choose starting points x 0 , v 0 , and an initial value ρ0 > 0 (or ρl0 , ρu0 > 0) for the penalty
parameter.

For k ≥ 0, while dx  > 1 and max(∇L(x k , v k , h(x k )) > 2 :

1. Evaluate f (x k ), ∇f (x k ), h(x k ), Ak , and the appropriate Hessian terms (or approxi-

mations).
2. Calculate the search step dx and v̄ using either the full-space or reduced-space systems,
with exact or quasi-Newton Hessian information.

i i

i i
 book_tem
i i
2010/7/27
page 118
i i

118 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

3. Update ρ k to satisfy (5.71), (5.73), or (5.74), or update ρlk and ρuk for the flexible
choice of penalty parameters.

4. Set α k = 1.

5. Test the line search condition (5.62) or (5.77).

6. If the line search condition is not satisfied, choose a new α k ∈ [τ α k , τ α k ] and go to

step 5. Otherwise, set

x k+1 = x k + α k dx , v k+1 = v̄. (5.79)

Because of the need to update ρ k at each iteration, we consider only “semiglobal”

properties, where stronger assumptions than in Chapter 3 are required on the sequence of
iterates.

Assumptions I: The sequence {x k } generated by Algorithm 5.3 is contained in a bounded,

convex set D with the following properties:

1. The functions f : Rn → R and h : Rn → Rm are twice differentiable and their first

and second derivatives (or approximations) are uniformly bounded in norm over D.

2. The matrix A(x) has full column rank for all x ∈ D, and there exist constants γ0 and
β0 such that

Y (x)[A(x)T Y (x)]−1  ≤ γ0 , Z(x) ≤ β0 (5.80)

for all x ∈ D.

3. The Hessian (or approximation) W (x, v) is positive definite on the null space of the
Jacobian A(x)T and uniformly bounded.

Lemma 5.9 [54] If Assumptions I hold and if ρ k = ρ is constant for all sufficiently large k,
then there is a positive constant γρ such that for all large k,
 
φp (x k ) − φp (x k+1 ) ≥ γρ (Z k )T ∇f (x k )2 + h(x k )p . (5.81)

The proof of Lemma 5.9 is a modification from the proof of Lemma 3.2 in [82] and
Lemma 4.1 in [54]. Key points in the proof (see Exercise 4) include showing that α k is
always bounded away from zero, using (5.64). Applying (5.72) and the fact that h(x) has
an upper bound in D leads to the desired result.
Finally, Assumptions I and Lemma 5.9 now allow us to prove convergence to a KKT
point for a class of line search strategies that use nonsmooth exact penalty merit functions
applied to Newton-like steps.

Theorem 5.10 If Assumptions I hold, then the weights {ρ k } are constant for all sufficiently
large k, limk→∞ ((Z k )T ∇f (x k )+h(x k )) = 0, and limk→∞ x k = x ∗ , a point that satisfies
the KKT conditions (5.2).

i i

i i
 book_tem
i i
2010/7/27
page 119
i i

5.6. Line Search Methods 119

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Proof: By Assumptions I, we note that the quantities that define ρ in (5.71), (5.73), or (5.74)
are bounded. Therefore, since the procedure increases ρ k by at least  whenever it changes
the penalty parameter, it follows that there is an index k0 and a value ρ such that for all
k > k0 , ρ k = ρ always satisfies (5.71), (5.73), or (5.74).
This argument can be extended to the flexible updates of ρ. Following the proof in
[78], we note that ρuk is increased in the same way ρ k is increased for the other methods,
so ρuk eventually approaches a constant value. For ρlk we note that this quantity is bounded
above by ρuk and through (5.78) it increases by a finite amount to this value at each iteration.
As a result, it also attains a constant value.
Now that ρ k is constant for k > k0 , we have by Lemma 5.9 and the fact that φp (x k )
decreases at each iterate, that


k
φp (x k0 ; ρ) − φp (x k+1 ; ρ) = (φp (x j ; ρ) − φp (x j +1 ; ρ))
j =k0

k
≥ γρ [(Z j )T ∇f (x j )2 + h(x j )p ].
j =k0

By Assumptions I, φp (x, ρ) is bounded below for all x ∈ D, so the last sum is finite, and
thus
lim [(Z k )T ∇f (x k )2 + h(x k )p ] = 0. (5.82)
k→∞

As the limit point, x ∗ , satisfies (Z ∗ )T ∇f (x ∗ ) = 0 and h(x ∗ ) = 0, it is easy to see that it

satisfies (5.2) as well.

5.6.2 Line Search Filter Method

For the filter line search method, we again consider a backtracking procedure at iteration
k where a decreasing sequence of step sizes α k,l ∈ (0, 1] (l = 0, 1, 2, . . .) is tried until some
acceptance criterion is satisfied for the corresponding trial point
x(α k,l ) := x k + α k,l dx . (5.83)
Here we require a sufficient reduction in one of the measures θ = h or f , with a margin
related to the infeasibility θ (x k ). More precisely, for small positive constants γθ , γf ∈ (0, 1),
we say that a trial step size α k,l provides sufficient reduction with respect to the current
iterate x k if
θ(x(α k,l )) ≤ (1 − γθ )θ (x k ) or f (x(α k,l )) ≤ f (x k ) − γf θ(x k ). (5.84)
These criteria provide sufficient reduction of the infeasibility, but additional criteria are
needed to provide sufficient reduction of the objective function, particularly for near feasible
trial points. Here we enforce an f -type switching condition
 sθ
mk (α k,l ) < 0 and [−mk (α k,l )]sf [α k,l ]1−sf > δ θ(x k ) (5.85)

with constants δ > 0, sθ > 1, sf ≥ 1, where mk (α) := α∇f (x k )T dx . If (5.85) holds, then
step dx is a descent direction for the objective function and we require that α k,l satisfies the

i i

i i
 book_tem
i i
2010/7/27
page 120
i i

120 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Armijo-type condition

f (x(α k,l )) ≤ f (x k ) + ηf mk (α k,l ), (5.86)

where ηf ∈ (0, 12 ) is a fixed constant. Note that several trial step sizes may be tried with (5.85)
satisfied, but not (5.86). Moreover, for smaller step sizes the f -type switching condition
(5.85) may no longer be valid and we revert to the acceptance criterion (5.84).
Note that the second part of (5.85) ensures that the progress for the objective function
enforced by the Armijo condition (5.86) is sufficiently large compared to the current con-
straint violation. Enforcing (5.85) is essential for points that are near the feasible region.
Also, the choices of sf and sθ allow some flexibility in performance of the algorithm. In
particular, if sf > 2sθ (see [403]), both (5.85) and (5.86) will hold for a full step, possibly
improved by a second order correction (see Section 5.8) [402], and rapid local convergence
is achieved.
During the optimization we make sure that the current iterate x k is always acceptable
to the current filter Fk . At the beginning of the optimization, the forbidden filter region is
normally initialized to F0 := {(θ , f ) ∈ R2 : θ ≥ θmax } for some upper bound on infeasibility,
θmax > θ(x 0 ). Throughout the optimization, the filter is then augmented in some iterations
after the new iterate x k+1 has been accepted. For this, the updating formula
" #
Fk+1 := Fk ∪ (θ , f ) ∈ R2 : θ ≥ (1 − γθ )θ (x k ) and f ≥ f (x k ) − γf θ (x k ) (5.87)

is used. On the other hand, the filter is augmented only if the current iteration is not an f -type
iteration, i.e., if for the accepted trial step size α k , the f -type switching condition (5.85)
does not hold. Otherwise, the filter region is not augmented, i.e., Fk+1 := Fk . Instead, the
Armijo condition (5.86) must be satisfied, and the value of the objective function is strictly
decreased. This is sufficient to prevent cycling.
Finally, finding a trial step size α k,l > 0 that provides sufficient reduction as defined by
criterion (5.84) is not guaranteed. In this situation, the filter method switches to a feasibility
restoration phase, whose purpose is to find a new iterate x k+1 that satisfies (5.84) and is also
acceptable to the current filter by trying to decrease the constraint violation. Any iterative
algorithm can be applied to find a less infeasible point, and different methods could even
be used at different stages of the optimization procedure. To detect the situation where no
admissible step size can be found, one can linearize (5.84) (or (5.85) in the case of an f -type
iteration) and estimate the smallest step size, α k,min , that allows an acceptable trial point.
The algorithm then switches to the feasibility restoration phase when α k,l becomes smaller
than α k,min .
If the feasibility restoration phase terminates successfully by delivering an admissible
point, the filter is augmented according to (5.87) to avoid cycling back to the problematic
point x k . On the other hand, if the restoration phase is unsuccessful, it should converge
to a stationary point of the infeasibility, θ (x). This “graceful exit” may be useful to the
practitioner to flag incorrect constraint specifications in (5.1) or to reinitialize the algorithm.
Combining these elements leads to the following filter line search algorithm.

Algorithm 5.4.
Given a starting point x 0 ; constants θmax ∈ (θ(x 0 ), ∞]; γθ , γf ∈ (0, 1); δ > 0; γα ∈ (0, 1];
sθ > 1; sf ≥ 1; ηf ∈ (0, 21 ); 0 < τ ≤ τ < 1.

i i

i i
 book_tem
i i
2010/7/27
page 121
i i

5.6. Line Search Methods 121

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

1. Initialize the filter F0 := {(θ , f ) ∈ R2 : θ ≥ θmax } and the iteration counter k ← 0.

2. Check convergence. Stop if x k is a KKT point of the nonlinear program (5.1).

3. Compute search direction. Calculate the search step dx using either the full-space or
reduced-space linear systems, with exact or quasi-Newton Hessian information. If
this system is detected to be too ill-conditioned, go to feasibility restoration phase in
step 8.

4. Backtracking line search.

(a) Initialize line search. Set α k,0 = 1 and l ← 0.

(b) Compute new trial point. If the trial step size becomes too small, i.e., α k,l <
α k,min , go to the feasibility restoration phase in step 8. Otherwise, compute the
new trial point x(α k,l ) = x k + α k,l dx .
(c) Check acceptability to the filter. If (f (x(α k,l )), θ(x(α k,l ))) ∈ Fk (i.e., in the for-
bidden region), then x(α k,l ) is not acceptable to the filter. Reject the trial step
size and go to step 4(e).
(d) Check sufficient decrease with respect to current iterate.
• Case I: α k,l is an f -step size (i.e., (5.85) holds): If the Armijo condition
(5.86) for the objective function holds, accept the trial step and go to step 5.
Otherwise, go to step 4(e).
• Case II: α k,l is not an f -step size (i.e., (5.85) is not satisfied): If (5.84)
holds, accept the trial step and go to step 5. Otherwise, go to step 4(e).
(e) Choose new trial step size. Choose α k,l+1 ∈ [τ α k,l , τ α k,l ], set l ← l + 1, and go
back to step 4(b).

5. Accept trial point. Set α k := α k,l and x k+1 := x(α k ).

6. Augment filter if necessary. If k is not an f -type iteration, augment the filter using
(5.87); otherwise leave the filter unchanged, i.e., set Fk+1 := Fk .

7. Continue with next iteration. Increase the iteration counter k ← k + 1 and go back to
step 2.

8. Feasibility restoration phase. Compute a new iterate x k+1 by decreasing the infeasi-
bility measure θ, so that x k+1 satisfies the sufficient decrease conditions (5.84) and is
acceptable to the filter, i.e., (θ (x k+1 ), f (x k+1 ))  ∈ Fk . Augment the filter using (5.87)
(for x k ) and continue with the regular iteration in step 7.

The filter line search method is clearly more complicated than the merit function
approach and is also more difficult to analyze. Nevertheless, the key convergence result for
this method can be summarized as follows.

Theorem 5.11 [403] Let Assumption I(1) hold for all iterates and Assumptions I(1,2,3) hold
for those iterates that are within a neighborhood of the feasible region (h(x k ) ≤ θinc for
some θinc > 0). In this region successful steps can be taken and the restoration phase need

i i

i i
 book_tem
i i
2010/7/27
page 122
i i

122 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

not be invoked. Then Algorithm 5.4, with no unsuccessful terminations of the feasibility
restoration phase, has the following properties:

lim h(x k ) = 0 and lim inf ∇L(x k , v k ) = 0. (5.88)

k→∞ k→∞

In other words, all limit points are feasible, and if {x k } is bounded, then there exists a limit
point x ∗ of {x k } which is a first order KKT point for the equality constrained NLP (5.1).

The filter line search method avoids the need to monitor and update penalty param-
eters over the course of the iterations. This also avoids the case where a poor update of
the penalty parameter may lead to small step sizes and poor performance of the algorithm.
On the other hand, the “lim inf ” result in Theorem 5.11 is not as strong as Theorem 5.7, as
only convergence of a subsequence can be shown. The reason for this weaker result is that,
even for a good trial point, there may be filter information from previous iterates that could
invoke the restoration phase. Nevertheless, one should be able to strengthen this property
to limk→∞ ∇x L(x k , v k ) = 0 through careful design of the restoration phase algorithm
(see [404]).

5.7 Trust Region Methods

In this section we describe trust region globalization strategies with both merit functions and
filter methods. As observed for unconstrained problems in Chapter 3, trust region methods,
in contrast to line search methods, allow the search direction to be calculated as a function
of the step length. This added flexibility leads to methods that have convergence properties
superior to line search methods. A key advantage is also a straightforward treatment of
indefinite Hessians and negative curvature, so that we can converge to limit points that are
truly local minima, not just stationary points. On the other hand, the computational expense
for each trust region iteration may be greater than for line search iterations. This remains
true for the equality constrained case.
Trust region methods cannot be applied to equality constrained problems in a straight-
forward way. In Section 5.3.1 Newton steps were developed and were related to the quadratic
programming problem (5.6). A simple way to limit the step length would be to extend
the QP to

1
min ∇f (x k )T dx + dxT W k dx (5.89)
dx 2
s.t. h(x k ) + (Ak )T dx = 0,
dx  ≤ .

Here dx  could be expressed as the max-norm leading to simple bound constraints. How-
ever, if the trust region is small, there may be no feasible solution for (5.89). As discussed
below, this problem can be overcome by using either a merit function or filter approach.
With the merit function approach, the search direction dx is split into two components with
separate trust region problems for each, and a composite-step trust region method is devel-
oped. In contrast, the filter method applies its restoration phase if the trust region  is too
small to yield a feasible solution to (5.89).

i i

i i
 book_tem
i i
2010/7/27
page 123
i i

5.7. Trust Region Methods 123

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

5.7.1 Trust Regions with Merit Functions

In the Byrd–Omojukun algorithm, the trust region problem is considered through the calcu-
lation of two composite steps. Because the quadratic program (5.89) may have no solution
with adjustment of the trust region, a decomposition is proposed that serves to improve the
p penalty function:

φp (x; ρ) = f (x) + ρ||h(x)||p .

This approach focuses on improving each term separately, and with p = 2 quadratic models
of each term can be constructed and solved as separate trust region problems. In particular,
a trust region problem that improves the quadratic model of the objective function can be
shown to give a tangential step, while the trust region problem that reduces a quadratic
model of the infeasibility leads to a normal step. Each trust region problem is then solved
using the algorithms developed in Chapter 3.
Using the 2 penalty function, we now postulate a composite-step trust region model
at iteration k given by
1
m2 (x; ρ) = ∇f (x k )T dx + dxT W k dx + ρh(x k ) + (Ak )T dx 2 .
2
Decomposing dx = Y k pY + Z k pZ into normal and tangential components leads to the fol-
lowing subproblems.

Normal Trust Region Subproblem

min h(x k ) + (Ak )T Y k pY 22

pY

s.t. Y k pY 2 ≤ ξN . (5.90)

Typically, ξN = 0.8. This subproblem is an NLP (with pY ∈ Rm ) with a quadratic objective

and a single quadratic constraint. Here, we assume Ak to be full column rank so that the
problem is strictly convex and can be solved with any of the convex methods in Section 3.5.1
in Chapter 3. Moreover, if m, the number of equations, is large, an inexact solution can be
obtained efficiently using Powell’s dogleg method.

Tangential Trust Region Subproblem

1
min ((Z k )T ∇f (x k + wk ))T pZ + pZT H̄ k pZ (5.91)
pZ 2
s.t. ||Y k pY + Z k pZ || ≤ ,

where H̄ k is the reduced Hessian (Z k )T W k Z k , or its approximation B̄ k , and where w k =

(Z k )T W k Y k pY , or its approximation. This subproblem is an NLP (with pZ ∈ R(n−m) ) with
a quadratic objective and a single quadratic constraint. If H̄ k and wk are exact, then (5.91)
may be a nonconvex model and can be solved with any of the nonconvex methods in
Section 3.5.2 in Chapter 3. Moreover, one can obtain an exact, global solution to (5.91)
using an eigenvalue decomposition and the methods in [282, 100]. Otherwise, if n − m is
large, a truncated Newton algorithm is generally applied to solve (5.91) inexactly.

i i

i i
 book_tem
i i
2010/7/27
page 124
i i

124 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Figure 5.5. Trust regions for composite steps.

If acceptable steps have been found that provide sufficient reduction in the merit
function, we select our next iterate as

x k+1 = x k + Y k pY + Z k pZ . (5.92)

If not, we reject this step, reduce the size of the trust region, and repeat the calculation of
the normal and tangential steps. These steps are shown in Figure 5.5. In addition, we obtain
Lagrange multiplier estimates at iteration k using the first order approximation:

v k = −((Y k )T Ak )−1 (Y k )T ∇f (x k ). (5.93)

The criteria for accepting new points and adjusting the trust region are based on
predicted and actual reductions of the merit function. The predicted reduction in the quadratic
models contains the predicted reduction in the model for the tangential step (qk ),
1
qk = −((Z k )T ∇f (x k ) + wk )T pZ − pZT B̄ k pZ , (5.94)
2
and the predicted reduction in the model for the normal step (ϑk ),

ϑk = ||h(x k )|| − ||h(x k ) + Ak Y k pY ||. (5.95)

The overall predicted reduction is now

1
pred = qk + ρ k ϑk − ∇f (x k )T Y k pY − pYT (Y k )T W k Y k pY , (5.96)
2
where ρ k is the updated penalty parameter calculated from (5.98). When we update the
reduced Hessian with quasi-Newton approximations (and do not know the exact Hessian,
W k ), we could omit the term 21 pYT (Y k )T W k Y k pY and obtain a larger value for ρ k . Otherwise,
we use the complete expression (5.96).

i i

i i
 book_tem
i i
2010/7/27
page 125
i i

5.7. Trust Region Methods 125

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

The actual reduction is just the difference in the merit function at two successive
iterates:

ared = f (x k )−f (x k +Y k pY +Z k pZ )+ρ k (||h(x k )||−||h(x k +Y k pY +Z k pZ )||). (5.97)

ared
The quality of the step is assessed through the ratio pred and, as in Chapter 3, the trust
region  is adjusted based on how well ared and pred match.
Finally, we choose an update strategy for the penalty parameter ρ that ensures that
pred is positive, and ρ is positive and monotonically increasing as we approach the solution.
With this in mind, we employ the following updating rule developed in [79]:

−(q k − ∇f (x k )T Y k pY − 12 pYT (Y k )T W k Y k pY )
ρ k = max ρ k−1 , , (5.98)
(1 − ζ )ϑk

where ζ ∈ (0, 1/2). Again, if we use quasi-Newton updates, we would drop the term
1 T k T k k k
2 pY (Y ) W Y pY in (5.98). A more flexible alternate update for ρ is also discussed
in [294].
With these elements, the composite-step trust region algorithm follows.

Algorithm 5.5.
Set all tolerances and constants; initialize B 0 if using quasi-Newton updates, as well as
x 0 , v 0 , ρ 0 , and 0 .

For k ≥ 0 while dxk  > 1 and ∇L(x k , v k ) > 2 :

1. At x k evaluate f (x k ), h(x k ), ∇f (x k ), Ak , Y k , and Z k .
2. Calculate v k from (5.93) and if exact second derivatives are available, calculate W k
and H̄ k = (Z k )T W k Z k . Else, calculate H̄ k = B k from a quasi-Newton update.
3. Calculate Y k pY from (5.90).
4. Set w k = (Z k )T W k YpY or an approximation.
5. Calculate pZ from (5.91).
6. Update the penalty parameter (ρ k ) from (5.98).
7. Calculate the merit function f (x k ) + ρ k ||h(x k )|| as well as ared and pred.
• If ared /pred < κ1 , set k+1 = γ1 k ,
• else, if ared /pred ≥ κ2 , set k+1 = k /γ1 ,
• else, if κ1 ≤ ared /pred < κ2 , set k+1 = k .
• If ared /pred > κ0 , then x k+1 = x k + dx . Else, x k+1 = x k .

Typical values for this algorithm are κ0 = 10−4 , κ1 = 0.25, κ2 = 0.75, γ1 = 0.5 and we
note that the trust region algorithm is similar to Algorithm 3.3 from Chapter 3.
The trust region algorithm is stated using reduced-space steps. Nevertheless, for large
sparse problems, the tangential and normal steps can be calculated more cheaply using only

i i

i i
 book_tem
i i
2010/7/27
page 126
i i

126 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

full-space information based on the systems (5.29) and (5.28). Moreover, the tangential
problem allows either exact or quasi-Newton Hessian information. The following conver-
gence property holds for all of these cases.

Assumptions II: The sequence {x k } generated by Algorithm 5.5 is contained in a convex

set D with the following properties:

1. The functions f : Rn → R and h : Rn → Rm are twice differentiable and their first and
second derivatives are Lipschitz continuous and uniformly bounded in norm over D.

2. The matrix A(x) has full column rank for all x ∈ D, and there exist constants γ0 and
β0 such that
Y (x)[A(x)T Y (x)]−1  ≤ γ0 , Z(x) ≤ β0 , (5.99)
for all x ∈ D.

Theorem 5.12 [100] Suppose Assumptions II hold; then Algorithm 5.5 has the following
properties:

lim h(x k ) = 0 and lim ∇x L(x k , v k ) = 0. (5.100)

k→∞ k→∞

In other words, all limit points are feasible and x ∗ is a first order stationary point for the
equality constrained NLP (5.1).

5.7.2 Filter Trust Region Methods

The filter methods were initially developed in the context of trust region methods. Because
these methods incorporate a feasibility restoration step, they can be applied directly to the
trust region constrained quadratic programming problem:

1
min mf (dx ) = ∇f (x k )T dx + dxT W k dx (5.101)
dx 2
s.t. h(x k ) + (Ak )T dx = 0, dx  ≤ k .

The filter trust region method uses similar concepts as in the line search case and can
be described by the following algorithm.

Algorithm 5.6.
Initialize all constants and choose a starting point (x 0 , v 0 ) and initial trust region 0 . Ini-
tialize the filter F0 := {(θ, f ) ∈ R2 : θ ≥ θmax } and the iteration counter k ← 0.

For k ≥ 0 while dxk , dvk  > 1 and ∇L(x k , v k ) > 2 :

1. Evaluate f (x k ), h(x k ), ∇f (x k ), Ak , and W (x k , v k ) at x k and v k .

2. For the current trust region, check whether a solution exists to (5.101). If so, continue
to the next step. If not, add x k to the filter and invoke the feasibility restoration phase

i i

i i
 book_tem
i i
2010/7/27
page 127
i i

5.7. Trust Region Methods 127

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

to find a new point x r , such that

1
min mf (dx ) = ∇f (x r )T dx + dxT W (x r )dx (5.102)
dx 2
s.t. h(x r ) + ∇h(x r )T dx = 0,
dx  ≤ k

has a solution and x r is acceptable to the filter. Set x k = x r .

3. Determine a trial step dx that solves (5.101).

4. Check acceptability to the filter.

• If x k + d k ∈ Fk or does not satisfy

θ (x k + dx ) ≤ (1 − γθ )θ (x k ) or f (x k + dx ) ≤ f (x k ) − γf θ(x k ),

then reject the trial step size, set x k+1 = x k and k+1 ∈ [γ0 k , γ1 k ] and go to
step 1.
• If

mf (x k ) − mf (x k + dx ) ≥ κθ θ(x k )2 (5.103)

and

f (x k ) − f (x k + dx )
πk = < η1 , (5.104)
mf (x k ) − mf (x k + dx )

then we have a rejected f -type step. Set x k+1 = x k and k+1 ∈ [γ0 k , γ1 k ]
and go to step 1.
• If (5.103) fails, add x k to the filter. Otherwise, we have an f -type step.

5. Set x k+1 = x k + dx and choose k+1 ∈ [γ1 k , k ] if π k ∈ [η1 , η2 ) or k+1 ∈

[k , γ2 k ] if π k ≥ η2 .

Typical values for these constants are γ0 = 0.1, γ1 = 0.5, γ2 = 2, η1 = 0.01, η2 = 0.9,
κθ = 10−4 . Because the filter trust region method shares many of the concepts from the
corresponding line search method, it is not surprising that it shares the same convergence
property.

Theorem 5.13 [135] SupposeAssumptions I hold; thenAlgorithm 5.6, with no unsuccessful

terminations of the feasibility restoration phase, has the following properties:

lim h(x k ) = 0 and lim inf ∇x L(x k , v k ) = 0. (5.105)

k→∞ k→∞

i i

i i
 book_tem
i i
2010/7/27
page 128
i i

128 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

5.8 Combining Local and Global Properties

The last part of the convergence analysis requires stitching together the fast local properties
of Newton-like steps from Sections 5.3 and 5.4 with global properties of line search or
trust region methods in Sections 5.6 and 5.7, respectively. In Chapter 3, we saw that line
search and trust region methods applied to unconstrained optimization problems could be
designed so that full steps (α k = 1) would be accepted near the solution, thus leading to
fast local convergence by Newton-like methods. For constrained optimization problems like
(5.1), neither the p merit function nor the filter criterion (regardless of a trust region or
line search context) can guarantee acceptance of full steps. This failure to allow full steps
near the solution is termed the Maratos effect and can lead to poor, “creeping” performance
near the solution. As a result, a further extension, called a second order correction, needs to
be added to the step calculation, whenever the algorithm approaches a neighborhood of the
solution.

5.8.1 The Maratos Effect

To appreciate the issues of poor local convergence, we first consider the following classical
example, analyzed in [93].

Example 5.14 (Slow Convergence Near Solution). Consider the following equality con-
strained problem:

min −x1 + τ ((x1 )2 + (x2 )2 − 1) (5.106)

s.t. (x1 )2 + (x2 )2 = 1

with the constant τ > 1. As shown in Figure 5.6, the solution to this problem is x ∗ = [1, 0]T ,
v ∗ = 1/2 − τ and it can easily be seen that ∇xx L(x ∗ , v ∗ ) = I at the solution. To illustrate the
effect of slow convergence, we proceed from a feasible point x k = [cos θ, sin θ ]T and choose
θ small enough to start arbitrarily close to the solution. Using the Hessian information at

Figure 5.6. Example of second order correction—detail in right graphic.

i i

i i
 book_tem
i i
2010/7/27
page 129
i i

5.8. Combining Local and Global Properties 129

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

x ∗ and linearizing (5.5) at x k allows the search direction to be determined from the linear
system
    
1 0 2x1k d1 1 − 2τ x1k
 
 0 1 2x2k   d2  =  −2τ x2k  (5.107)
k
2x1 2x2 k 0 v̄ 1 − (x1 ) − (x2 )
k 2 k 2

yielding d1 = sin2 θ , d2 = − sin θ cos θ and x k + d = [cos θ + sin2 θ, sin θ (1 − cos θ )]T . As
seen in Figure 5.6, x k + d is much closer to the optimum solution, and one can show that
the ratio
x k + d − x ∗  1
= (5.108)
x k − x ∗ 2 2
is characteristic of quadratic convergence for all values of θ . However, with the p merit
function, there is no positive value of the penalty parameter ρ that allows this point to be
accepted. Moreover, none of the globalization methods described in this chapter accept this
point either, because the infeasibility increases:

h(x k ) = 0 → h(x k + d) = sin2 θ

and the objective function also increases:

f (x k ) = − cos θ → f (x k + d) = − cos θ + (τ − 1) sin2 θ.

As a result, x k + d leads to an increase in an p merit function and is also not acceptable

to a filter method at x k . Consequently, line search or trust region methods will take smaller
steps that lead to slower convergence.

To avoid small steps that “creep toward the solution,” we develop a second order
correction at iteration k. The second order correction is an additional step taken in the range
space of the constraint gradients in order to reduce the infeasibility from x k + dx . Here we
define the correction by

h(x k + dx ) + A(x k )T d̄x = 0, (5.109)

where d̄ = Y k p̄ and can be determined from

d̄x = Y k p̄ = −Y k (A(x k )T Y k )−1 h(x k + dx ). (5.110)

Example 5.15 (Maratos Example Revisited). To see the influence of the second order cor-
rection, we again refer to Figure 5.6. From (5.110) with Y k = A(x k ), we have

p̄ = − sin2 θ/4 and d̄ = −[sin2 θ cos θ/2, sin3 θ/2]T , (5.111)

and we can see that x k + dx + d̄x is even closer to the optimum solution with

x k + dx − x ∗  = 1 − cos θ
≥ 1 − cos θ − (cos θ + 5)(sin2 θ )/4
= x k + dx + d̄x − x ∗ ,

i i

i i
 book_tem
i i
2010/7/27
page 130
i i

130 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

which is also characteristic of quadratic convergence from (5.108). Moreover, while the
infeasibility still increases from h(x k ) = 0 to h(x k + dx + d̄x ) = sin4 θ/4, the infeasibility is
considerably less than at x k + dx . Moreover, the objective function becomes

f (x k + dx + d̄x ) = −(cos θ + sin2 θ (1 − cos θ/2)) + τ sin4 θ/4.

For (1 − cos θ/2)/ sin2 θ > (τ + ρ)/4, the objective function clearly decreases from f (x k ) =
− cos θ. Moreover, this point also reduces the exact penalty and is acceptable to the filter.
Note that as θ → 0, this relation holds true for any bounded values of τ and ρ. As a
result, the second order correction allows full steps to be taken and fast convergence to be
obtained.

Implementation of the second order correction is a key part of all of the globalization
algorithms discussed in this chapter. These algorithms are easily modified by first detect-
ing whether the current iteration is within a neighborhood of the solution. This can be
done by checking whether the optimality conditions, say max(h(x k ), (Z k )T ∇f (x k )) or
max(h(x k ), ∇L(x k , v k )), are reasonably small. If x k is within this neighborhood, then
d̄x is calculated and the trial point x k + dx + d̄x is examined for sufficient improvement.
If the trail point is accepted, we set x k+1 = x k + dx + d̄x . Otherwise, we discard d̄x and
continue either by restricting the trust region or line search backtracking.

5.9 Summary and Conclusions

This chapter extends all of the topics that relate to Newton-type algorithms in Chapter 3
to equality constrained NLPs. The addition of equality constraints adds a number of subtle
complications, which are reflected both in the calculation of search steps and globaliza-
tion of the algorithm. For the calculation of well-defined Newton-type steps, some care is
needed to maintain nonsingularity of the KKT matrix as well as the correct inertia; this is a
direct extension of maintaining a positive definite Hessian for unconstrained minimization.
Moreover, depending on the problem structure, these steps can be calculated either in the
full variable space or in reduced spaces, with steps that are tangential and normal to the
linearized constraints. In the absence of exact second derivatives, quasi-Newton methods
can be applied, and it should be noted that only the reduced Hessian requires a positive
definite approximation.
Globalization strategies require that the constraint violations vanish and the objective
function be minimized in the constrained space. For this, p penalty functions or the more
recent filter strategy are typically applied to monitor these competing goals. Either strategy
can be applied to the line search or trust region approach. For the line search strategy, a step
size α can be found that either improves the exact penalty function or is acceptable to the
filter. Otherwise, the extension from the unconstrained case is reasonably straightforward.
On the other hand, the trust region method needs to be modified to avoid infeasible solutions
of (5.89). This is done either with a composite-step trust region method (with exact penalty)
or through a restoration phase (filter strategy).
Finally, unlike the unconstrained case, these globalization methods may not allow full
Newton-type steps to be taken near the solution, and “creeping steps” can be observed close
to x ∗ . As a result, a correction step needs to be introduced near the solution to compensate
for additional constraint curvature and promote fast convergence.

i i

i i
 book_tem
i i
2010/7/27
page 131
i i

5.11. Exercises 131

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

5.10 Notes for Further Reading

The analysis of the KKT matrix is due to Gould [169] and the inertia corrections are discussed
in [404]. The development of reduced-space methods is described in [162, 294, 134] and
also in [82, 54], and the development of quasi-Newton methods is described in [405, 83]
for the full space and [82, 54] for the reduced space. Local convergence properties with
exact and quasi-Newton Hessian information are analyzed and developed in [63, 82, 54].
Penalty functions have a rich literature and the classical treatment of their properties is given
in Fiacco and McCormick [131]. The development of properties for p penalty functions
is due to Han and Mangasarian [185], while augmented Lagrange functions are described
in [39, 134, 294]. The treatment of line search and trust region globalization with penalty
functions follows along the lines of [294, 100]. Filter methods were developed in [137] and
the development and analysis of these methods can be found in [135] and [403] for the trust
region and line search cases, respectively. Moreover, a summary on the development of filter
methods for a broad variety of algorithms can be found in [139]. Finally, the Maratos effect
example is due to [93] and also presented in [294]. In addition to second order corrections,
an alternative approach is to monitor improvement after executing a series of full Newton
steps. Particular implementations of this concept include the Watchdog algorithm [93] and
nonmonotone methods (see [174]), which also lead to fast convergence and overcome the
Maratos effect.

5.11 Exercises
1. Prove Theorem 5.1 by modifying the proof of Theorem 2.20.
2. Consider the penalty function given by (5.56) with a finite value of ρ, and compare the
KKT conditions of (5.1) with a stationary point of (5.56). Argue why these conditions
cannot yield the same solution.
3. Show that the tangential step dt = Z k pZ and normal step dn = Y k pY can be found
from (5.28) and (5.29), respectively.
4. Prove Lemma 5.9 by modifying the proof of Lemma 3.2 in [82] and Lemma 4.1
in [54].
5. Fill in the steps used to obtain the results in Example 5.15. In particular, show that for
(1 − cos θ/2)/ sin2 θ > τ + ρ,
the second order correction allows a full step to be taken for either the p penalty or
the filter strategy.
6. Consider the problem
1
min f (x) = (x12 + x22 ) (5.112)
2
s.t. h(x) = x1 (x2 − 1) − θx2 = 0, (5.113)
where θ is an adjustable parameter. The solution to this problem is x1∗ = x2∗ = v ∗ = 0
2 L(x ∗ , v ∗ ) = I . For θ = 1 and θ = 100, perform the following experiments
and ∇xx
with xi0 = 1/θ , i = 1, 2.

i i

i i
 book_tem
i i
2010/7/27
page 132
i i

132 Chapter 5. Newton Methods for Equality Constrained Optimization

Downloaded 04/07/15 to 169.230.243.252. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

• Setting B k = I , solve this problem with full-space Newton steps and no glob-
alization.
• Setting B̄ k = (Z k )T Z k and w k = 0, solve the above problem with reduced-space
Newton steps and no globalization, using orthogonal bases (5.26).
• Setting B̄ k = (Z k )T Z k and w k = 0, solve the above problem with reduced-space
Newton steps and no globalization, using coordinate basis (5.23).
• Setting B̄ k = (Z k )T Z k and w k = (Z k )T Y k pY , solve the above problem with
reduced-space Newton steps and no globalization, using coordinate basis (5.23).

i i

i i