TMA947 / MMG621 – Nonlinear optimization Lecture 7

TMA947 / MMG621 — Nonlinear optimization

Lecture 7 — The Karush-Kuhn-Tucker conditions

Emil Gustavsson, Zuzana Nedělková

November 13, 2017

[Minor revision: Axel Ringh - August, 2023]

Consider a problem of the form

min f (x), (1a)

subject to gi (x) ≤ 0, i = 1, . . . , m (1b)

where now f : Rn → R, and gi : Rn → R, i = 1, . . . , m are all C 1 , i.e., we take S to be of the form

S := {x ∈ Rn | gi (x) ≤ 0, i = 1, . . . , m}.

1 KKT conditions

We begin by developing the KKT conditions when we assume some regularity of the problem.
We assume that the problem considered is well behaved, and postpone the issue of whether any
given problem is well behaved until later.

Definition 1 (Abadie’s constraint qualification). We say that the problem (1) satifies Abadie’s con-
straint qualification if TS (x) = G(x) for all x ∈ S.

Remark: Abadie’s constraint qualification should be viewed as an abstract condition expressing

"(1) is well-behaved"

Theorem 1. Assume that the problem (1) satisfies Abadie’s CQ, then at any locally optimal point x∗ the
system

X
m
∇f (x∗ ) + µi ∇gi (x∗ ) = 0, (2)
i=1
µi gi (x∗ ) = 0, i = 1, . . . , m, (3)
µi ≥ 0, i = 1, . . . , m. (4)

has a solution µ.

Proof. See Theorem 5.29 in the book.

1
TMA947 / MMG621 – Nonlinear optimization Lecture 7

Note that the KKT conditions are precisely the Fritz-John conditions, with the added requirement
that µ0 = 1. We call the vector µ solving the KKT system for some fixed x ∈ S a Lagrange
multiplier.

2 Constraint qualifications

Our final task in showing that the KKT conditions are usable is to find practical conditions under
which we can guarantee that Abadie’s CQ holds. We start with one of the simplest and most
useful ones.

Definition 2 (LICQ). We say that the linear independence constraint qualification (LICQ) holds at x ∈ S
if the gradients ∇gi (x), i ∈ I are linearly independent.
Proposition 1. The LICQ implies Abadie’s CQ.

Proof. See Proposition 5.40 and 5.46 in the book

The other constraint qualifications we will consider in this course comes from comparing the
cones defined previously. As a first example we consider the Mangasarian-Fromowitz CQ.
Definition 3 (MFCQ). The Mangasarian-Fromowitz constraint qualification (MFCQ) holds at x ∈ S if
◦
G(x) is nonempty.

Proposition 2. The MFCQ implies the Abadie’s CQ.

Proof. See Proposition 5.40 in the book.

The MFCQ can be used to get other constraint qualifications as well.

Definition 4 (Slater CQ). Slater CQ holds for (1) if gi , i = 1, . . . , m are convex functions, and there
exists an interior point, i.e., a point x0 such that gi (x0 ) < 0 for all i ∈ {1, . . . , m}.

Proposition 3. The Slater CQ implies Abadie’s CQ.

Proof. See Proposition 5.43 in the book.

Definition 5 (Affine constraints CQ). The affine constraints holds for (1) if gi , i = 1, . . . , m are affine
functions.

Proposition 4. The affine constraints CQ imply Abadie’s CQ.

Proof. If the constraints are affine then for any p and any x ∈ S and any i ∈ I(x) we have
gi (x + tp) = gi (x) + t∇gi (x)T p = t∇gi (p)T p. So if p ∈ G(x) we have gi (x + tp) ≤ 0, and thus
x + tp is feasible for all small enough t ≥ 0, i.e., p ∈ RS (x). Hence G(x) = RS (x) from which it
follows that TS (x) = G(x).

2
TMA947 / MMG621 – Nonlinear optimization Lecture 7

3 Equality constraints

So far we have only talked about problems with inequality constraints, we briefly outline how to
apply the above theory for the problem to

min f (x), (5a)

subject to gi (x) ≤ 0, i = 1, . . . , m, (5b)
hj (x) = 0, j = 1, . . . , l. (5c)
where all the functions above are C 1 . The main idea is to replace the equality constraints hj (x) = 0,
with two inequality constraints, i.e., by hj (x) ≤ 0 and hj (x) ≥ 0, and apply the KKT theory to the
problem

min f (x), (6a)

subject to gi (x) ≤ 0, i = 1, . . . , m, (6b)
hj (x) ≤ 0, j = 1, . . . , l, (6c)
−hj (x) ≤ 0, j = 1, . . . , l. (6d)
The main observation is that the equality constraints are always active in any feasible solution,
and they will enter the KKT system with non-negative multipliers of opposite sign, which we can
rewrite as a multiplier with any sign restrictions. The KKT system becomes

X
m X
l
∇f (x∗ ) + µi ∇gi (x∗ ) + λj ∇hj (x∗ ) = 0, (7a)
i=1 j=1

µi gi (x∗ ) = 0, i = 1, . . . , m, (7b)
µi ≥ 0, i = 1, . . . , m. (7c)

The main difficulty is what happens to the constraint qualifications when we add equality con-
straints.
◦
The main CQ was the MFCQ previously, i.e., G(x) 6= ∅. But in the presence of equality constraints
we need to introduce a cone
H(x) := {p ∈ Rn | ∇hj (x)T p = 0, j = 1, . . . , l}
◦
and try to develop the theory of the previous sections when replacing the cones G(x) and G(x)
◦
by G(x) ∩ H(x) and G(x) ∩ H(x) respectively. We cannot immediately generalize the statement
◦ ◦
G(x) ⊆ TS (x) to the statement G(x) ∩ H(x) ⊆ TS (x). However, it turns out that we need to
require the set of gradients {∇hj (x)}lj=1 to be linearly independent. The constraint qualifications
above then have to be modified to include the statement "and the set of gradients {∇hj (x)}lj=1 is
linearly independent"1 . Refer to the book for detailed statements of all the CQs.
1 Except the affine constraints CQ and Abadie’s CQ.

3
TMA947 / MMG621 – Nonlinear optimization Lecture 7

4 Sufficiency of the KKT conditions under convexity

We have developed neccessary optimality conditions for the problem (1) so far. However, we want
to know whether the KKT conditions are sufficient for optimality, that is, if the KKT system is
solvable at x∗ can we conclude that x∗ is optimal in (1)? In the unconstrained case, we saw that
the property that allows such statements is convexity, and it turns out that what we need in the
constrained case is also convexity, but in terms of both the objective function and the constraints.

Theorem 2. If, in (1), the objective function f and all constraint functions gi , i = 1, . . . , m are convex,
then the KKT conditions are a sufficient optimality condition.

Proof. See Theorem 5.49 in the book.

Note that if we apply the above theorem to problems with equality constraints, then we must
require that the equality constraints are affine.