Numerical Optimization

Constrained Optimization - Algorithms

Shirish Shevade
Computer Science and Automation
Indian Institute of Science
Bangalore 560 012, India.

NPTEL Course on Numerical Optimization

Shirish Shevade Numerical Optimization

 Barrier and Penalty Methods

Consider the problem:

min f (x)
s.t. x ∈ X

where X ∈ Rn .
Idea:
Approximation by an unconstrained problem
Solve a sequence of unconstrained optimization problems
Penalty Methods
Penalize for violating a constraint

Barrier Methods
Penalize for reaching the boundary of an inequality constraint
Shirish Shevade Numerical Optimization
 min f (x)
s.t. x ∈ X

Define a function,

0 if x ∈ X
ψ(x) =
+∞ if x ∈
/X

Solve an equivalent unconstrained problem:

min f (x) + ψ(x)

Not a practical approach

Replace ψ(x) by a sequence of continuous non-negative
functions that approach ψ(x)

Shirish Shevade Numerical Optimization

 Penalty Methods

min f (x)
s.t. x ∈ X
Let x∗ be a local minimum
Let X = {hj (x) ≤ 0, j = 1, . . . , l}
Define
l
1X
P(x) = [max(0, hj (x)]2
2 j=1
Define q(x, c) = f (x) + cP(x)
Define a sequence {ck } such that ck ≥ 0 and ck+1 > ck
∀ k.
Let xk = argminx q(x, ck )
Ideally, {xk } → x∗ as {ck } → +∞
Shirish Shevade Numerical Optimization
Nonlinear Program (NLP)

min f (x)
s.t. hj (x) ≤ 0, j = 1, . . . , l
ei (x) = 0, i = 1, . . . , m

Define
l m
1X 1X 2
P(x) = [max(0, hj (x)]2 + e (x)
2 j=1 2 i=1 i

and
q(x, c) = f (x) + cP(x).

Assumption: f , hj ’s and ei ’s are sufficiently smooth

Shirish Shevade Numerical Optimization
Lemma
If xk = argminx q(x, ck ) and ck+1 > ck , then
q(xk , ck ) ≤ q(xk+1 , ck+1 )
P(xk ) ≥ P(xk+1 )
f (xk ) ≤ f (xk+1 ).
Proof.
q(xk+1 , ck+1 ) = f (xk+1 ) + ck+1 P(xk+1 )
≥ f (xk+1 ) + ck P(xk+1 )
≥ f (xk ) + ck P(xk )
= q(xk , ck )
Also, f (xk ) + ck P(xk ) ≤ f (xk+1 ) + ck P(xk+1 ) . . . (1)
f (xk+1 ) + ck+1 P(xk+1 ) ≤ f (xk ) + ck+1 P(xk ). . . . (2)

Adding (1) and (2), we get P(xk ) ≥ P(xk+1 ).

f (xk+1 ) + ck P(xk+1 ) ≥ f (xk ) + ck P(xk ) ⇒ f (xk+1 ) ≥ f (xk )
Shirish Shevade Numerical Optimization
Lemma
Let x∗ be a solution to the problem,
min f (x)
. . . (P1)
s.t. x ∈ X.

Then, for each k, f (xk ) ≤ f (x∗ ).

Proof.
f (xk ) ≤ f (xk ) + ck P(xk )
≤ f (x∗ ) + ck P(x∗ ) = f (x∗ )

Theorem
Any limit point of the sequence, {xk } generated by the penalty
method is a solution to the problem (P1).
Shirish Shevade Numerical Optimization
Nonlinear Program (NLP)

min f (x)
s.t. hj (x) ≤ 0, j = 1, . . . , l
ei (x) = 0, i = 1, . . . , m

Penalty Function Method (to solve NLP)

∞
(1) Input: {ck }k=0 , 
(2) Set k := 0, initialize xk
(3) while (q(xk , ck ) − f (xk )) > 
(a) xk+1 = argminx q(x, ck )
(b) k := k + 1
endwhile
Output : x∗ = xk
Shirish Shevade Numerical Optimization
Example:
1
min 2
[(x1 − 3)2 + (x2 − 2)2 ]
s.t. −x1 + x2 ≤ 0
x1 + x2 ≤ 1
−x2 ≤ 0
x∗ = (1, 0)T

Shirish Shevade Numerical Optimization

 1
min 2
[(x1 − 3)2 + (x2 − 2)2 ]
s.t. −x1 + x2 ≤ 0
x1 + x2 ≤ 1
−x2 ≤ 0

1 c
q(x, c) = [(x1 − 3)2 + (x2 − 2)2 ] + [(max(0, −x1 + x2 ))2
2 2
+(max(0, x1 + x2 − 1))2 + (max(0, −x2 ))2 ]
Let x0 = (3, 2)T (Violates the constraint x1 + x2 ≤ 1 )
At x0 ,
1 c
q(x, c) = [(x1 − 3)2 + (x2 − 2)2 ] + [(x1 + x2 − 1)2 ].
2 2 2c+3

∇x q(x, c) = 0 ⇒ x1 (c) = 2c+1
2 = x∗ (c)
2c+1
Shirish Shevade Numerical Optimization
 1
min 2
[(x1 − 3)2 + (x2 − 2)2 ]
s.t. −x1 + x2 ≤ 0
x1 + x2 ≤ 1
−x2 ≤ 0

At x0 = (3, 2)T ,

1 c
q(x, c) = [(x1 − 3)2 + (x2 − 2)2 ] + [(x1 + x2 − 1)2 ].
2 2
 2c+3 
1
∇x q(x, c) = 0 ⇒ x (c) = 2c+1
2 = x∗ (c)
2c+1
Taking limit as c → ∞, x∗ = (1, 0)T

Shirish Shevade Numerical Optimization

Consider the problem,
min f (x)
s.t. e(x) = 0
Let (x∗ , µ∗ ) be a KKT point (∇ f (x∗ ) + µ∗ ∇ e(x∗ ) = 0)
Penalty Function: q(x, c) = f (x) + cP(x)
As c → ∞, q(x, c) = f (x)
Consider the perturbed problem,
min f (x)
s.t. e(x) = θ
and the penalty function,
q̂(x, c) = f (x) + c(e(x) − θ)2
= f (x) − 2cθe(x) + ce(x)2 (ignoring constant term)
= f (x) + µe(x) +ce(x)2
| {z }
L(x,µ)

= L̂(x, µ, c) (Augmented Lagrangian Function)

Shirish Shevade Numerical Optimization
At (x∗ , µ∗ ), ∇x L(x∗ , µ∗ ) = ∇ f (x∗ ) + µ∗ ∇ e(x∗ ) = 0.
∴ ∇x q̂(x∗ , c) = ∇x L̂(x∗ , µ∗ , c)
= ∇x L(x∗ , µ∗ ) + 2ce(x∗ )∇ e(x∗ )
= 0 ∀c

Q. How to get an estimate of µ∗ ?

Let x∗c be a minimizer of L(x, µ, c). Therefore,

∇x L(x∗c , µ, c) = ∇ f (x∗c ) + µ∇ e(x∗c ) + ce(x∗c )∇ e(x∗c ) = 0

∴ ∇ f (x∗c ) = − (µ + ce(x∗c )) ∇ e(x∗c )
| {z }
estimate of µ∗

Shirish Shevade Numerical Optimization

Program (EP)

min f (x)
s.t. e(x) = 0

Augmented Lagrangian Method (to solve EP)

(1) Input: c, 
(2) Set k := 0, initialize xk , µk
(3) while (L̂(xk , µk , c) − f (xk )) > 
(a) xk+1 = argminx L̂(xk , µk , c)
(b) µk+1 = µk + ce(xk )
(c) k := k + 1
endwhile
Output : x∗ = xk
Shirish Shevade Numerical Optimization
Nonlinear Program (NLP)

min f (x)
s.t. hj (x) ≤ 0, j = 1, . . . , l
ei (x) = 0, i = 1, . . . , m
Easy to extend the Augmented Lagrangian Method to NLP
Rewrite the inequality constraint, h(x) ≤ 0 as an equality
constraint,
h(x) + y2 = 0

Shirish Shevade Numerical Optimization

 Barrier Methods

Typically applicable to inequality constrained problems

min f (x)
s.t. hj (x) ≤ 0, j = 1, . . . , l

Let X = {x : hj (x) ≤ 0, j = 1, . . . , l}
Some Barrier functions (defined on the interior of X)
l l
X 1 X
B(x) = − or B(x) = − log(−hj (x))
j=1
hj (x) j=1

Approximate problem using Barrier function (for c > 0)

min f (x) + 1c B(x)
s.t. x ∈ Interior of X
Shirish Shevade Numerical Optimization
 Cutting-Plane Methods
Primal Problem Dual Problem

min f (x)
max θ(λ, µ)
s.t. hj (x) ≤ 0, j = 1, . . . , l
ei (x) = 0, i = 1, . . . , m s.t. λ ≥ 0
x∈X
X is a compact set.
Dual Function: z = θ(λ, µ) = minx∈X f (x) + λT h(x) + µT e(x)
Equivalent Dual problem
max z
z,µ,λ

s.t. z ≤ f (x) + λT h(x) + µT e(x), x ∈ X

λ≥0

Linear Program with infinite constraints

Shirish Shevade Numerical Optimization
Equivalent Dual problem
max z
z,µ,λ

s.t. z ≤ f (x) + λT h(x) + µT e(x), x ∈ X

λ≥0
Idea: Solve an approximate dual problem.
k−1
Suppose we know {xj }j=0 such that

z ≤ f (x) + λT h(x) + µT e(x), x ∈ {x0 , . . . , xk−1 }

Approximate Dual Problem
max z
z,µ,λ

s.t. z ≤ f (x) + λT h(x) + µT e(x), x ∈ {x0 , . . . , xk−1 }

λ≥0
Let (zk , λk , µk ) be the optimal solution to this problem.
Shirish Shevade Numerical Optimization
Approximate Dual Problem

max z
z,µ,λ

s.t. z ≤ f (x) + λT h(x) + µT e(x), x ∈ {x0 , . . . , xk−1 }

λ≥0
T T
If zk ≤ f (x) + λk h(x) + µk e(x) ∀ x ∈ X, then (zk , λk , µk ) is
the solution to the dual problem.
T T
Q. How to check if zk ≤ f (x) + λk h(x) + µk e(x) ∀ x ∈ X?
Consider the problem,

T T
min f (x) + λk h(x) + µk e(x)
s.t. x∈X
and let xk be an optimal solution to this problem.
Shirish Shevade Numerical Optimization
 T T
min f (x) + λk h(x) + µk e(x)
s.t. x∈X
and let xk be an optimal solution to this problem.
T T
If zk ≤ f (xk ) + λk h(xk ) + µk e(xk ), then (λk , µk ) is an
optimal solution to the Lagrangian dual problem.
T T
If zk > f (xk ) + λk h(xk ) + µk e(xk ), then add the
constraint, z ≤ f (xk ) + λT h(xk ) + µT e(xk ) to the
approximate dual problem.

Shirish Shevade Numerical Optimization

Nonlinear Program (NLP)
min f (x)
s.t. hj (x) ≤ 0, j = 1, . . . , l
ei (x) = 0, i = 1, . . . , m
x∈X
Summary of steps for Cutting-Plane Method:
Initialize with a feasible point x0
while stopping condition is not satisfied
argmaxz,λ,µ z
k
(z , λ , µ ) = s.t. z ≤ f (xj ) + λT h(xj ) + µT e(xj ), j = 0, . . . , k − 1
k k

λ≥0

T T
argminx f (x) + λk h(x) + µk e(x)
xk =
x∈X
T T
Stop if zk ≤ f (xk ) + λk h(xk ) + µk e(xk ). Else, k := k + 1.
Shirish Shevade Numerical Optimization