Scribd
Upload
112 views21 pages

Augmented Lagrangian & The Method of Multipliers

This document discusses the augmented Lagrangian method and method of multipliers for solving constrained optimization problems. [1] It introduces the augmented Lagrangian which adds a quadratic penalty term to the objective function to soften constraints compared to barrier methods. [2] It shows that if the penalty parameter c is sufficiently large, the original problem's local minima will also be local minima for the augmented Lagrangian. [3] The method then iteratively minimizes the augmented Lagrangian with updates to the Lagrangian multipliers to converge to the original problem's solution.

Uploaded by

Jerry
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
112 views21 pages

Augmented Lagrangian & The Method of Multipliers

This document discusses the augmented Lagrangian method and method of multipliers for solving constrained optimization problems. [1] It introduces the augmented Lagrangian which adds a quadratic penalty term to the objective function to soften constraints compared to barrier methods. [2] It shows that if the penalty parameter c is sufficiently large, the original problem's local minima will also be local minima for the augmented Lagrangian. [3] The method then iteratively minimizes the augmented Lagrangian with updates to the Lagrangian multipliers to converge to the original problem's solution.

Uploaded by

Jerry
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 21

Augmented Lagrangian &

the Method of Multipliers


Constrained optimization
So far:
•  Projected gradient descent
•  Conditional gradient method
•  Barrier and Interior Point methods
•  Augmented Lagrangian/Method of Multipliers (today)
Quadratic Penalty Approach
Add a quadratic penalty instead of a barrier. For some c > 0

Note: Problem is unchanged – has same local minima

Augmented Lagrangian:
> c
Lc (x, ) = f (x) + h(x) + kh(x)k2
2
•  Quadratic penalty makes new objective strongly convex if
c is large
•  Softer penalty than barrier – iterates no longer confined to
be interior points.
Quadratic Penalty Approach
Solve unconstrained minimization of Augmented Lagrangian:
x = arg min Lc (x, )
x2X
where
> c
Lc (x, ) = f (x) + h(x) + kh(x)k2
2

When does this work?


Convergence mechanisms
1)  Take λ close to λ*.

Let x*, λ* satisfy the sufficiency conditions of second-order


for the original problem. We will show that if c is larger than
a threshold, then x* is a strict local minimum of the
Augmented Lagrangian Lc(., λ*) corresponding to λ*.

This suggest that if we set λ close to λ* and do


unconstrained minimization of Augmented Lagrangian:

x = arg min Lc (x, )


x2X

Then we can find x close to x*.


Second-order sufficiency conditions

We will show that if c is larger than a threshold, then x* also


satisfies these conditions for the Augmented Lagrangian Lc(.,
λ*) and hence is a strict local minimum of the Augmented
Lagrangian Lc(., λ*) corresponding to λ*.
Convergence mechanisms
Augmented Lagrangian:
> c
Lc (x, ) = f (x) + h(x) + kh(x)k2
2
Gradient and Hessian of Augmented Lagrangian:

If x*, λ* satisfy the sufficiency conditions of second-order for


original problem, we get:
Convergence mechanisms

Since
from sufficiency condition, we have for large enough c

using the following lemma:


Convergence mechanisms
1)  Take λ close to λ*.

2)  Take c very large, c !∞.

If c is very large, then solution of unconstrained Augmented


Lagrangian x is nearly feasible
Example

L(x, )

We also have for all c>0

We also have for all λ


Example
Quadratic Penalty Approach
How to choose λ and c?
Solve sequence of unconstrained minimization of Augmented
Lagrangian:

where
Basic convergence result

•  Assumes we can do exact minimization of the unconstrained


Augmented Lagrangian
Inexact minimization

Assume m. Then
Practical issues

Example:
Method of Multipliers
Solve sequence of unconstrained minimization of Augmented
Lagrangian:

where

and using the following multiplier update:

•  Note: Under some reasonable assumptions this works


even if {ck} is not increased to ∞.
Method of Multipliers
Example: Convex problem

Method of Multipliers:
Method of Multipliers
Example: Non-convex problem

Method of Multipliers:
Practical issues
Inequality constraints
Inequality constraints

•  Can show this reduces to:

•  Under similar assumptions as before,


⇤
! i ! µ⇤j

You might also like