NLP MultiVAr Constrained

NON-LINEAR PROGRAMMING
MULTIVARIABLE, CONSTRAINED
We will complete our investigations into the optimization
of continuous, non-linear systems with the multiple
variable case.
THE BIG ENCHILADA!
min f ( x )
x
s .t .
h( x ) = 0
g ( x) ≥ 0
x min ≤ x ≤ xmax
Opportunity for optimization.
Opt. Basics I Objective, feasible region, ..
Minimum: f(x*) < f(x*+δ x);

Opt. Basics II necessary & sufficient conditions.
We have been Single-Var. Line search for 1D optimization.

building Concept of Newton’s method.
Opt.
understanding
while learning
useful methods MultiVariable Algorithm that combines search
direction and line search.
and Opt.
algorithms
Concept of constrained optimization.
Opt. Basics III The Lagrangian and KKT conditions.
CLASS EXERCISE: Is this optimization constrained or
unconstrained?
Optimization of $$$ by adjusting XD, XB, Tf, P, …..

flooding
Reboiled vapor
Inputs
Heat
P, Xf, F, Tf Operation transfer
Tcw, Fcw Outputs safety
Freflux
Freboil XD weeping
Equip. XB Pressure
NT, NF Min ≤ Freflux ≤ Max
A, Ntubes, .. Min ≤ Freboil ≤ Max
Challenging, but I’m ready now!

We’ll learn methods in Multivariable Opt.

all three categories.
Numerical
Function values First order Second order

only
Non-continuous Smooth 1st derivatives Smooth 1st and 2nd der.

functions & derivatives
• Successive LP (SLP) • GRG, Aug Lagrangian
• COMPLEX
Function values only
SIMPLEX for unconstrained

Worst objective point,
Xh, which is dropped
Can we extend this for
constrained optimization?
13
9
8.9
x2
11.3 8.5
x1
COMPLEX for constrained
13
9 g(x) >0
8.9
x2
What do we
do now?
11.3 8.5
x1 Infeasible
point
COMPLEX for constrained
COMPLEX
g(x) >0 1. Use more points in simplex

2. Start at feasible point
x2
3. Reflect as usual
4. In infeasible, backtrack
x1 Infeasible until feasible
point
5. If backtrack not successful,
use next worst for reflection
Not
NON-LINEAR PROGRAMMING recommended
COMPLEX for constrained GENERALLY NOT SUCCESSFUL
COMPLEX
1. Cannot have equality
g(x) >0
constraints
x2 2. Requires initial feasible pt.
3. Very inefficient for large
number of variables
x1 Infeasible
point 4. Does not effectively move
along inequality constraints.
Sequential Linear Programming
SLP
First order approximation
Increasing profit Linear
Variable x2
Programming is
Optimum very successful.
Can we develop a
method for NLP
using a sequence of
LP approximations?
Variable x1
SLP, first order approximation
1. We remember that the LP solution will be at

boundaries of the feasible region, and always at a
corner point.
2. Thus, the solution to any iteration that involves a
LP will appear at the boundary (corner point) of an
approximate feasible region.
3. Therefore, our approach must place a limit on the
change in the variables per iteration
4. The common approach is to bound the change with
a “box” or hypercube.
N-L Opt. Problem N-L Opt. converted to LP approximation
min f ( x k ) + ∇f T ( x k )∆x k
min f ( x )
x s.t .
s .t . h ( x k ) + ∇ h T ( x k ) ∆x k − b h = 0
h( x ) = bh g( x k ) + ∇g T ( x k )∆x k − b g ≥ 0
g ( x ) ≥ bg ( ∆x min )k ≤ ∆x k ≤ ( ∆x max )k
x min ≤ x ≤ xmax x min ≤ x k + ∆x k ≤ x max
What is the basis for the approximation?

improvement
= starting point
= linearized constraints
Variable x2
= box limitation on ∆ x
Variable x1 Iteration No. 1

improvement
Why are the constraint

approximations (blue dashed
lines) so poor?
Variable x2
Are we sure of being

feasible with respect to
the N-L problem?

improvement
= starting point
Variable x2

improvement
= starting point
Variable x2

improvement
= starting point
Variable x2

N-L Opt. converted to LP approximation

N-L Opt. Problem
min f ( x ) min f ( xk ) + ∇f T ( xk )∆xk + w∑ ( S pi + S ni )
x i
s .t . s .t .
h( x ) = bh h( xk ) + ∇hT ( xk )∆xk − b = ( S pi + S ni )
g ( x ) ≥ bg ( ∆xmin ) k ≤ ∆xk ≤ ( ∆xmax ) k
x min ≤ x ≤ xmax xmin ≤ xk + ∆xk ≤ xmax
Convert to =
using slacks Love that Taylor Series!

f ( x k ) + ∇f T ( x k )∆x k + w ∑
i
(S pi + S ni ) The penalty term sets goal
of constraint satisfaction
s.t .
h( x k ) + ∇h T ( x k )∆x k − b = (S pi + S ni ) The slacks allow violation
if required for feasibility.
( ∆x min ) k ≤ ∆x k ≤ ( ∆x max )k
x min ≤ x k + ∆x k ≤ x max BOX: The bounds on ∆ x
must be reduced to
converge to an interior
optimum.
See: Baker & Lasdon, Successful SLP at EXXON, Man. Sci., 31, 3, 264-274 (1985)
f ( x k ) + ∇f T ( x k )∆x k + w ∑ (S
i
pi + S ni )
s.t .
h( x k ) + ∇h T ( x k )∆x k − b = (S pi + S ni )
BOX: The bounds on ∆ x must be
( ∆x min )k ≤ ∆x k ≤ ( ∆x max )k reduced to converge to an interior
x min ≤ x k + ∆x k ≤ x max optimum .
If ∆ xk changes sign from ∆ xk-1 , halve the box size

If ∆ xk is located at same bound for 3 executions, double box size
See: Baker & Lasdon, Successful SLP at EXXON, Man. Sci., 31, 3, 264-274 (1985)
Major steps in Successive Linear Programming algorithm

1. Define an initial point (x), box size, and penalty coefficients, w
2. Calculate the non-linear terms and their gradients at the
current point
3. Solve the Linear Program
4. If |∆ xk| < tolerance, terminate
5. Adjust box size according to rules previously presented
6. Set last solution to the current point and iterate
End
SLP - Successive Linear Programming
Good Features
1. Uses reliable LP codes
2. Equality and inequality
3. Requires only 1st derivatives
4. Uses advanced basis for all iterations
5. Separate linear and non-linear terms; only have to update
NL terms
Poor Features
1. Follows infeasible path (for N-L constraints)
2. Performance is often poor for problems that
- are highly non-linear
- many N-L variables
- solution not at constraints
WHEN TO USE SLP
1. SLP functions well for “mostly linear” problems

(< 5% NL coefficients in LP)
2. The non-linearities should be “mild”
3. When second derivatives are not available or
unreliable/inaccurate
4. Most (all?) variables are determined by constraints at
solution.
5. Problem already in LP; needs few NL terms.
CURRENT STATUS
• Not available in general purpose modelling systems (GAMS,

AMPL, etc.)
• Used often in some industries
• SLP is available in commercial products tailored for specific
applications in selected industries
NON-LINEAR PROGRAMMING MULTIVARIABLE,
CONSTRAINED
SLP - Class Workshop - Pooling

Formulate the problem and convert to SLP
V1
3% sulfur x = % sulfur Sulfur ≤ 2.5%
6 $/BL
V1
pool Blend 1 0 ≤ B1 ≤ 100
9 $/BL
V2 No change F1
in
1% sulfur
inventory
16 $/BL allowed
V2 Sulfur ≤ 1.5%
0 ≤ B2 ≤ 200
Blend 2
F 10 $/BL
2% sulfur
10 $/BL
F2
Second order approximation
min f ( x )
Let’s quickly review key x
s .t .
results from h( x ) = 0
Optimization Basics III g ( x) ≥ 0
L ( x, λ , u ) = f ( x ) + ∑ λ j h j ( x ) + ∑ u k [ g k ( x )]
j k
Stationarity ∇ x , λ ,u L ( x, λ , u ) = 0
Curvature [∇ 2xx L( x, λ , u )]h =0, g a =0 pos. definite
Experience indicates that even min f ( x )

x
approximate information about
the curvature improves the
performance for most problems.
gradient hessian
T T 2
f ( x ) ≈ f ( x k ) + [∇f ( x k )] ( ∆x ) + 1 / 2[∆x] [∇ x f ( x k )][∆x]
Finding the stationarity after move gives the famous

“Newton Step”
∆x = − [∇ 2x ] −1
f ( xk ) ∇ x f ( x k )
L ( x, λ , u ) = f ( x ) + ∑ λ j h j ( x ) + ∑ u k [ g k ( x )]
j k
By using the Lagrangian as the objective function, we take

advantage of constraint curvature information and work
toward the stationarity (∇ L=0) that is required for
optimality.
x2 Every second-order NLP method will

have the following components
x1 • Select an initial point
Local descent direction • Linearize (evaluate gradients)
Curvature information • Evaluate the hessian
Uses gradient and hessian • Determine a search direction
Ensures improvement • Perform a line search
• Check convergence, iterate
Reduced Gradient min f ( x ) Line Search

x
Search in the (few) How do we tailor L.S.
“optimization degrees s .t . to constrained
of freedom”. h( x ) = 0 problems
Active Set g ( x) ≥ 0 Feasible Path
How do we tailor
solution methods for
x min ≤ x ≤ xmax How do we tailor L.S.
to constrained
constrained problems. problems
Let’s see how we build on unconstrained

and Basics III. These issues are critical!
Reduced Gradient - Second order approximation
We group the variables (x) into two categories. The

problem has n variables and m equations, with n > m.
xB = the basic variables that are

max f ( x) determined by the m
x
equations
s.t.
xNB = the n-m independent variables
h ( x) = 0 that are adjusted to achieve
the optimum
We apply the “equality constraint” approach of evaluating the

total derivative of the objective and constraints.
max f ( x1 , x2 ) max f 2 ( x1 )
x x1
s.t.
h ( x1 , x2 ) = 0
f(x)
All points on this curve
satisfy h(x1, x2)=0
x1 f2(x1)
x2
x1
Equality constraint
Which are the basic &

Where is the optimum? non-basic variables?
We will use the “equality constrained” approach to
determine the proper derivative direction
 dx1 
 ∂f ∂f   
max f ( x) df =  .....   ...  Basic
x  ∂x1 ∂xm 
dxm 
s.t.
dxm +1 
h ( x) = 0  ∂f ∂f    Non-
+ .....  ...  Basic
∂ x
 m +1 ∂x n
 dxn 
df = (∇ xB f ) dxB + (∇ xNB f ) dx NB
T T
∂h1 ∂h1 
 ∂x1 .. ∂xm   dx1 
max f ( x) dh( x ) =  .. .. ..   .. 
Basic
x  
∂hm dx 
 .. ..
∂xm   m 
s.t. 
∂h1 .. ∂h1 
h ( x) = 0  ∂xm +1 ∂xm +1  dxm +1 
  .. 
Non-
+ .. .. .. Basic
  
 .. ..
∂hm  dx 
∂xn   n 
?

dh = (∇ xB h)dxB + (∇ xNB h)dx NB = 0

Solve for xB in terms of the non-basic variables, xNB .

Substitute into df equation to yield the reduced gradient!
−1
dxB = −(∇ xB h) (∇ xNB h)dx NB Satisfy the
equations
Is it true
−1
df = −(∇ xB f ) (∇ xB h) ∇ xNB h dx NB
T that we have
satisfied
equations?
+ (∇ xNB f ) dx NB
T
Rearranging, we obtain df/dxNB , which is the

famous REDUCED GRADIENT!
 df 
 [ ]
 = (∇ xNB f ) − (∇ xB h) −1 ∇ xNB h T (∇ xB f )
 dx NB  h =0
An unconstrained method could use this gradient in

determining the search direction at each iteration. The
search space is often of much lower dimension!
max f ( x1 , x2 )
x Rearranging, we obtain
s.t. df/dxNB , which is the famous
REDUCED GRADIENT!
h ( x1 , x2 ) = 0
f(x)
Draw the following from
the current point
• unconstrained gradient
x1 • reduced gradient
x2
Current Equality
point constraint
Rearranging, we obtain df/dxNB , the famous REDUCED GRADIENT!
P C
1
For the distillation tower,

17
L C -1
determine the following
16 • number of variables
Feed has 6 15 • number of equations
components
• dimension of the reduced
3
space
2
L C -2
Active Set: We do not know which inequality
constraints will be active at the solution. Therefore, we
will allow constraints to change (active/inactive) status
during solution.
At an iteration, we will divide the degrees of freedom
(non-basic variables) into two groups.
1. Defined by an active inequality (become basic)
Search space for
2. “Free” or unconstrained (super basic) the optimization
No. of variables = No. XB + No. XSB

No. of equalities + No. of active inequalities
Active Set: The active or working set can change as
the method performs many iterations.
Example: Feasible region

inside the box
Key
letters = constraints
1
2 numbers = iterations
3 4
6
5
B Which
constraints are
active at each
iteration?
See Sofer and Nash, 1996

Example: Feasible region

inside the box Active Set
Iteration Constraints No. xSB
1
2
3 4
1 A 2
2 A 2
6
5 3 A, B 1
B 4 A, B, C 0
5 B, C 1
6 C 2
Example: Feasible region How do we determine

inside the box active constraints
at each iteration?
1
2
3 4
• Basic variable: If a variable is
6
5 at a bound and its Lagrange
B mult. is positive.
• Super basic: If a variable is
between bounds or at bound
with negative Lagrange
multiplier.
How do we divide When a constraint
variables into basic becomes active, which
and super basic? variable becomes basic?
• These decisions are not unique. In theory, many

selections would give equivalent results. However, we
want sub-problems that are easily solved.
• In practice, good selections are crucial to success. Good
algorithms have heuristics that provide reasonable
performance for a wide range (not all) problems.
NON-LINEAR PROGRAMMING: MULTIVARIABLE, CONSTRAINED
Active Set: The active set can change as the method performs many iterations.
P C
1
For the distillation tower,
determine the following
• recall the dimension of
the reduced space
L C -1
17
16 • what can change the

Feed has 6 15 dimension of reduced space
components
• can the dimension = 0?
3
L C -2
Line Search: This explores along the search direction

to find a distance giving acceptable ( greater than
minimum) improvement. Now, the search is in the
reduced space of superbasic variables.
What could go wrong with
a line search in a constrained
f(x+α s)
NLP?
0 α *
Linear search distance

Line Search: Now, the search is in the reduced space of

superbasic variables. Issue #1
infeasible • The line search could

extend out of the feasible
region.
f(x+α s)
• Therefore, the distance

must be limited by any
constraint status changing
from inactive to active
0 α *
How would we know?
Line Search: Now, the search is in the reduced space

of superbasic variables. Issue #2
The direction, s, is based on a
quadratic approximation
Often, the initial values of x
are far from x*.
• The search direction at xk
f(x+α s)
should not be extrapolated

too far.
• The line search might
require large changes in x,
which could lead to lack of
0 α *
convergence.
Line Search: Now, the search is in the reduced space

of superbasic variables. Issue #2
s
x2
Trust region: One approach to limiting the
search is to define a trust region. The
search is limited to within a region
x1
min f ( xk ) + ∇ x f ( xk )T [ s ] + 1 / 2[ s ]∇ 2x f ( xk )[ s ]
s
s.t.
s ≤ ∆k
Line Search: Issue # 2 ⇒ Trust region

min f ( xk ) + ∇ x f ( xk )T [ s ] + 1 / 2[ s ]∇ 2x f ( xk )[ s ]
s
x2
s
s.t.
Vector norm, e.g., 2-norm
x1
s ≤ ∆k
|| x || = x12 + x22 + ...
• The objective is the quadratic approximation.

• Solves for direction and distance simultaneously.
• If ∆ is “large”, we solve for the Newton step. If ∆ is “small”,
we solve for the Steepest Descent. Strategy?
• The ∆ k value of can be modified between iterations
Feasible/Infeasible Path: All methods will converge

to a feasible solution (if successful). Some methods require
“exact” feasibility during iterations toward the optimum; some
allow infeasibility wrt to original problem for intermediate
results.
x2 g(x) > 0
• What are advantages?
h(x) = 0
• Are there disadvantages?
x1
min f ( x )
x
Reduced Gradient Line Search
s.t.
Active Set h( x ) = 0 Feasible Path
g ( x) ≥ 0
Let’s look at two well-known NLP software

packages to see how they are implemented,
GRG2 and MINOS.
GRG2*
• Excel solver - most widely available

• Reduced gradient used as the basis for the search
direction
• Quasi-Newton - uses second-order information
• Feasible path - Ensures feasibility at every iteration
• Derivatives - Numerically (in Excel)
* = based on publicly available information

GRG2*
• Active set - At each iteration, selects which
inequality constraints are active/inactive. This
makes each iteration “unconstrained”.
• Line Search - No trust region
* = based on publicly available information

GRG2
1. Check active set, define h(x) = 0 and xB, xN and xSB

2. Determine the reduced gradient, ∇ SB f, update the
approximate hessian, and find the direction
3. Perform line search (feasible path)
- every point is a solution for all N-L equations
- go until ~ minimum or new constraint encountered
4. Check convergence ∇ SB f ≈ 0, ∆ x ≈ 0, ∆ f ≈ 0, and
ua > 0 (Lagrange mult. for active are positive).
GRG2
1. Since every point is feasible, L(x) = f(x)

2. Choice of xB is very important; we want an
invertible and “well conditioned” inverse basis.
Iterations and heuristics are required.
3. The size of the reduced space changes as constraints
are encountered. Must change the size of gradient
and hessian!
GRG2
Good Features Poor Features
1. Intermediate results are 1. Can be very slow with non-
feasible. linear constraints
2. Works in reduced space of 2. Can require extensive
low dimension computations because of
solving all non-linear
3. Can find initial feasible using
equations at every point on
“Phase I” approach.
the line search.
3. Convergence can require
good initial variable values.
MINOS
• GAMS, AMPL: MINOS is an optional solver (one

of many) in many modelling systems. It is also used
in other commercial products
• Reduced gradient used as the basis for the search
direction.
• Quasi-Newton - Uses second-order information
• Infeasible path - Merit is the Lagrangian
MINOS
• Active set - At each iteration, selects which
inequality constraints are active/inactive. This
makes each iteration “unconstrained”.
• Line Search - Can limit change per iteration in non-
linear variables through options file
• Linearized iterations - Used to speed progress
NLP - MULTIVARIABLE, CONSTRAINED
MINOS- Minor iter. using linearized constraints
I. Linearize the constraints (not objective function).
1. Check active set, define h(x) = 0 and xB, xN and xSB
Major 2. Determine the reduced gradient, ∇ SB L, update the

iteration approximate hessian, and find search direction
3. Perform line search
Minor - non-linear objective
iteration - linearized constraints
Check convergence on ∆ f ≈ 0 & max minor
4.
iterations.
II. Check convergence: ∇ SB L ≈ 0, ∆ x ≈ 0, ∆ f ≈ 0, andua > 0
(Lagrange mult. for active are positive; this checks active set)
MINOS
1. No intermediate point is guaranteed to be feasible;
optimum is where ∇ L = 0 and [∇ 2L]reduced is positive
definite.
2. Choice of xB is very important; we want an
invertible and “well conditioned” inverse basis.
Iterations and heuristics are required.
3. The size of the reduced space changes as constraints
are encountered. Must change the size of gradient
and hessian!
MINOS
4. The algorithm does not force exact solution of h=0 at every

iteration, but, we don’t want deviations “too large”. We
“augment” the merit function with a penalty for the
infeasibilities!
L( x, λ ) = f ( x ) + ∑ λ j h j ( x ) + 1 / 2 ρ hT ( x )h( x )
j
At optimum, hj(x)=0 and λ j≠ 0 (for active i) and the

merit function becomes f(x). The penalty is large away
from the solution and is small near the solution. This
speeds approach toward feasibility and prevents severe
distortion near solution.
MINOS
Good Features Poor Features

1. Works in reduced space of 1. Convergence can require
low dimension good initial variable values.
2. Does not require an initial 2. Numerical methods not
feasible solution. suitable for very large
problems.
3. Takes advantage of linear
constraints. 3. Feasibility only at solution.
4. Tends to function well for
wide range of problems.
CURRENT STATUS
• Generally, much faster and more reliable convergence than

first order.
• Required for optimization of “first principles” models
• Analytical first derivatives highly recommended
(considerable engineering time if programmed)
• Use within a modelling manager (GAMS, AMPL, etc) highly
recommended
• Often, user must adjust algorithm parameters for best
performance (tuning again!)
• Used in many engineering optimization products
See also, Quadratic Programming and SQP

NLP MultiVAr Constrained

Uploaded by

Copyright:

Available Formats

NLP MultiVAr Constrained

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NLP MultiVAr Constrained

Uploaded by

Copyright:

Available Formats

NON-LINEAR PROGRAMMING

Minimum: f(x*) < f(x*+δ x);

We have been Single-Var. Line search for 1D optimization.

Optimization of $$$ by adjusting XD, XB, Tf, P, …..

Challenging, but I’m ready now!

We’ll learn methods in Multivariable Opt.

Function values First order Second order

Non-continuous Smooth 1st derivatives Smooth 1st and 2nd der.

SIMPLEX for unconstrained

COMPLEX for constrained

COMPLEX for constrained

g(x) >0 1. Use more points in simplex

COMPLEX for constrained GENERALLY NOT SUCCESSFUL

1. We remember that the LP solution will be at

N-L Opt. Problem N-L Opt. converted to LP approximation

What is the basis for the approximation?

Variable x1 Iteration No. 1

Why are the constraint

Are we sure of being

Variable x1 Iteration No. 1

Variable x1 Iteration No. 2

Variable x1 Iteration No. 3

Variable x1 Iteration No. 4

N-L Opt. converted to LP approximation

N-L Opt. converted to LP approximation

If ∆ xk changes sign from ∆ xk-1 , halve the box size

Major steps in Successive Linear Programming algorithm

WHEN TO USE SLP

1. SLP functions well for “mostly linear” problems

• Not available in general purpose modelling systems (GAMS,

SLP - Class Workshop - Pooling

Experience indicates that even min f ( x )

Finding the stationarity after move gives the famous

By using the Lagrangian as the objective function, we take

x2 Every second-order NLP method will

Reduced Gradient min f ( x ) Line Search

Let’s see how we build on unconstrained

We group the variables (x) into two categories. The

xB = the basic variables that are

We apply the “equality constraint” approach of evaluating the

Which are the basic &

dh = (∇ xB h)dxB + (∇ xNB h)dx NB = 0

Solve for xB in terms of the non-basic variables, xNB .

Rearranging, we obtain df/dxNB , which is the

An unconstrained method could use this gradient in

For the distillation tower,

No. of variables = No. XB + No. XSB

Example: Feasible region

See Sofer and Nash, 1996

Example: Feasible region

Example: Feasible region How do we determine

• These decisions are not unique. In theory, many

16 • what can change the

Line Search: This explores along the search direction

Linear search distance

Line Search: Now, the search is in the reduced space of

infeasible • The line search could

• Therefore, the distance

Line Search: Now, the search is in the reduced space

Minimum: f(x) < f(x+δ x);