Introduction QP Methods Final Remarks

Lecture 4: Quadratic Programming

Prof. Marcelo Escobar

PPGEQ – UFRGS
Prof. Jorge Otávio Trierweiler

September 11, 2011

logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Outline

1 Introduction

2 QP Methods
Equality Constrained QP
General QP

3 Final Remarks

logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Basic Concepts

Quadratic Programming

1 T
minimize x Hx + c T x
f (x) :
x 2
subject to h(x) : Ax = b
g(x) : Cx ≤ d
where:
f (x): quadratic objective function
h(x): linear equality constraints
g(x): linear inequality constraints

Sufficient Conditions for Unique Optimum:

Convexity - H must be positive (semi)-definite logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Basic Concepts

Quadratic Programming: Geometry

x*

x*

The solution can be either inside the feasible region or at the border, It
is not a vertex point!
logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Quadratic Programming (QP)

Consider the equality constrained mathematical program:

minimize x T Hx + c T x
x
subject to Ax = b

H: symetric matrix n × n
A: a matrix m × n
x: a vector n × 1

Note: Any square matrix M can be converted in symetric matrix

H = (M + M T )/2 and x T Hx = x T Mx! logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Quadratic Programming (QP)

Consider the equality constrained mathematical program:

1 T
minimize x Hx + x T c
x 2
subject to Ax = b

Lagrangean Function:
1 T
L(x, λ) = x Hx + x T c + (Ax − b)T λ
2
First order necessary conditions (KKT):

∇x L(x, λ) = 0 ∇λ L(x, λ) = 0
logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

KKT System

First order necessary conditions:

H AT x
    
−c
=
A 0 λ b

The system can be rewritten in a form that is useful for computation:

H AT d
    
−c − Hxk
=
A 0 λ b − Axk
where xk+1 = xk + d

Note: The system is called KKT system and the matrix of coefficients
is the KKT Matrix.! logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Numerical Example (qp0.m )

minimize 3x12 + 2x1 x2 + x1 x3 + 2.5x22 + 2x2 x3 + 2x32 − 8x1 − 3x2 − 3x3

x
subject to x1 + x3 = 3
x2 + x3 = 0
In a matricial form:
   
6 2 1 −8    
1 0 1 3
H = 2 5 2 c = −3 A= b=
0 1 1 0
1 2 4 −3
The solution x ∗ and optimal Lagrange multiplier vector λ∗ are given by:
 
2  
∗ ∗ 3
x = −1  λ =
−2 logo
1

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Methods for solving the KKT System

Full Space Method, i.e. direct solution of KKT system

Range-Space Method
Null-Space Method

Note: Exploit the block structure in the KKT system to decouple

it into two smaller systems!
logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Range-Space Method

Perform block elimination on the system. Multiply the first equation by

AH −1 and then subtract the second equation:

(AH −1 AT )λ = (−AH −1 (Hxk + c) + Axk − b)

Solve this symetric system for λ∗ and recover d from the first equation:

Hd = −AT λ∗ − (Hxk + c)

This method is most useful when:

H is well conditioned and easy to invert;
H −1 is known explicitly through a Quasi-Newton Method;
the number of equality constraints m is small.
logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Null-Space Method
Suppose that we partition the vector d into two components:
d = YdY + ZdZ
where Z is the n ×
 (n − m) null-space matrix (AZ = 0), and Y is any n × m
matrix such that Y | Z is nonsingular. By substituting d into the second
equation of the KKT system:
AYdY = −(Axk − b)
From the first equation:
HYdY + HZdZ + AT λ = −(Hxk + c)
multiplty this equation by Z T to obtain:

Z T HZdZ = −Z T HYdY − Z T (Hxk + c)

To obtain the Lagrange multiplier, we multiply the first equation by Y T :
logo
(AY )T λ = Y T (Hxk + c + Hd)

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Null-Space Method: Example (qp0a.m )

Consider the previous numerical example:

   
−1 2 −1
1
Z =  1  Y = −1 2 
3
1 1 1

For this particular choice of Y , we have AY = I . Suppose that we have

x T = 0, 0, 0 . Then:

T
dY = 3, 0 dZ = 0
so that
T
d = YdY + ZdZ = 2, −1, 1
After recovering λ and x:

T
T
x ∗ = 2, −1, 1 λ∗ = 3, −2
logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Equality Constrained QP

Null-Space: Numerical Computation

Find the n × (n − m) matrix Z such that:

AZ = 0
QR Factorization:

AT = Q1
 
Q2 R

where Q1 is n × m and Q2 is n × (n − m)

The columns of Q2 span the null space of A:

Z = Q2 logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

General QP

Inequality constrained Problem

Quadratic Programming:
1
minimize cT x + xT H x
x 2
subject to Cx = d
Ax ≤ b

Methods for general QP:

Classical active-set method (most widely used);
Gradient Projection Method;
Interior Point Method;
logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

General QP

Inequality constrained Problem

Quadratic Programming:
1
minimize cT x + xT H x
x 2
subject to Cx = d
Aj x = bj , j ∈ A(x ∗ )

Lagrangean Function:
1
L(x, λ, µ) = c T x + x T H x + λ(Cx − d) + µ(AA x − bA )
2
Note: A(x ∗ ) is the active set at the optimal point x ∗ which is unknown
in advance! logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

General QP

QP Subproblem

Quadratic Programming Subproblem at Iteration k:

1
minimize (Hxk + c)T dx + dxT H dx
dx 2
subject to Cdx = d − Cxk
Aj dx = bj − Aj xk , j ∈ Wk

where dx = x − xk and Wk is the working set at iteration k.

For a given point xk and the working set Wk solve the KKT system:

H C T AT
    
j dx −Hxk − c
C 0 0   λ  =  d − Cxk 
Aj 0 0 µ b j − A j xk logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

General QP

Line Search
Suppose that dx is nonzero:

xk+1 = xk + αk dx αk ∈ (0, 1]

decide how far to move along this direction retaining feasibility:

ajT (xk + αk dx ) ≤ bj , ∀j ∈
/ Wk

This constraint will be satisfied for all nonengative choices of αk only if:

bj − ajT xk
αk ≤
ajT dx

Since we want αk to be as large as possible in (0, 1]:

 
bj −aT xk
αk = min 1, min aT dj x
j ∈W
/ k j

Note: Blocking constraint(s) for which αk is defined must be added to the logo

working set for the next iteration!

Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks

General QP

Active-Set Method for Convex QP: Algorithm

0. Compute a feasible starting point x0 , Set W0 to be a subset of the active
constraints at x0 ;
1. Solve QP subproblem to find dx
1 T
minimize (Hxk + c)T dx + d H dx
dx 2 x
subject to Cdx = d − Cxk
aj dx = bj − aj xk , j ∈ Wk

2. If dx = 0, compute Lagrange Multipliers

If µ ≥ 0, STOP with solution x ∗ = xk ;
else drop one constraint with wrong sign (µ < 0)
xk+1 = xk ; Wk+1 ← Wk − j;
3. If dx 6= 0, Compute αk and xk+1 ← xk + αk dx ;
If there are blocking constraints, add one blocking constraint to Wk
else Wk+1 ← Wk ; logo

4. Set k = k + 1 and Go to 2
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks

General QP

Active-Set Method: Example 01

minimize (x1 − 1)2 + (x2 − 2.5)2

x1 ,x2

subject to g1 (x) : x1 − 2x2 + 2 ≥ 0

g2 (x) : −x1 − 2x2 + 6 ≥ 0
g3 (x) : −x1 + 2x2 + 2 ≥ 0
g4 (x) : x1 ≥ 0
g5 (x) : x2 ≥ 0

Suppose that we choose x0 = (2, 0). Constraints 3 and 5 are active at this
point, and we set W0 = 3, 5 .

 
Note: We could have chosen W0 = 3 , or W0 = 5 or even W0 = ∅, each logo
would lead the algorithm to perform quite different.
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks

General QP

Active-Set Method: Example 01 (qp01.m )

Iteration 0: dx = (0, 0)T and (µ∗3 , µ∗5 ) = (−2, −1)

Remove the constraint 3 from the active set. Set W1 = 5

Iteration 1: dx = (−1, 0)T , α1 = 1, then x2 = (1, 0)T


There is no blocking constraint, thenW2 = W1 = 5

Iteration 2: dx = (0, 0)T , and µ∗5 = −5

Remove the constraint 5 from the active set. Set W3 = ∅

Iteration 3: dx = (0, 2.5)T , and α3 = 0.6, then x4 = (1, 1.5)T


Add the blocking constraint 1 to the active set. Set W4 = 1

Iteration 4: dx = (0.4, 0.2)T , and α3 = 1, then x5 = (1.4, 1.7)T


There is no blocking constraint, thenW5 = W4 = 1 .

Iteration 5: dx = (0, 0)T , and µ∗1 = 1.25

logo
This is the SOLUTION!!!

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

General QP

Active-Set Method: Example 02

minimize x12 + x22 − 4x1 − 4x2

x1 ,x2

subject to g1 (x) : −2x1 + x2 + 2 ≥ 0

g2 (x) : −x1 + x2 + 1 ≥ 0
g3 (x) : x1 + x2 + 1 ≥ 0
g4 (x) : 2x1 − x2 + 2 ≥ 0

Suppose that we choose x0 = (−1, 0). Constraints 3 and 4 are active at this
point, and we set W0 = 3, 4 .

logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

General QP

Active-Set Method: Example 02 (qp02.m )

Iteration 0: dx = (0, 0)T and (µ∗3 , µ∗4 ) = 31 (−14, −2)


Remove the constraint 3 from the active set. Set W1 = 4

Iteration 1: dx = (1.4, 2.8)T , α1 = 0.71, then x2 = (0, 2)T


Add the blocking constraint 1, then W2 = 1, 4

Iteration 3: dx = (0, 0)T , and (µ∗1 , µ∗4 ) = (1, −1)


Remove the constraint 4 from the active set. Set W4 = 1

Iteration 4: dx = (0.4, −0.8)T , and α4 = 1, then x5 = (0.4, 1.2)T


There is no blocking constraint, thenW5 = W4 = 1 .

Iteration 5: dx = (0, 0)T , and µ∗1 = 1.6

This is the SOLUTION!!!
logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

General QP

Final Remarks: Active Set Method

Different initial working set W0 leads to a different iteration

sequence;
Different choices of the blocking constraint to remove leads to a
different iteration sequence;
Selecting the constraints with the most negative µ always works
well in practice;
The working set changes slowly along iterations;
This algorithm has a finite termination!

logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Final Remarks

Final Remarks

For indefinite QP the direction must be corrected as a descent

direction (direction of negative curvature) in order to avoid
convergence to a saddle point
The Gradient Projection Method is based on variable elimination.
It turns out to be Reduced Gradient Method and will be
presented for general NLP;
The Interior Point Method will be presented for general NLP;
Main Application in Chemical Engineering: Model Predictive logo
Control.
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks

Final Remarks

Further Readings

Numerical Optimization. Nocedal

and Wright (1999): Chapter 16.

logo

Prof. Marcelo Escobar Optimization of Chemical Processes

Introduction QP Methods Final Remarks

Final Remarks

Help, Comments, Suggestions

logo

Prof. Marcelo Escobar Optimization of Chemical Processes

This presentation is over

Thank You for your attention!!!!

What you want to do now:
restart presentation

quit