Chapter 5.

Linear and Non-linear

Programming

1
 Why Talk About Linear Programming?

• LP is simpler than NLP, hence, good for a

foundation
• Linearity has some unique features for
optimization
• A lot of problems are or can be converted to a LP
formulation
• Some NLP algorithms are based upon LP simplex
method

2
 Simplex Method

• A large variety of Simplex-based algorithms exist to solve

LP problems.
• Other (polynomial time) algorithms have been developed
for solving LP problems:
– Khachian algorithm (1979)
– Kamarkar algorithm (AT&T Bell Labs, mid 80s)
BUT,
none of these algorithms have been able to beat Simplex
in actual practical applications.
HENCE,
Simplex (in its various forms) is and will most likely
remain the most dominant LP algorithm for at least the
near future
3
 Fundamental Theorem
• Extreme point (or Simplex filter) theorem:
If the maximum or minimum value of a linear
function defined over a polygonal convex region
exists, then it is to be found at the boundary of the
region.
• Convex set:
• A set (or region) is convex if, for any two points
(say, x1 and x2) in that set, the line segment
joining these points lies entirely within the set.
• A point is by definition convex.
 What does the extreme point theorem imply?

• A finite number of extreme points implies a

finite number of solutions
• Hence, search is reduced to a finite set of
points
• However, a finite set can still be too large for
practical purposes
• Simplex method provides an efficient
systematic search guaranteed to converge in a
finite number of steps.

5
 Basic Steps of Simplex

1. Begin the search at an extreme point (i.e., a

basic feasible solution).
2. Determine if the movement to an adjacent
extreme can improve on the optimization of
the objective function. If not, the current
solution is optimal. If, however, improvement
is possible, then proceed to the next step.
3. Move to the adjacent extreme point which
offers (or, perhaps, appears to offer) the most
improvement in the objective function.
4. Continue steps 2 and 3 until the optimal
solution is found or it can be shown that the
problem is either unbounded or infeasible.
6
 Step 0 – Obtain Canonical Form

IMPORTANT: Simplex only deals with equalities

General Simplex LP model:
min (or max) z =  ci xi
s.t.
Ax=b
x0
In order to get and maintain this form, use
• slack, if x  b, then x + slack = b
• surplus, if x  b, then x - surplus = b
• artificial variables (sometimes need to be added to ensure all
variables  0, see page 101)

Compare
Compare constraint
constraint conversion
conversion withwith goal
goal
conversions
conversionsusing
usingdeviation
deviationvariables
variables
7
 Different "components" of a LP model

• LP model can always be split into a basic and a

non-basic part.
• “Transformed” or “reduced” model is another
good way to show this.
• This can be represented in mathematical terms
as well as in a LP or simplex tableau.

8
 Movement to Adjacent Extreme Point

Given any basis we move to an adjacent

extreme point (another basic feasible
solution) of the solution space by
exchanging one of the columns that is in
the basis for a column that is not in the
basis.

Two things to determine:

1) which (nonbasic) column of A should be brought
into the basis so that the solution improves?
2) which column can be removed from the basis
such that the solution stays feasible?

9
Entering and Departing Vector (Variable) Rules

General rules:
• The one non-basic variable to come in is
the one which provides the highest
reduction in the objective function.
• The one basic variable to leave is the one
which is expected to go infeasible first.
NOTE: THESE ARE HEURISTICS!!
Variations on these rules exist, but are rare.

10
 Simplex Variations

Various variations on the simplex method

exist:
• "regular" simplex
• two-phase method: Phase I for feasibility
and Phase II for optimality
• condensed/reduced/revised method: only
use the non-basic columns to work with
• (revised) dual simplex

11
 Computational Considerations

• Unrestricted variables (unboundedness)

• Redundancy (linear dependency,
modeling errors)
• Degeneracy (some basic variables = 0)
• Round-off errors

12
 Limitations of Simplex

1. Inability to deal with multiple objectives

2. Inability to handle problems with integer variables
Problem 1 is solved using Multiplex
Problem 2 has resulted in:
• Cutting plane algorithms (Gomory, 1958)
• Branch and Bound (Land and Doig, 1960)
However,
solution methods to LP problems with integer or
Boolean variables are still far less efficient than those
which include continuous variables only

13
 Example Problem

Maximize Z = 5x1 + 2x2 + x3

subject to
x1 + 3x2 - x3 ≤ 6,

x2 + x3 ≤ 4,

3x1 + x2 ≤ 7,

x1, x2, x3 ≥ 0.

14
 Simplex and Example Problem

Step 1. Convert to Standard Form

a11 x1 + a12 x2 + ••• + a1n xn ≤ b1, a11 x1 + a12 x2 + ••• + a1n xn + xn+1 = b1,

a21 x1 + a22 x2 + ••• + a2n xn ≥ b2, a21 x1 + a22 x2 + ••• + a2n xn - xn+2 = b2,

am1 x1 + am2 x2 + ••• + amn xn ≤ bm, am1 x1 + am2 x2 + ••• + amn xn + xn+k = bm,
In our example problem:

x1 + 3x2 - x3 ≤ 6, x1 + 3x2 - x3 + x4 = 6,

x2 + x3 ≤ 4, x2 + x 3 + x5 = 4,

3x1 + x2 ≤ 7, 3x1 + x2 + x6 = 7,

x1, x2, x3 ≥ 0. x1, x2, x3, x4, x5, x6 ≥ 0.

15
 Simplex: Step 2

Step 2. Start with an initial basic feasible solution

(b.f.s.) and set up the initial tableau.
In our example
Maximize Z = 5x1 + 2x2 + x3

x1 + 3x2 - x3 + x4 = 6,

x2 + x3 + x5 = 4, c
BBas
i
s c
j Co
n
st
an
ts
521 000
3x1 + x2 + x6 = 7, x1x
2x
3x4x
5x
6
0x 41 3-
1 100 6
0x 50 11 010 4
x1, x2, x3, x4, x5, x6 ≥ 0. 0x 63 10 001 7
cr
ow 521 000Z=0

16
 Step 2: Explanation

Adjacent Basic Feasible Solution

If we bring a nonbasic variable xs into the basis, our system
changes from the basis, xb, to the following (same notation as the
x1 + ā1sxs= b1 xi = b i  a is for i =1, …, m
book):
xr + ā rsxr= b r xs = 1
xj = 0 for j=m+1, ..., n and js
b
xm + āmsxs= s
The new value of the objective function becomes:
m
Z  c (b  a
i 1
i i is )  c s

Thus the change in the value of Z per unit increase in xs is

cs = new
m value of Z - old mvalue of Z
=  c (b  a
i 1
i
m
i is )  c s  c b
i 1
i i
This is the Inner Product rule
= c  c a
s i is
i 1
17
 Simplex: Step 3

Use the inner product rule to find the relative profit coefficients
c
BBas
i
s c
j Co
n
st
an
ts
521 000
x1x
2x
3x4x
5x
6
0x 41 3-
1 100 6
0x 50 11 010 4
0x 63 10 001 7
cr
ow 521 000Z=0

c j c j  cB Pj
c1 = 5 - 0(1) - 0(0) - 0(3) = 5 -> largest positive
c2 = ….
c3 = ….
Step 4: Is this an optimal basic feasible solution?

18
 Simplex: Step 5

Apply the minimum ratio rule to determine the basic variable

to leave the basis.
The new values of the basis variables:
xi = b i  a is x s for i = 1, ..., m

 bi 
max x s min  
a is 0 a is
 
In our example:
c
BBas
i
s c
j Co
n
st
an
ts
521000 Row Basic Variable Ratio
x1x
2x
3x4x
5x
6 1 x4 6
0x 41 3-
1 100 6 2 x5 -
0x 50 11010 4 3 x6 7/3
0x 63 10001 7
cr
ow521000 Z =0

19
 Simplex: Step 6

Perform the pivot operation to get the new tableau and the b.f.s.
c
BBas
i
s c
j Co
n
st
an
ts
521000
x1x
2x
3x4x
5x
6
0x 41 3-
1 100 6
0x 50 11010 4
0x 63 10001 7
cr
o
w521000 Z =0

New iteration:
find entering c
BBas
i
s c
j C
o
n
st
an
ts
variable: 521000
c j c j  c B Pj x1x
2x3x4x
5x6
0x 40 8
/
3 -
1 100 11/
3
0x 50 11010 4
cB = (0 0 5) 5x 11 1
/
3 0001/37/3
c2 = 2 - (0) 8/3 - (0) 1 - (5) 1/3 = 1/3 cr
o
w0 1
/
3 100 -
5/
3 Z
=
35
/3
c3 = 1 - (0) (-1) - (0) 1 - (5) 0 = 1
c6 = 0 - (0) 0 - (0) 0 - (5) 1/3 = -5/3
20
 Final Tableau

c
BBas
i
s c
j C
o
n
st
an
ts x3 enters basis,
521000
x5 leaves basis
x1x
2x3x4x
5x6
0x 40 8
/
3 -
1 100 11/
3
0x 50 11010 4
5x 11 1
/
3 0001/37/3
cr
o
w0 1
/
3 100 -
5/
3 Z
=
35
/3

Wrong value!
4 should be 11/3 c
BBas
i
s c
j Co
n
st
an
ts
521000
x
1x2x
3x4x
5x 6
0x 4 040110 2 3/
3
1x 3 011010 4
5x 1 11
/3000 1
/37/3
cr
ow0 -
2
/300 -
1 -
5
/3 Z
=
47
/3

21
5.2.Nonlinear Programming
 Introduction
• This part of the course deals with techniques that are applicable to the
solution of the constrained optimization problem:

• There are many techniques available for the solution of a constrained

nonlinear programming problem. All the methods can be classified into
two broad categories: direct methods and indirect methods.

• In the direct methods, the constraints are handled in an explicit manner,

whereas in most of the indirect methods, the constrained problem is solved
as a sequence of unconstrained minimization problems.
Characteristic of a constrained problem
 Characteristic of a constrained problem
In the presence of the constraints, an optimization
problem may have the following features:

• The constrainst may have no effect on the

optimum point; that is, the constrained minimum
is the same as the unconstrained minimum as
shown in the figure. In this case, the minimum
point X* can be found by making use of the
necessary and sufficient conditions as follows:

• However to use these conditions one must be

certain that the constraints are not going to have
any effect on the minimum.
 Characteristic of a constrained problem
• The optimum, unique solution occurs on a constraint boundary as shown.
In this case, the Kuhn-Tucker necessary conditions indicate that the
negative of the gradient must be expressible as a positive linear
combination of the gradients of the active constraints.
 Characteristic of a constrained problem
• If the objective function has two or more unconstrained local minima, the
constrained problem may have multiple minima as shown in the figure.
 Characteristic of a constrained problem
• In some cases, even if the objective function has a single unconstrained minimum,
the constraints may introduce multiple local minima as shown in the figure.
• A constrained optimization technique must be able to locate the minimum in all the
situations outlined above.
 Direct Methods
RANDOM SEARCH METHODS

The random search methods described for unconstrained minimization can be used with
minor modifications to solve a constrained optimization problem. The basic procedure can
be described by the following steps:

1. Generate a trial design vector using one random number for each design variable.

2. Verify whether the constraints are satisfied at the trial design vector. Usually, the equality
constraints are considered satisfactory whenever their magnitudes lie within a specified
tolerance. If any constraint is violated, continue generating new trial vectors until a trial
vector that satisfies all the constraints is found.

3. If all the constraints are satisfied, retain the current trial vector as the best design if it gives
a reduced objective function value compared to the previous best available design.
Otherwise, discard the current feasible trial vector and proceed to step 1 to generate a new
trial design vector.

4. The best design available at the end of generating a specified maximum number of trial
design vectors is taken as the solution of the constrained optimization problem.
 Direct Methods
RANDOM SEARCH METHODS
• It can be seen that several modifications can be made to the basic
procedure indicated above. For example, after finding a feasible trial
design vector, a feasible direction can be generated (using random
numbers) and a one-dimensional search can be conducted along the
feasible direction to find an improved feasible design vector.
• Another procedure involves constructing an unconstrained function,
F(X), by adding penalty for violating any constraint as:

where

indicate the squares of violations of inequality and equality constraints,

respectively, and a and b are constants.
 Direct Methods
RANDOM SEARCH METHODS
• The equation

indicates that while minimizing the objective function f (X), a positive penalty is
added whenever a constraint is violated, the penalty being proportional to the
square of the amount of violation. The values of the constants a and b can be
adjusted to change the contributions of the penalty terms relative to the
magnitude of the objective function.

• Note that the random search methods are not efficient compared to the other
methods described in this chapter. However, they are very simple to program
and are usually reliable in finding a nearly optimal solution with a sufficiently
large number of trial vectors. Also, these methods can find near global optimal
solution even when the feasible region is nonconvex.
 Direct Methods
SEQUENTIAL LINEAR PROGRAMMING

• In the sequential linear programming (SLP) method, the solution of the original
nonlinear programming problem is found by solving a series of linear programming
problems.

• Each LP problem is generated by approximating the nonlinear objective and

constraint functions using first-order Taylor series expansions about the current
design vector Xi.

• The resulting LP problem is solved using the simplex method to find the new
design vector Xi+1.

• If Xi+1 does not satisfy the stated convergence criteria, the problem is relinearized
about the point Xi+1 and the procedure is continued until the optimum solution X* is
found.
 Direct Methods
SEQUENTIAL LINEAR PROGRAMMING

• If the problem is a convex programming problem, the linearized

constraints always lie entirely outseide the deasible region. Hence the
optimum solution of the approximating LP problem, which lies at a vertex
of the new feasible region, will lie outside the original feasible region.

• However, by relinearizing the problem about the new point and repeating
the process, we can achieve convergence to the solution of the original
problem in a few iterations.

• The SLP method is also known as the cutting plane method.

 Direct methods
SEQUENTIAL LINEAR PROGRAMMING
1. Start with an initial point X1 and set the iteration number as i = 1. The
point X1 need not be feasible.

2. Linearize the objective and constraint functions about the point Xi as

 Direct methods
SEQUENTIAL LINEAR PROGRAMMING
3. Formulate the approximating linear programming problem as:

Notice that the LP problem in the above equation may sometimes have an
unbounded solution. This can be avoided by formulating the first
approximating LP problem by considering only the following constraints:

li  xi  ui i=1,2,...,n
li and ui represent the lower and upper bounds on xi , respectively.
 Direct methods
SEQUENTIAL LINEAR PROGRAMMING
4. Solve the approximating LP problem to obtain the
solution vector Xi+1

5. Evaluate the original constraints at Xi+1; that is, find

If gj (Xi+1)ϵ for j = 1,2,...,m, and |hk (Xi+1) |> ϵ for

some k, find the most violated constraint, for
example, as
 Direct methods
SEQUENTIAL LINEAR PROGRAMMING

Relinearize the constraint gk (X)0 about the point Xi+1 as

and add this as the (m+1)th inequality constraint to the

previous LP problem.

6. Set the new iteration number as i=i+1, the total number of

constraints in the new approximating LP problem as m+1
inequalities and p equalities, and go to step 4.
 Direct methods
The sequential linear programming method has several advantages:
1. It is an efficient technique for solving complex programming problems
with nearly linear objective functions.

2. Each of the approximating problems will be a LP problem and hence

can be solved quite efficiently. Moreover, any two consecutive
approximating LP problems differ by only one constraint, and hence the
dual simplex method can be used to solve the sequence of
approximating LP problems much more efficiently.

3. The method can easily be extended to solve integer programming

problems. In this case, one integer LP problem has to be solved in each
stage.
 Direct methods
Geometric Interpretation of the
Method:

The SLP method can be illustrated with

the help of a one-variable problem:

Minimize f (x) = c1x

subject to g(x)  0

where c1 is a constant and g (x) is a

nonlinear function of x.
Let the feasible region and the contour of
the objective function be as shown in the
figure:
 Direct methods
Geometric Interpretation of the
Method:

To avoid any possibility of an unbounded

solution, let us first take the constraints
on x as:
cxd

where c and d represent the lower and

upper bounds on x. With these
constraints, we formulate the LP
problem:

Minimize f (x) = c1x

subject to c  x  d
 Sequential Linear Programming Problem
• The optimum solution of this approximating
LP problem can be seen to be x* = c. Next, we
linearize the constraint g(x) about point c and
add it to the previous constraint set. Thus, the
new LP problem becomes:
Minimize f (x) = c1x
subject to c  x  d
dg
g (c )  (c)( x  c) 0
dx

The feasible region of x, according to the above

two constraints, is given by e  x  d as
shown in the figure. The optimum solution of
the approximating LP problem given by the
above equations can be seen to be x* = e.
Next, we linearize the constraint g(x)  0
about the current solution x=e* and add it to
the previous constraint set to obtain the next
approximating LP problem as:
Sequential Linear Programming Problem
Minimize f (x) = c1x
subject to cxd
dg
g (c )  (c)( x  c) 0
dx
dg
g (e)  (e)( x  e) 0
dx
The permissable range of x, according
to the above constraints can be seen
to be f  x  d from the figure. The
optimum solution of the LP problem
of the above equations can be
obtained as x*=f.
 Sequential Linear Programming
• We then linearize g (x)  0 about the present point x*=f and add it to the
previous constraint set to define a new approximating LP problem. This
procedure has to be continued until the optimum solution is found to the
desired level of accuracy.

• As can be seen from the figures, the optimum of all the approximating LP
problems (e.g., points c,e,f,...) lie outside the feasible region and converge
toward the optimum point, x = a.

• The process is assumed to have converged whenever the solution of an

approximating problem satisfies the original constraint within some
specified tolerance level as
g(xk*)  ϵ
where ϵ is a small positive number and xk* is the optimum solution of the
kth approximating LP problem.
 Sequential Linear Programming
• It can be seen that the lines (hyperplanes in a general problem) defined by
g (xk*)+dg/dx (xk*)(x-xk*) cut off a portion of the existing feasible region.
Hence this method is called the cutting plane method.

Example: Minimize f(x1,x2)=x1 - x2

Subject to g(x1,x2)=3x12 - 2x1 x2+ x22-1  0
using the cutting plane method. Take the convergence limit in step 5 as
ϵ = 0.02.

Note: This example was originally given by Kelly. Since the constraint
boundary represents an ellipse, the problem is a convex programming
problem. From graphical representation, the optimum solution of the
problem can be identified as x1* = 0, x2* = 1, and f min= -1
 Sequential Linear Programming
Steps 1, 2, 3:Although we can start the solution from any initial point X1, to
avoid the possible unbounded solution, we first take the bounds on x1 and
x2 as
-2  x1  2
-2  x2  2

And solve the following LP problem:

 Sequential Linear Programming
The solution of the problem can be obtained as:
 - 2
X    with f ( X)  4
 2
Step 4: Since we have solved one LP problem, we can take

Step 5: Since g1(X2) = 23 > ϵ, we linearize g1(X) about point X2 as

 Sequential Linear Programming
As

The equation

becomes
 Sequential Linear Programming
• By adding this constraint to the previous LP problem, the new LP problem
becomes:

• Step 6: Set the iteration number as i = 2 and go to step 4.

• Step 7: Solve the approximating LP problem stated in the above equation
and obtain the solution
 Sequential Linear Programming
• This procedure is continued until the specified convergence criterion,
g1 (Xi)  ϵ, in step 5 is satisfied. The computational results are
summarized in the table.
 Sequential Quadratic Programming
• The sequential quadratic programming is one of the
most recently developed and perhaps one of the
best methods of optimization.

• The method has a theoretical basis that is related to

1. The solution of a set of nonlinear equations using

Newton’s method.

2. The derivation of simultaneous nonlinear equations using

Kuhn-Tucker conditions to the Lagrangian of the
constrained optimization problem.
 Sequential Quadratic Programming
• Consider a nonlinear optimization problem with only equality constraints
as:

• The extension to include inequality constraints will be considered at a later

stage. The Lagrange function, L (X,), corresponding to the problem of the
above equation is given by:

where k is the Lagrange multiplier for the kth equality constraint.

 Sequential Quadratic Programming
• The Kuhn-Tucker necessary consitions can be stated as:

where [A] is an n x p matrix whose kth column denotes the gradient of the
function hk. The above equations represent a set of n+p nonlinear
equations in n+p unknowns (xi, i=1,...,n and k, k=1,...,p ). These
nonlinear equations can be solved using Newton’s method. For
convenience, we rewrite the above equations as:
 Sequential Quadratic Programming
• According to the Newton’s method, the solution of the above equation can
be found as:

where Yj is the solution at the start of the jth equation and ∆Yj is the change
in Yj necessary to generate the improved solution, Yj+1, and [F]j=
[F(Yj)]j is the (n+p) x (n+p) Jacobian matrix of the nonlinear equations
whose ith column denotes the gradient of the function Fi (Y) with respect
to the vector Y.
 Sequential Quadratic Programming
• By substituting

into

we obtain:
Sequential Quadratic Programming Method
where

denotes the Hessian matrix of the Lagrange function. The first set of
equations in

can be written separately as:

 Sequential Quadratic Programming Method
• The equation

and the second set of equations in the equation

can now be combined as:

The above equation can be solved to find the change in the design vector
∆Xj and the new values of the Lagrange multipliers, j+1.The iterative
process indicated by the above equation can be continued until
convergence is achieved.
 Sequential Quadratic Programming Method
• Now consider the following quadratic programming problem:
• Find ∆X that minimizes the quadratic objective function

subject to the linear equality constraints

• The Lagrange function L corresponding to the above problem is given by:

where k is the Lagrange multiplier associated with the kth equality

constraint.
 Sequential Quadratic Programming Method

• The Kuhn-Tucker necessary conditions can be stated as:

• The above equations can be identified to be same as

in matrix form.
 Sequential Quadratic Programming Method
• This shows that the original problem of the equation

can be solved iteratively by solving the quadratic programming problem

defined by the equation

• In fact, when inequality constraints are added to the original problem, the
quadratic programming problem of the above equation becomes:
Sequential Quadratic Programming Method

with the Lagrange function given by:

• Since the minimum of the Augmented Lagrange function is involved, the

sequential quadratic programming method is also known as the projected
Lagrangian method.
Sequential Quadratic Programming Method
Solution Procedure
As in the case of Newton’s method of unconstrained minimization, the
solution vector ∆X in the equation

is treated as the search direction S, and the quadratic programming

subproblem (in terms of the design vector S) is restated as:
Sequential Quadratic Programming Method
where [H] is a positive definite matrix that is taken initially as the identity
matrix and is updated in subsequent iterations so as to converge to the
Hessian matrix of the Lagrange function of the equation:

and

are constants used to ensure that the linearized constraints do not cut off the
feasible space completely. Typical values of these constants are given by:
Sequential Quadratic Programming Method
• The subproblem of the equation

is a quadratic programming problem and hence methods for minimizing

quadratic functions can be used for their solution. Alternatively, the
problem can be solved by any of the methods described in this lecture
since the gradients of the function involved can be evaluated easily. Since
the Lagrange multipliers associated with the solution of the above problem,
are needed, they can be evaluated:
Sequential Quadratic Programming Method
 Sequential Quadratic Programming Method
• Once the search direction, S, is found by solving the problem in the
equation

the design vector is updated as:

where * is the optimal step length along the direction S found by

minimizing the function (using an exterior penalty function approach):
Sequential Quadratic Programming Method

• The one-dimensional step length * can be found by any of the methods

discussed before for one-dimensional minimization.
• Once Xj+1 is found from the equation , for the next
iteration of the Hessian matrix [H] is updated to improve the quadratic
approximation in the equation
Sequential Quadratic Programming Method
• Usually, a modified BFGS formula, given below is used for this purpose:

where is given by

and the constants 0.2 and 0.8 in the above equation can be changed, based
on numerical experience.
Sequential Quadratic Programming Method
Example 1: Find the solution of the problem

using the sequential quadratic programming technique

Sequential Quadratic Programming Method
- Example 1
Solution: Let the starting point be:

With g1(X1)= g3(X1)=0, g2(X1)=-5.8765, and f (X1)=1.5917. The gradients of

the objective and constraint functions at X1 are given by:
Sequential Quadratic Programming Method
- Example 1
Solution:
We assume the matrix [H1] to be the identity matrix and hence the
objective function of the equation

becomes
Sequential Quadratic Programming Method
- Example 1
Solution:
Equation

gives 1=3=0 since g1= g3=0 and 2=1.0 since g2<0, and hence the
constraints of the equation
Sequential Quadratic Programming Method
- Example 1
Solution:
can be expressed as:

We solve this quadratic programming problem directly with the use of the
Kuhn-Tucker conditions which are given by:
Sequential Quadratic Programming Method
- Example 1
• The equations

can be expressed in this case as:

• By considering all possibilities of active constraints, we find that the

optimum solution of the quadratic programming problem is given by
Sequential Quadratic Programming Method
- Example 1
• The new design vector X can be expressed as:

where  can be found by minimizing the function  in equation

Sequential Quadratic Programming Method
- Example 1

• By using the quadratic interpolation technique (unrestricted search method

can also be used for simplicity), we find that  attains its minimum value
of 1.48 at *=64.93, which corresponds to the new design vector

with f (X2)=1.38874 and g1 (X2)=0.0074932 (violated slightly)

Sequential Quadratic Programming Method
- Example 1
• Next we update the matrix [H] using

with
Sequential Quadratic Programming Method
- Example 1
Sequential Quadratic Programming Method
- Example 1
• We can now start another iteration by defining a new quadratic
programming problem using

and continue the procedure until the optimum solution is found. Note that
the objective function reduced from a value of 1.5917 to 1.38874 in one
iteration when X changed from X1 to X2.
 Indirect methods
TRANSFORMATION TECHNIQUES
• If the constraints gj (X) are explicit functions of the variables xi and have
certain simple forms, it may be possible to make a transformation of the
independent variables such that the constraints are satisfied
automatically.

• Thus, it may be possible to convert a constrained optimization problem

into an unconstrained one by making change of variables. Some typical
transformations are indicated below:
1. If lower and upper bounds are satisfied as

these can be satisfied by transforming the variable xi as:

where yi is the new varible, which can take any value.

 Indirect methods
TRANSFORMATION TECHNIQUES
2. If a variable xi is restricted to lie in the interval (0,1), we can use the
transformation:

3. If the variable xi is constrained to take only positive values, the

transformation can be:
 Indirect methods
TRANSFORMATION TECHNIQUES
4. If the variable is restricted to take values lying only in between -1 and 1,
the transformation can be

Note the following aspects of the transformation techniques:

5. The constraints gj(X) have to be very simple functions of xi.
6. For certain constraints, it may not be possible to find the necessary
transformation.
7. If it is not possible to eliminate all the constraints by making change of
variables, it may be better not to use the transformation at all. The partial
transformation may sometimes produce a distorted objective function
which might be more difficult to minimize than the original function.
 Indirect methods-Example 1
Find the dimensions of a rectangular prism type box that has the largest
volume when the sum of its length, width and height is limited to a
maximum value of 60 in, and its length is restricted to a maximum value of
36 in.
Solution: Let x1, x2, and x3 denote the length, width, and height of the box,
respectively. The problem can be stated as follows:
 Indirect methods-Example 1
Solution: By introducing the new variables as:

The constraints of the equations (E2) to (E4) can be restated as:

where the upper bound, for example, on y2 is obtained by setting x1=x3=0 in

the equation E2. The constraints of the equation E7 will be satisfied
automatically if we define new variables zi, i=1,2,3, as
 Indirect methods-Example 1
• Thus the problem can be stated as an unconstrained problem as follows:

• The necessary conditions of optimality yield the relations:

 Indirect methods
• Equation E12 gives the nontrivial solution as cos z3=0 or sin2z3=1. Hence
the equations E10 and E11 yield sin2z1=5/9 and sin2z2=1/3.

• Thus the optimal solution is given by x1*=20 in., x2*=20 in., x3*=20 in.,
and the maximum volume =8000 in3.

BASIC APPROACH OF THE PENALTY FUNCTION METHOD

• Penalty function methods transform the basic optimization problem into

alternative formulations such that numerical solutions are sought by
solving a sequence of unconstrained minimization problems.
 Penalty function method

• This problem is converted into an unconstrained minimization problem

by constructing a function of the form:

where Gj is some function of the constraint gj, and rk is a positive

constant known as the penalty parameter.
• If the unconstrained minimization of the  function is repeated for a
sequence of values of the penalty parameter rk (k=1,2,...), the solution
may be brough to converge to that of the original problem stated in the
first equation in this slide. This is the reason why the penalty function
methods are also known as sequential unconstrained minimization
techniques (SUMTs).
 Penalty function method
• The penalty function formulations for inequality constrained problems can
be divided into two categories: interior and exterior methods. In the
inetraior formulations, some popularly used forms of Gj are given by:

• Some commonly used forms of the function Gj in the case of exterior

penalty function formulations are
 Penalty function method
• In the interior methods, the unconstrained minima of k all lie in the
feasible region and converge to the solution of

as rk is varied in a particular manner.

• In the exterior methods, the unconstrained minima of k all lie in the
infeasible region and converge to the desired solution from the outside as rk
is changed in a specified manner.
 Penalty function method
• In the exterior methods, the unconstrained minima of k all lie in the infeasible
region and converge to the desired solution from the outside as rk is changed in a
specified manner. The convergence of the unconstrained minima of k is illustrated in
the figure for the simple problem:
subject to
 Penalty function method
• It can be seen from the figure that the unconstrained minima of (X,rk)
converge to the optimum point X* as the parameter rk is increased
sequentially.
 Penalty function method
• The interior method shown in the figure below gives convergence as the
parameter rk is decreased sequentially.
 Penalty function method
• There are several reasons for the appeal of the penalty function
formulations. One main reason, which can be observed from the figures in
the previous slides is that the sequential nature of the method allows a
gradual or sequential approach to criticality of the constraints.

• In addition, the sequential process permits a graded approximation to be

used in the analysis of the system. This means that if the evaluation of f
and gj [and hence  (X,rk) ] for any specified design vector X is
computationally very difficult, we can use coarse approximations during
the early stages of optimization (when the unconstrained minima of k are
far away from the optimum) and finer or more detailed analysis
approximation during the final stages of optimization.
 Interior Penalty Function Method
• In the interior penalty function
methods, a new function (
function) is constructed by
augmenting a penalty term to the
objective function.
• The penalty term is chosen such
that its value will be small at
points away from the constraint
boundaries and will tend to
infinity as the constraint
boundaries are approached.
Hence the value of the  function
also blows up as the constraint
boundaries are approached. This
behaviour can also be seen from
the figure.
 Interior Penalty Function Method
• Thus once the unconstrained
minimization of  (X, rk) is started
from any feasible point X1, the
subsequent points generated will
always lie within the feasible
domain since the constraint
boundaries act as barriers during
the minimization process. This is
why the interior penalty function
methods are also known as
barrier methods.

• The  function defined originally

by Carroll is
 Interior Penalty Function Method
• It can be seen that the value of the function  will always be greater than f
since gj (X) is negative for all feasible points X. If any constraint gj (X) is
satisfied critically (with equality sign), the value of  tends to infinity. It is
to be noted that the penalty term in the equation below is not defined if X
is infeasible. This introduces serious shortcomings while using the below
equation:

• Since the above equation does not allow any constraint to be violated, it
requires a feasible starting point for the search toward the optimum point.
However, in many engineering problems, it may not be very difficult to
find a point satisfying all the constraints gj (X) 0, at the expense of large
values of the objective function, f (X).
 Interior Penalty Function Method
• Since the initial point as well as each of the subsequent points generated
in this method lies inside the acceptable region of the design space, the
method is classified as an interior penalty function formulation.

• Since the constraint boundaries act as barriers, the method is also known
as a barrier method.
The iteration procedure of this method can be summarized as follows:

1. Start with an initial feasible point X1 satisfying all the constraints with
strict inequality sign, that is, gj (X1) < 0 for j=1,2,...,m, and an initial
value of r1. Set k =1.
2. Minimize  (X, rk) by using any of the unconstrained minimization
methods and obtain the solution Xk*.
 Interior Penalty Function Method
3. Test whether Xk* is the optimum solution of the original problem. If Xk* is found
to be optimum, terminate the process. Otherwise, go to the next step.
4. Find the value of the next penalty parameter rk+1, as

where c < 1.
5. Set the new value of k=k+1, take the new starting point as X 1=Xk* , and go to
step 2.
Although the algorithm is starightforward, there are a number of points to be
considered in implementing the method. These are:
1. The starting feasible point X may not be readily available in some cases.
6. A suitable value of the initial penalty parameter (r1) has to be found.
7. A proper value has to be selected for the multiplication factor c.
8. Suitable convergence criteria have to be chosen to identify the optimum point.
9. The constraints have to be normalized so that each one of them vary between -1
and 0 only.
 Starting Feasible Point X1
• In most engineering problems, it will not be very difficult to find an initial
point X1 satisfying all the constraints gj (X1) < 0. As an example, consider
the problem of minimum weight design of a beam whose deflection under
a given loading condition has to remain less than or equal to a specified
value. In this case, one can always choose the cross-section of the beam to
be very large initially so that the constraint remains satisfied. The only
problem is that the weight of the beam (objective) corresponding to this
initial design will be very large. Thus, in most of the practical problems,
we will be able to find a feasible starting point at the expense of a large
value of the objective function.

• However, there may be some situations where the feasible design points
could not be found so easily. In such cases, the required feasible starting
points can be found by using the interior penalty function method itself as
follows:
 Starting Feasible Point X1
1. Choose an arbitrary point X1 and evaluate the constraints gj (X) at the
point X1. Since the point X1 is arbitrary, it may not satisfy all the
constraints with strict inequality sign. If r out of the total of m
constraints are violated, renumber the constraints such that the last r
constraints will become the violated ones, that is,

2. Identify the constraint that is violated most at the point X1, that is, find
the integer k such that
 Starting Feasible Point X1
3. Now formulate a new optimization problem as:

subject to

4. Solve the optimization problem formulated in step 3 by taking the point

X1 as a feasible starting point using the interior penalty function method.
Note that this optimization method can be terminated whenever the
value of the objective function gk (X) drops below zero. Thus, the
solution obtained XM will satisfy at least one more constraint than did
the original point X1 .
 Starting Feasible Point X1
5. If all the constraints are not satisfied at the point XM, set the new starting
point as X1= XM, and renumber the constraints such that the last r
constraints will be the unsatisfied ones (this value of r will be different
from the previous value), and go to step 2.

This procedure is repeated until all the constraints are satisfied and a
point X1= XM is obtained from which gj (X) < 0, j=1,2,...,m.

• If the constraints are consistent, it should be possible to obtain, by

applying the procedure, a point X1 that satisfies all the constraints.
However, there may exist situations in which the solution of the problem
formulated in step 3 gives the unconstrained or constrained local
minimum of gk (X) that is positive. In such cases, one has to start a fresh
with a new point X1 from step 1 onward.
 Initial value of the penalty parameter r1
• Since the unconstrained minimization
of  (X, rk) is to be carried out for a
decreasing sequence of rk , it might
appear that by choosing a very small
value of r1, we can avoid an excessive
number of minimizations of the
function . But from a computational
point of view, it will be easier to
minimize the unconstrained function 
(X, rk) if rk is large. This can be seen
qualitatively from the figure. It can be
seen that as the value of rk becomes
smaller, the value of the function 
changes more rapidly in the vicinity of
the minimum k*.
 Initial value of the penalty parameter r1
• Since it is easier to find the minimum
of a function whose graph is smoother,
the unconstrained minimization of 
will be easier if rk is large. However,
the minimum of  k , X k * will be
farther away from the desired minimum
X* if rk is large. Thus it requires an
excessive number of unconstrained
minimizations of  (X, rk) (for several
values of rk ) to reach the point X* if r1
is selected to be very large.
 Initial value of the penalty parameter r1
• Thus, a moderate value has to be
chosen for the initial penalty parameter
r1. In practice, a value of r1 that gives
the value of  (X1, r1) approximately
equal to 1.1 to 2.0 times the value of f
(X1) has been found to be quite
satisfactory in achieving quick
convergence of the process. Thus, for
any initial feasible starting point X1, the
value of r1 can be taken as
 Subsequent values of the penalty
parameter
• Once the initial value of rk is chosen, the subsequent values of rk have to be
chosen such that

For convenience, the values of rk are chosen according to the relation

where c<1. The value of c can be taken as 0.1, 0.2 or 0.5.

 Convergence Criteria
• Since the unconstrained minimization of  (X, rk) has to be carried out for
a decreasing sequence of values rk , it is necessary to use proper
convergence criteria to identify the optimum point and to avoid an
unnecessarily large number of unconstrained minimizations. The process
can be terminated whenever the following conditions are satisfied:

1. The relative difference between the values of the objective function

obtained at the end of any two consecutive unconstrained minimizations
falls below a small number 1, that is,
 Convergence Criteria
2. The difference between the optimum points X k * and X k-1 * becomes
very small. This can be judged in several ways. Some of them are given
below:
 Normalization of Constraints
• A structural optimization problem , for example, might be having
constraints on the deflection  and the stress  as:

where the maximum allowable values are given by  max =0.5 in and
max=20,000 psi. If a design vector X 1 gives the values of g1 and g2 as -0.2
and -10,000, the contribution of g1 will be much larger than that of g2 (by
an order of 104) in the formulation of the  function given by the equation
 Normalization of Constraints
• This will badly affect the convergence rate during the minimization of 
function. Thus it is advisable to normalize the constraints so that they vary
between -1 and 0 as far as possible. For the constraints shown in the below
equations

the normalization can be done as:

 Normalization of Constraints
• If the constraints are not normalized as shown in the equations below

the problem can still be solved effectively by defining different penalty

parameters for different constraints as:

where R1, R2,......, Rm are selected such that the contributions of different
gj(X) to the  function will be approximately the same at the initial point
X1.
 Normalization of Constraints
• When the unconstrained minimization of  (X, rk) is carried for a
decreasing sequence of values of rk , the values of R1, R2,......, Rm will not
be altered; however, they are expected to be effective in reducing the
disparities between the contributions of the various constraints to the 
function.

Example:

Subject to
g 2 ( x1 , x2 )  x2 0
Solution: To illustrate the interior penalty function method, we use the
calculus method for solving the unconstrained minimization problem in
this case. Hence there is no need to have an initial feasible point X1.
 Example 1
• The  function is:

• To find the unconstrained minimum of , we use the necessary conditions:

• These equations give:

 Example 1
• To obtain the solution of the original problem, we know that:
 Example 2
Example

 xi 0, i 1,2,3
Solution: The interior penalty function method coupled with the Davidon-Fletcher-
Powell method of unconstrained minimization and cubic interpolation method of
one-dimensional search is used to solve this problem. The necessary data is
assumed as follows:
Starting feasible point:
 Example 2
• The optimum solution of this problem is known to be:
Example 2
 Example 2
• Convergence Proof: The following theorem proves the convergence of
the interior penalty function method.

Theorem: If the function

is minimized for a decreasing sequence of values of rk, the unconstrained

minima Xk* converge to the optimum solution of the constrained problem
stated in

as