See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/315450072

Penalty function methods using matrix laboratory ( MATLAB )

Article · November 2012

DOI: 10.5897/AJMCSR12.027

CITATIONS READS
7 1,457

1 author:

Hailay Weldegiorgis Berhe

Jomo Kenyatta University of Agriculture and Technology
6 PUBLICATIONS   20 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Optimization and control theory View project

Modelling infectious diseases and optimal control View project

All content following this page was uploaded by Hailay Weldegiorgis Berhe on 20 March 2017.

The user has requested enhancement of the downloaded file.

African Journal of Mathematics and Computer Science Research Vol. 5(13), pp. 209-246, November, 2012
Available online at http://www.academicjournals.org/AJMCSR
DOI: 10.5897/AJMCSR12.027
ISSN 2006-9731 ©2012 Academic Journals

Full Length Research Paper

Penalty function methods using matrix laboratory

(MATLAB)
Hailay Weldegiorgis Berhe
Department of Mathematics, Haramaya University, Ethiopia. E-mail: [email protected]. Tel: +251913710270.
Accepted 15 May, 2012

The purpose of the study was to investigate how effectively the penalty function methods are able to
solve constrained optimization problems. The approach in these methods is to transform the
constrained optimization problem into an equivalent unconstrained problem and solved using one of
the algorithms for unconstrained optimization problems. Algorithms and matrix laboratory (MATLAB)
codes are developed using Powell’s method for unconstrained optimization problems and then
problems that have appeared frequently in the optimization literature, which have been solved using
different techniques compared with other algorithms. It is found out in the research that the sequential
transformation methods converge to at least to a local minimum in most cases without the need for the
convexity assumptions and with no requirement for differentiability of the objective and constraint
functions. For problems of non-convex functions it is recommended to solve the problem with different
starting points, penalty parameters and penalty multipliers and take the best solution. But on the other
hand for the exact penalty methods convexity assumptions and second-order sufficiency conditions for
a local minimum is needed for the solution of unconstrained optimization problem to converge to the
solution of the original problem with a finite penalty parameter. In these methods a single application of
an unconstrained minimization technique as against the sequential methods is used to solve the
constrained optimization problem.

Key words: Penalty function, penalty parameter, augmented lagrangian penalty function, exact penalty
function, unconstrained representation of the primal problem.

INTRODUCTION

Optimization is the act of obtaining the best result under minimum as compared to unconstrained ones. Several
given circumstances. In design, construction and situations can be identified depending on the effect of
maintenance of any engineering system, engineers have constraints on the objective function. The simplest
to take many technological and managerial decisions at situation is when the constraints do not have any
several stages. The ultimate goal of all such decisions is influence on the minimum point. Here the constrained
either to minimize the effort required or to maximize the minimum of the problem is the same as the
desired benefit. Since the effort required or the benefit unconstrained minimum, that is, the constraints do not
desired in any practical situation can be expressed as a have any influence on the objective function. For simple
function of certain decision variables, optimization can be optimization problems it may be possible to determine
defined as the process of finding the conditions that give before hand, whether or not the constraints have any
the maximum or minimum value of a function. It can be influence on the minimum point. However, in most of the
taken to mean minimization since the maximum of a practical problems, it will be extremely difficult to identify
function can be found by seeking the minimum of the it. Thus, one has to proceed with general assumption that
negative of the same function. the constraints will have some influence on the optimum
Optimization can be of constrained or unconstrained point. The minimum of a nonlinear programming problem
problems. The presence of constraints in a nonlinear will not be, in general, an extreme point of the feasible
programming creates more problems while finding the region and may not even be on the boundary. Also, the
210 Afr. J. Math. Comput. Sci. Res.

problem may have local minima even if the most cases) optimization, which has to be solved
corresponding unconstrained problem is not having local iteratively. The sequential transformation method is also
minima. Furthermore, none of the local minima may called the classical approach and is perhaps the simplest
correspond to the global minimum of the unconstrained to implement. Basically, there are two alternative
problem. All these characteristics are direct approaches. The first is called the exterior penalty
consequences of the introduction of constraints and function method (commonly called penalty function
hence, we should have general algorithms to overcome method), in which a penalty term is added to the objective
these kinds of minimization problems. function for any violation of constraints. This method
The algorithms for minimization are iterative generates a sequence of infeasible points, hence its
procedures that require starting values of the design name, whose limit is an optimal solution to the original
variable x. If the objective function has several local problem. The second method is called interior penalty
minima, the initial choice of x determines which of these function method (commonly called barrier function
will be computed. There is no guaranteed way of finding method), in which a barrier term that prevents the points
the global optimal point. One suggested procedure is to generated from leaving the feasible region is added to the
make several computers run using different starting objective function. The method generates a sequence of
points and pick the best. Majority of available methods feasible points whose limit is an optimal solution to the
are designed for unconstrained optimization where no original problem.
restrictions are placed on the design variables. In these Penalty function methods are procedures for approxi-
problems, the minima exist if they are stationary points mating constrained optimization problems by uncon-
(points where gradient vector of the objective function strained problems. The approximation is accomplished by
vanishes). There are also special algorithms for adding to the objective function a term that prescribes a
constrained optimization problems, but they are not easily high cost for the violation of the constraints. Associated
accessible due to their complexity and specialization. with this method is a parameter µ that determines the
All of the many methods available for the solution of a severity of the penalty and consequently the degree to
constrained nonlinear programming problem can be which the unconstrained problem approximates the
classified into two broad categories, namely, the direct original problem. As µ →∞ the approximation becomes
methods and the indirect methods approach. In the direct increasingly accurate.
methods the constraints are handled in an explicit Thus, there are two fundamental issues associated with
manner whereas in the most of the indirect methods, the this method. The first has to do how well the uncon-
constrained problem is solved as a sequence of strained problem approximates the constrained one. This
unconstrained minimization problems or as a single is essential in examining whether, as the parameter µ is
unconstrained minimization problem. Here we are increased towards infinity, the solution of the uncon-
concerned on the indirect methods of solving constrained strained problem converges to a solution of the
optimization problems. A large number of methods and constrained problem. The other issue, most important
their variations are available in the literature for solving from a practical view point, is the question of how to solve
constrained optimization problems using indirect a given unconstrained problem when its objective
methods. As is frequently the case with nonlinear function contains a penalty term. It turns out that as µ is
problems, there is no single method that is clearly better increased yields a good approximating problem; the
than the others. Each method has its own strengths and corresponding structure of the resulting unconstrained
weaknesses. The quest for a general method that works problem becomes increasingly unfavorable thereby
effectively for all types of problems continues. The main slowing the convergence rate of many algorithms that
purpose of this research is to present the development of may be applied. Therefore it is necessary to device
two methods that are generally considered for solving acceleration procedures that circumvent this slow
constrained optimization problems, the sequential convergence phenomenon. To motivate the idea of
transformation methods and the exact transformation penalty function methods consider the following nonlinear
methods. programming problem with only inequality constraints:
Sequential transformation methods are the oldest
methods also known as sequential unconstrained Minimize f(x), subject to g(x) 0 (P) x X;
minimization techniques (SUMT) based upon the work of
Whose feasible region we denote by S =
Fiacco and McCormick (1968). They are still among the n n m
. Functions f: R → R and g: R → R
most popular ones for some cases of problems, although
are assumed to be continuously differentiable and X is a
there are some modifications that are more often used.
nonempty set in Rn. Let the set of minimum points of
These methods help us to remove a set of complicating
constraints of an optimization problem and give us a problem (P) be denoted by M (f, S), where M (f, S) ≠ .
frame work to exploit any available methods for And we consider a real sequence {µk} such that ≥ 0.
unconstrained optimization problems to be solved, perhaps, The number is called penalty parameter, which controls
approximately. However, this is not without a cost. In fact, the degree of penalty for violating the constraints. Now we
this transforms the problem into a problem of non smooth (in consider functions θ: X (R+ {0}) → R, as defined by
 Berhe 211

θ(x, µ):= f(x) + µp(x), (x, µ) {(X) (R+ {0})}, (1.1) Objectives of the research

where X Rn and p(x) is called penalty function, to be General objective

used throughout this paper, and µp(x) is called penalty
term. µ is a strictly increasing function. Throughout this The purpose of the research is generally to see how the
paper we use penalty function methods for exterior penalty methods are successful to solve constrained
penalty function methods optimization problems.
Another apparently attractive idea is to define an exact
penalty function in which the minimizer of the penalty
function and the solution of the constrained primal Specific objectives
problem coincide. The idea in these methods is to choose
a penalty function and a constant penalty parameter so The specific objectives of the research are to:
that the optimal solution of the unconstrained problem is
also a solution of the original problem. This avoids the i. Describe the essence of penalty function methods,
inefficiency inherent in sequential techniques. The two ii. Clearly identify the procedures in solving constrained
popular exact penalty functions are l1 exact penalty optimization problems using penalty function methods,
function and augmented Lagrangian penalty function. iii. Develop an algorithm and MATLAB code for penalty
More emphasis is given here for sequential function methods,
transformation methods and practical examples, which iv. Solve real life application problems which frequently
appeared frequently in the optimization literature (which appeared in the optimization literature and facility location
have been solved using different methods.) and facility problems using the investigated code and compare with
locations are solved using the MATLAB code given in the other methods,
appendix with special emphasis given to facility location v. Compare the effectiveness of the penalty methods.
problems.
Some important techniques in the approach of the
primal problem and the corresponding unconstrained Significance of the research
penalty problems will be discussed later. We also discuss
properties of the penalty problem, convergence i. All algorithms for constrained optimization are
conditions and the structure of the Hessian objective unreliable to a degree. Any one of them works well on
function of the penalty problem and the methods for one problem and fails to another. Thus, this work will be
solving unconstrained problems; the general description having its own contribution in bridging the gap,
of the algorithm for the penalty function problems in ii. It will also pave way and serves as an eye opener to
addition to the considerations for the implementation of other researchers to carry out an extensive and/or detail
the method. A major challenge in the penalty function study along the same or other related issue.
methods is the ill-conditioning of the Hessian matrix of
objective function as the penalty parameter approaches
to infinity, the choice of the initial starting points, penalty PENALTY FUNCTION METHODS
parameters and subsequent values of the penalty
parameters. In this section, we are concerned with exploring the
In the later part of this study, the exact penalty computational properties of penalty function methods. We
methods, the exact l1 penalty function and the augmented present and prove an important result that justifies using
Lagrangian penalty function methods will be discussed in penalty function methods as a means for solving
detail. The sequential methods suffer from numerical constrained optimization problems. We also discuss
difficulties in solving the unconstrained problem. some computational difficulties associated with these
Furthermore, the solution of the unconstrained problem methods and present some techniques that should be
approaches the solution of the original problem in the used to overcome such difficulties. Using the special
limit, but is never actually equal to the exact solution. To structure of the penalty function, a special purpose one-
overcome these shortcomings, the so-called exact dimensional search procedure algorithm is developed.
penalty functions have been developed. The procedure is based on Powell’s method for
unconstrained minimization technique together with
bracketing and golden section for one dimensional
Statement of the problem search.
When solving a general nonlinear programming
The focus of this research paper is on investigating problem in which the constraints cannot easily be
constraint handling to solve constrained optimization eliminated, it is necessary to balance the aims of
problems using penalty function methods and thereby reducing the objective function and staying inside or
indicating ways of revitalizing them by bringing to close to the feasible region, in order to induce global
attention. convergence (that is convergence to a local solution from
212 Afr. J. Math. Comput. Sci. Res.

any initial approximation). This inevitably leads to the et al. (2006).

idea of a penalty function, which is a combination of the Now consider the following problem with single
some constraints that enables the objective function to be inequality constraint g(x) 0:
minimized whilst controlling constraint violations (or near
constraint violations) by penalizing them. The philosophy Minimize f(x),
of penalty methods is simple; you give a “fine” for subject to g(x) 0.
violating the constraints and obtain approximate solutions
to your original problem by balancing the objective It is clear that the form f(x) + µg2(x) is not appropriate,
function and a penalty term involving the constraints. By since a penalty will be incurred where g(x) < 0 or g(x) > 0;
increasing the penalty, the approximate solution is forced that is a penalty is added to the objective function
to approach the feasible domain and hopefully, the whether x is inside or outside the feasible region.
solution of the original constrained problem. Early penalty Needless to say, a penalty is desired only if the point is
functions were smooth so as to enable efficient not feasible, that is, if g(x) > 0. A suitable unconstrained
techniques for smooth unconstrained optimization to be problem is therefore given by:
used.
The use of penalty functions to solve constrained Minimize f(x) + µmaximum {0, g(x)},
optimization problems is generally attributed to Courant. subject to x ∈ Rn.
He introduced the earliest penalty function with equality
constraint in 1943. Subsequently, Pietrgykowski (1969) Note that if g(x) 0, then maximum {0, g(x)} = 0, and no
discussed this approach to solve nonlinear problems. penalty is incurred on the other hand, if g(x) > 0, then
However, significant progress in solving practical maximum {0, g(x)} > 0, and the penalty term µg(x) is
problems by use of penalty function methods follows the realized. However, it is observe that at points x where
classic work of Fiacco and McCormick under the title g(x) = 0, the forgoing objective function might not be
sequential unconstrained minimization technique differentiable, even though g is differentiable.
(SUMT). The numerical problem of how to change the
parameters of the penalty functions have been Example 1
investigated by several authors.
Fiacco and McCormick (1968) and Himmelblau (1972) Minimize x,
discussed effective unconstrained optimization algorithms subject to –x + 2 0,
for solving penalty function methods. According to
Fletcher, several extensions to the concepts of penalty the constraint, g(x) = -x + 2 0 is active at x = 2 and the
functions have been made; first, in order to avoid the corresponding forgoing objective function is:
difficulties associated with the ill-conditioning as the
penalty parameter approaches to infinity, several
parameter-free methods have been proposed. We will f(x) + µmaximum {0, g(x)} =
discuss some of the effective techniques in reducing
these difficulties in the following sections. Clearly this is not differentiable at x = 2. If differentiability
is desirable in such cases, then one could, for example,
consider instead a penalty function term of the type
The concept of penalty functions µ(maximum{0, g(x)})2.
In general, a suitable penalty function must incur a
Consider the following problem with single constraint h(x) positive penalty for infeasible points and no penalty for
= 0: feasible points. If the constraints are of the form gi(x) ≤ 0
for i = 1. . . m, then a suitable penalty function p is
Minimize f(x), defined by
subject to h(x) = 0.
p(x) = , (2.1a)
Suppose this problem is replaced by the following
unconstrained problem, where µ > 0 is a large number: where:

Minimize f(x) + µh2(x), is a continuous function satisfying the following

subject to x ∈ properties:

we can intuitively see that an optimal solution to the (y) = 0 if y ≤ 0 and (y) > 0 if y > 0. (2.1b)
above problem must have h2(x) close to zero, otherwise a
2
large penalty term µh (x) will be incurred and hence f(x) + Typically is of the form
2
µh (x) approaches to infinity which makes it difficult to
minimize the unconstrained problem (Bazaraa (y) = (maximum {0, y}) q,
 Berhe 213

where q is a nonnegative real number. Thus, the penalty The θ-function will have continuous first derivatives.
function p is usually of the form p(x) These derivatives are given by
= .
n = + .
Definition 1. A function p : R → R is called a penalty
function if p satisfies
n
Generally, the value of q is chosen as 2 in practical
i. p(x) is continuous on R computations and hence, will be used as q = 2 in the
ii. p(x) = 0 if g(x) 0 and subsequent discussion of the penalty method with
iii. p(x) > 0 if g(x) 0.
p(x) = .
An often-used class of penalty functions for optimization
problems with only inequality constraints is:

p(x) = , where q ≥ 1 Example 2

We refer to the function f(x) + μp(x) as the auxiliary Consider the optimization problem in example1,
function. Denoting θ(x, μ): = f(x) + Let p(x) = , then,
μ , for the auxiliary function. The
effect of the second term on the right side is to increase p(x) =
θ(x, μ) in proportion to the qth power of the amount by
which the constraints are violated. Thus there will be a
penalty for violating the constraints and the amount of Note that the minimum of f + μp occurs at the point 2 –
penalty will increase at a faster rate compared to the ( ) and approaches the minimum point x = 2 of the
amount of violation of a constraint (for q Rao (2009).
Let us see the behavior of θ(x, μ) for various values of q. original problem as μ → ∞. The penalty and auxiliary
functions are as shown in Figure 1.
If the constraints are of the form gi(x) ≤ 0 for I = 1, . . . ,
i. q = 0,
m and hi(x) = 0 for I = 1, . . . , l, then a suitable penalty
θ(x, μ) = f(x) + μ function p is defined by

= p(x) = + , (2.2a)

This function is discontinuous on the boundary of the where and ψ are continuous functions satisfying the
acceptable region and hence it will be very difficult to following properties:
minimize this function.
(y) = 0 if y ≤ 0 and (y) > 0 if y > 0. (2.2b)
ii. 0 (y) = 0 if y = 0 and (y) > 0 if y ≠ 0.

Here the θ-function will be continuous, but the penalty for Typically, and are of the forms
violating a constraint may be too small. Also the
q
derivatives of the function are discontinuous along the (y) = (maximum {0, y})
boundary. Thus, it will be difficult to minimize the θ- (y) = |y|
q

function.
where q is a nonnegative real number. Thus, the penalty
iii. q = 1 function p is usually of the form
In this case, under certain restrictions, it has been shown
that there exists a μo large enough that the minimum of θ p(x) = + .
is exactly the constrained minimum of the original
problem for all μk μo; however, the contours of the θ-
function posses discontinuous first derivatives along the Definition 2
boundary.
Hence, in spite of the convenience of choosing a single A function p : Rn → R is called a penalty function if p
μk that yields the constrained minimum in one satisfies
unconstrained minimization, the method is not very
attractive from computational point of view. i. p(x) is a continuous function on Rn
ii. p(x) = 0 if g(x) 0 and h(x) = 0 and
iv. q iii. p(x) > 0 if g(x) 0 and h(x) 0.
214 Afr. J. Math. Comput. Sci. Res.

(a) (b)
Figure 1. Penalty and auxiliary functions.

An often used class of penalty functions for this is: Minimize f(x)
subject to g(x) 0
p(x) = + , where q ≥ h(x) = 0 (P)
1. x X,

We note the following: whose feasible region we denote by S:

= The constraints h(x) = 0
If q = 1, p(x) is called “linear penalty function.” This and g(x) ≤ 0 are referred to as functional constraints,
function may not be differentiable at points where gi(x) = while the constraint x X is a set constraint. The set X is
0 or hi(x) = 0 for some i. a nonempty set in Rn and might typically represent simple
Setting q = 2 is the most common form that is used in constraints that could be easily handled explicitly, such
practice and is called the “quadratic penalty function”. as lower and upper bounds on the variables. We
We focus here mainly on the quadratic penalty function emphasize the set constraint, assuming in most cases
and investigate how penalty function methods are useful that either X is in the whole Rn or that the solution to (P) is
to solve constrained optimization problems by changing in the interior of X.
into the corresponding unconstrained optimization By converting the constraints “ (x) = 0” to
problems. “ (x) or considering only problems with
inequality constraints we can assume that (P) is of the
form:
Penalty function methods for mixed constraints
Minimize f(x) (P)
Consider the following constrained optimization problem: subject to g(x) 0
x X,
Minimize f(x) (P)
subject to ≤ 0, I = 1, . . . , m whose feasible region we denote by S:
, I = 1, . . . , l, = We then consider solving the
following penalty problem,
where functions f, hi(x), I = 1, . . . , l and gi, I = 1, . . . , m
are continuous and usually assumed to posses θ(μ): Minimize f(x) + μp(x)
continuous partial derivatives on Rn. For notational subject to x ∈ X,
simplicity, we introduce the vector-valued functions h =
(h1, h2, . . . , hl)T Rl and g = (g1, g2, . . . , gm)T ∈ Rm and and investigate the connection between a sequence
rewrite (P) as: { }, M(f, X), a minimum point of θ and a solution of
 Berhe 215

the original problem (P), where the set of minimum points computing θ(µ) for a sufficiently large µ.This result is
of θ is denoted by M(f, X) and the set of all minimum established in Theorem 2. First, the lemma theorem is
points of (P) is denoted by M(f, S). needed.
The representation of penalty methods above has
assumed either that the problem (P) has no equality Lemma 1 (Penalty Lemma)
constraints, or that the equality constraints have been
converted into inequality constraints. For the latter, the Suppose that f, g1, . . . , gm, h1, . . . , hl are continuous
conversion is easy to do, but the conversion usually n n
functions on R , and let X be a nonempty set in R . Let p
violates good judgments in that it unnecessarily n
be a continuous function on R as given by definition 1,
complicates the problem. Furthermore, it can cause the and suppose that for each µ, there exists an x µ ∈ X, which
linear independence condition to be automatically is a solution of θ(µ), where θ(µ) := f(xµ ) + µp(xµ ).
violated for every feasible solution. Therefore, instead let Then, the following statements hold:
us consider the constrained optimization problem (P) with
both inequality and equality constraints since the above 1. p(xµ) is a non-increasing function of µ.
can be easily verified from this. To describe penalty 2. f(xµ) is a non-decreasing function of µ.
methods for problems with mixed constraints, we denote 3. θ(µ) is a non-decreasing function of µ.
the penalty parameter by l(μ) = μ ≥ 0, which is a
4. nf{f(x) : x S} ≥ , where θ(µ) = inf{f(x) +
monotonically increasing function and the penalty
function P(x) = + , satisfying the µp(x) : x X}, and g, h are vector valued functions whose
components are g1, g2, . . . , gm and h1, h2 , . . . , hl
properties given in (2.2b) and then consider the following
respectively.
Primal and Penalty problems:
Proof: Assume that µ and λ are penalty parameters such
Primal problem that λ < µ.

Minimize f(x) 1. By the definition of θ(λ), xλ is a solution of θ(λ) such

subject to g(x) that,
0 h(x) = 0 (P) x X.
θ(λ) = f(xλ) + λp(xλ) ≤ inf{f(x) + λp(x), for all x X}, which
follows
Penalty problem
f(xλ) + λp(xλ) ≤ f(xµ) + λp(xµ), since xµ ∈ X. (2.3a)
The basic penalty function approach attempts to solve
the following problem: Again by the definition of θ(µ)

Maximize θ(μ) θ(µ) = f(xµ) + µ p(xµ) ≤ inf{f(x) + µ p(x), for all x ∈ X} which
subject to μ ≥ 0, follows that

where θ(μ) = inf{f(x) + μp(x) : x X}. The penalty problem f(xµ) + µp(xµ) ≤ f(xλ) + µp(xλ), since ∈ X. (2.3b)
consists of maximizing the infimum (greatest lower
bound) of the function {f(x) + μp(x): x X}; therefore, it is Adding equation (2.3a) and (2.3b) holds:
a max-min problem. Therefore the penalty problem can
be formulated as: f(xλ) + λp(xλ) + f(xµ) + µp(xµ) ≤ f(xµ) + λp(xµ) + f(xλ) + µp(xλ)

Find which is equivalent to the and simplifying like term, we get

form;
λp(xλ) + µ p(xµ) ≤ λp(xµ) + µp(xλ),
Find We remark here that,
strictly speaking, we should write the penalty problem as
which implies by rearranging that
sup{θ(μ), μ ≥ 0}, rather than maximum{θ(μ), μ ≥ 0}, since
the maximum may not exist. The main theorem of this
section states that: (λ - µ)[p(xλ) – p(xµ)] ≤ 0.

inf{f(x) : x S} = = . Since λ - µ ≤ 0 by assumption, p(x λ) – p(xµ) ≥ 0. Then,

p(xλ) ≥ p(xµ).
From this result, it is clear that we can get arbitrarily close
to the optimal objective value of the original problem by Therefore, p(xµ) is a non increasing function of µ.
216 Afr. J. Math. Comput. Sci. Res.

2. By (2.3a) above Proof

f(xλ) + λp(xλ) ≤ f( ) + λp( ). We first show that p( ) → 0 as µ → ∞. Let y be any

feasible point and ε > 0.
Since p(xλ) ≥ p( ) by part 1, we concluded that Let x1 be an optimal solution to the problem minimize {f(x)
+ µp(x), x ∈ X}, for µ = 1.
f(xλ) ≤ f( ). If we choose µ ≥ |f(y) – f(x1)| + 2, then by part 2 of
Lemma 1 we have f( ) ≥ f(x1).
3. θ(λ) = f(xλ) + λp(xλ) ≤ f( ) + λp( ) We now show that p( ) ≤ ε. By contradiction, suppose
≤ f( ) + µp( ) = θ(µ). p(xµ) > ε. Noting part 4 of lemma 1, we get

4. Suppose be any feasible solution to problem (P) inf{f(x), x ∈ S} ≥ θ(µ) = f(xµ) + µp(xµ) ≥ f(x1)+µp(xµ)
with > f(x1) + ( |f(y) – f(x1)| + 2)ε
= f (x1) + |f(y) – f(x1)| + 2ε > f(y)
g( ) ≤ 0, h( ) = 0 and p( ) = 0, where ∈ X. Then,
f( ) + µp( ) = inf{f(x), x ∈ S} which implies that it follows that inf{f(x), x ∈ S} > f(y). This is not possible in
f( ) = inf{f(x), x ∈ S}. (2.3c) the view of feasibility of y.
1
By the definition of θ(µ) Thus, p( ) ≤ ε for all µ ≥ |f(y) – f(x1)| + 2. Rearranging

θ(µ) = f( ) + µp( ) ≤ f( ) + µp( ) = inf{f(x), x ∈ S }, for the above we get ε ≥ |f(y) – f(x1))| + , since ε > 0 is
all µ ≥ 0. arbitrary, p( ) → 0 as µ → ∞.
To show inf{f(x) : x ∈ S } = = .
Therefore, ≤ {inf{f(x), x ∈ S }}. Let { } be any arbitrary convergent sequence of {xµ},
and let be its limit. Then,
The next result concerns convergence of the penalty
method. It is assumed that f(x) is bounded below on the ≥ θ(µk) = f( )+ p( ) ≥ f( ).
(nonempty) feasible region so that the minimum exists.
Since → and f is continuous function with
= f( ) , then the above inequality implies
Theorem 2 (Penalty convergence theorem)
that
Consider the following Primal problem:
≥ f( ). (2.4)
Minimize f(x)
subject to g(x) 0 Since p( ) → 0 as µ → ∞, then p( ) = 0 , that is, is a
h(x) = 0 (P) feasible solution to the original problem (P) which follows
x X, that inf{f(x) : x ∈ S } = f( ).

where f, g, h are continuous functions on Rn and X is a By part 3 of Lemma 1 θ(µ) is a nondecreasing function of
nonempty set in Rn. Suppose that the problem has a µ, then
feasible solution denoted by , and p is a continuous
function of the form (2.2). Furthermore, suppose that for = . (2.5a)
each µ, there exists a solution ∈ X to the problem to
minimize {f(x) + µp(x) subject to x ∈ X}, and {xµ} is is an optimal solution to (P) by assumption implies that
contained in a compact subset X then,
inf{f(x) : x ∈ S } = f( ) (2.5b)
inf{f(x) : x ∈ S } = = ,
and by part 4 of the Lemma 1 above
where θ(µ) = inf{f(x) + µp(x) : x ∈ X} = f( ) + µp( ). ≤ inf{f(x) : x ∈ S }. (2.5c)
Furthermore, the limit of any convergent subsequence
of { } is an optimal solution to the original problem, and Equating (2.4), (2.5a), (2.5b) and (2.5c), we get
µp( ) → 0 as µ → ∞. inf{f(x) : x ∈ S } = =
 Berhe 217

To show µp( ) → 0 as µ → ∞. referred to as an exterior penalty function method.

θ(µ) = f( ) + µp( )
µp( ) = θ(µ) – f( ).
Karush Kuhn Tucker multipliers at optimality
Taking the limit as µ → ∞ to both sides Under certain conditions, we can use the solutions to the
= sequence of penalty problems to recover the KKT
Lagrange multipliers associated with the constraints at
= – f( ) optimality. Suppose X = Rn for simplicity and consider the
= f( ) – f( ) primal problem (P) and the penalty function given in (2.2).
= 0. In the penalty methods we solved, for various values of µ,
the unconstrained problem is
So that µp( ) → 0 as µ → ∞.
Minimize f(x) + µp(x) (2.6)
Note: It is interesting to observe that this result is
subject to x ∈ X.
obtained in the absence of differentiability or Karush
Kuhn-Tucker regularity assumptions. Most algorithms require that the objective function has
Corollary 3 continuous first partial derivatives. Hence we shall
assume that f, g, h ∈ C1. It is natural to require, that the
If p( ) = 0 for some µ, then is an optimal solution to penalty function p ∈ C1. As we explained earlier, the
the original problem (P) derivative of maximum {0, (x)} is usually discontinuous
at points where (x) = 0 and thus, some restrictions must
Proof be placed on in order to guarantee p ∈ C1. We assume
that the functions and are continuously differentiable
If p( ) = 0 ,then is a feasible solution to the problem and satisfy:
(P). Furthermore, since
inf{f(x), x ∈ S} ≥ θ(µ) = f( )+ µp( ) = f( ) it follows that (y) = 0 if y ≤ 0 and (y) ≥ 0 for all y. (2.7)
inf{f(x), x ∈ S} ≥ f( ) In view of this assumption p is differentiable whenever f,
g, h are differentiable, that is, f, g, h ∈ C1 implies p ∈ C1
it immediately follows that is an optimal solution to (P). and we can write
Note the significance of the assumption that { } is
contained in a compact subset X. obviously, this p(x) = + .
assumption holds if X is compact. Without this
assumption, it is possible that the optimal objective Assuming that the conditions of Theorem 2 hold true,
values of the primal problem and the penalty problems since solves the problem to minimize {f(x) + µp(x), x ∈
are not equal. This assumption is not restricted in most X}, the gradient of the objective function of this penalty
practical cases, since the variables usually lie between problem must vanish at . This gives
finite lower and upper bounds.
From the above theorem, it follows that the optimal f( ) + p( ) = 0 for all μ,
solution to the problem to minimize f(x) + µp(x) subject
to x ∈ X can be made arbitrarily close to the feasible that is,
region by choosing µ large enough. Furthermore, by
choosing µ large enough, f( ) + µp( ) can be made f(xμ) + +
arbitrarily close to the optimal objective value of the = 0.
original problem. One popular scheme for solving the
penalty problem is to solve a sequence of problems of Now let be an accumulation point of the generated
the form: sequence { }. Without loss of generality, assume that
{ } itself converges to and so is an optimal solution
Minimize f(x) + µp(x)
subject to x ∈ X,
to (P).
for an increasing sequence of penalty parameters. The
optimal points { } are generally infeasible as seen in Denoted by:
proof of the Theorem 2, as the penalty parameter µ is
made large, the points generated approach an optimal I = {I : ( ) = 0} to be the set of inequality constraints that
solution from outside the feasible region. are binding at and
Hence, as mentioned earlier, this technique is also N = {I : ( ) < 0} for all constraints not binding at .
218 Afr. J. Math. Comput. Sci. Res.

Since gi( ) < 0 for all elements of N then by Theorem 2.2, follows that ≥ 0 for I ∈ I, for all μ sufficiently large.
We have gi( ) < 0 for sufficiently large μ which results
= 0 (by assumption). Hence, we can write the → , as μ → ∞. Then = 0 for I ∈ N.
foregoing identity as From the continuity of all functions involved,

f(xμ) + + = 0, (2.8a) f(xμ) + + = 0, implies

f( ) + + = 0.
for all μ large enough, where and are vectors
having components From the above remarks, we also have ≥ 0 and =0
for all I ∈ N. Thus ( , are vectors of Karush-Kuhn-
μ )) ≥ 0 for all I ∈ I and = μ ( ( )) Tucker multipliers. It therefore remains to show
for all I = 1, . . . , l. (2.8b) → , as μ → ∞ for some unique ( , ).
Suppose has no accumulation point, then
Let us now assume that is a regular solution such that
and are linearly independent then, we ||(uμ, )|| → ∞. But then define (W μ, )= ( ), and
know that there exist unique scalars , I ∈ I and , I then ||(W μ, )|| = 1 for all μ, and so the sequence
= l, . . . , l such that has some accumulation point ( , ) point.
f(xμ) + + = 0. For all I ∈ N, = 0 for all μ large, where by = 0 for
all I ∈ N, and
Since g, h, , are all continuously differentiable and ( )+ ( = ( )+
since {xμ} → , which is a regular point, we must then ( )
have in (2.8) that
= ( )+ ( )
, for all I ∈ I and → , for all I = 1, . . . , l. =-

For sufficiently large values of μ, the multipliers given in

(2.8) can be used to estimate KKT Lagrange multipliers for μ large. As μ → ∞, we have x μ → , (W μ, ) → ,
at optimality and so we can interpret and v μ as a sort and ||(uμ, )|| → ∞ by assumption, and so the above
of vector of Karush-Kuhn-Tucker multipliers. The result equation becomes;
stated in next lemma insures that → and vμ → .
( )+ ( = 0, and ||(W μ, )|| =
1, which violates the linear independence condition.
Lemma 4 Therefore { } is bounded sequence, and so has at
least one accumulation point.
Suppose (y) and ψ(y) are continuously differentiable Now suppose that { } has two accumulation
and satisfy (2.7), and that f, g, h are differentiable. points, ( , ) and ( , ). Note = 0 and = 0 for I ∈ N,
Let , vμ) be defined by (2.8). Then, if → , and and so
satisfies the linear independence condition for gradient
vectors of active constraints ( is a regular solution), then + = - = +
→ , , where , ) are vectors of KKT ,
multipliers for the optimal solution of (P). so that
+ = 0.
Proof: From the Penalty Convergence Theorem, is an
optimal solution of (P). But by the linear independence condition, = 0 for all
I ∈ I, and . This implies that ( , ) = ( ).
Let

I = {I | ( ) = 0} and Remark: The quadratic penalty function satisfies the

condition (2.7), but the linear penalty function does not
satisfy.
N = {I | ( ) < 0}.
As a final observation we note that in general if → ,
For I ∈ N, ( ) < 0 for all μ sufficiently large, so =0 then since μ )) → and =
for all μ sufficiently large, whereby = 0 for I ∈ N. μ ( ( )) → , the sequence xμ approaches from
From (2.8b) and the definition of a penalty function, it outside the constraint region. Indeed as → all
 Berhe 219

constraints that are active at and have positive Example 4

Lagrange multipliers will be violated at x μ because the
corresponding )) are positive. Thus, if we assume Minimize x
that the active constraints are non degenerate (all subject to -x + 2 ≤ 0.
Lagrange multipliers are strictly positive), every active
constraint will be approached from outside of the feasible The corresponding penalty problem is:
region.
Consider the special case if p is the quadratic penalty Minimize x + μ
function given by: subject to x ∈ R,
= 1 + 2μ[maximum ](-1) = 0, for x < 2
p(x) = + , then p(y) = which implies that 1 + 2μ – 4μ = 0.
+ , 2maximum{0, y} Therefore, =2– , and
and 2y. Hence, from (2.8), we obtain ( ( )), for I ∈ I
= 2μ(-( ) + 2)
, for all I ∈ I, and
= 2μ (xμ) for all I = 1, . . . , l. (2.9) =1
It follows that Uµ = 1.
In particular, observe that if > 0 for some I ∈ I, then Note that, as μ → ∞, 1, the optimal value of the
> 0 for μ large enough and then (xμ) > 0 and by
Lagrange multiplier for the primal problem.
our assumption )) > 0 for (xμ) > 0. This means
that (x) ≤ 0 is violated all along the trajectory leading to
, and in the limit ( ) = 0. Hence, if > 0 for some I ∈ I, Example 5
≠ 0 for all I, then all the constraints binding at are
violated along the trajectory { } leading to . Minimize +2
subject to - – + 1 ≤ 0, x .

For this problem the Lagrangian is given by

Example 3
L(x, u) = + 2 + u( - – + 1). The KKT
conditions yield:
Consider the following optimization problem:

Minimize + =2 – u = 0 and =4 – u = 0; u(- – +

subject to + =1 1) = 0.

and the corresponding penalty problem: Solving these results in = 2/3; = 2/3; = 4/3; (u = 0
yields an infeasible solution).
Minimize μ To consider this example using penalty method, define
subject to ( ) , the penalty function

where μ is a large number.

p(x) =
Xμ = [ ] is the solution for the penalty problem.
The unconstrained problem is then,
And
h(xμ) = -1 = ; and so,
minimize +2 + .
If p(x) = 0, then the optimal solution is x* = (0, 0) which is
(vμ) = 2μh(xμ) = 2μ( ) infeasible.

Implies Therefore, p(x) = = + 2

+ and the necessary conditions for the
optimal solution yield the following:
vμ = from (2.8)

=2 + 2µ( )(-1), and

As μ → ∞, , which is the optimal value of the
Lagrange multiplier for this example. =2 +2 )(-1) = 0.
220 Afr. J. Math. Comput. Sci. Res.

Thus, = and = for any fixed µ. standard type of analysis will be applicable to the tail of
such a sequence (Luenberger, 1974).
When μ → ∞, this converges to the optimum solution of
=( ).
Consider the constrained optimization problem:
Now suppose we use (2.8) to define

= , then Minimize {f(x), x ∈ S}

= 1 – {2μ/(2+3μ)} – {μ/(2+3μ)}) whose feasible region we denote by S: = {x ∈ X | g(x) ≤ 0,

= 2μ(1 – {3μ/(2+3μ)}) h(x)} and the corresponding unconstrained problem:
= (4μ/(2+3μ)).
( ) = f(x) + μp(x), p(x) = + ,
Then it is readily seen that = 4/3 =
(the optimal Lagrangian multiplier for this example). where f, g, h, , ψ are assumed to be twice continuously
Therefore the above Lemma 4 is true under some differentiable at . Then denoting by and the
regularity conditions. gradient and the Hessian operators for the functions Q, f,
g, h, respectively, and denoting the first and second
derivatives of and ψ as , and , (all with
Ill-Conditioning of the Hessian matrix
respect to x) we have,
Since the penalty function method must, for various
(large) values of μ, solve the unconstrained problem: = f( ) + +

Minimize f(x) + μp(x) And

subject to x X,

It is important, in order to evaluate the difficulty of such a Q(x, µ) = + ] +

problem, to determine the eigenvalue structure of the + .
Hessian of this modified objective function. The (2.10)
motivation for this is that the eigenvalue structure of the
Hessian of the objective function determines the natural To estimate the convergence rate of algorithms designed
rates of convergence for algorithms designed for to solve the modified objective function let us examine
unconstrained optimization problems. We show here that the eigenvalue structure of (2.10) as μ → ∞, and under
the structure of the eigenvalue of the corresponding the conditions of Theorem 2.2, as x , an
unconstrained problem becomes increasingly unfavora-
optimum solution to the given problem. Assuming that
ble as μ increases. Although, one usually insists for
→ and is a regular solution, we have from (2.8)
computational as well as theoretical purposes that the
function p ∈ c1, one usually does not insist that p ∈ c2. In that,
particular, the most popular penalty function p(x) =
(maximum {0, y})2, has discontinuity in its second μ )) → ≥ 0 for i ∈ I and μ ( ( )) → , i = 1, . .
derivative at any point where the component of g is zero, . , l,
that is, 2(maximum{0, y}), but would have
been undefined at y = 0 (as shown below). Hence, the where the optimal Lagrange multipliers associated with
Hessian of the unconstrained problem would be the constraint. Hence, the term in [.] approaches the
undefined at points having binding inequality constraints. Hessian of the Lagrangian function of the original
At first this might appear to be a serious drawback, since problem as → , which is
it means the Hessian is discontinuous at the boundary of
the constraint region-right where, in general, the solution = + ,
is expected to lie.
However, as pointed out above, the penalty method and has a limit that is independent of μ. The other term in
generates points that approach a boundary solution from (2.10), however, is strongly tied in with μ, and is
the outside the constraint region. Thus, except for some potentially explosive.
possible chance occurrences, the sequence will, as
→ , be at points where the Hessian is well-defined. For example, if and ψ(y) = ,
Furthermore, in iteratively solving the above as the popular quadratic penalty functions for the
unconstrained problem with a fixed µ, a sequence will be inequality and equality constraints and considering a
generated that converges to which is (for most value of primal problem with equality or inequality constraints
µ) a point where the Hessian is well-defined, and the separately we have two matrices.
 Berhe 221

means that the structure of the Lagrangian in the

neighborhood of interest is close to that of Lagrangian at
)) =
the true solution. Secondly, we conclude that, for large µ,
the matrix Q is positive definite. For any µ, Q must be at
least positive semi definite at the solution to the penalty
where problem: it is indicated that a stronger condition holds for
large µ.

=
Example 6
Thus, Consider the auxiliary function, (x, µ) =
+ , of example 2.5. The Hessian is:
= 2μ ,
H= .
which is 2µ times a matrix that approaches
. This matrix has rank equal to the rank
of the active constraints at (Luenberger, 1974). Suppose we want to find its eigenvalues by solving det |H
Assuming that there are r1 active constraints at the – | = 0,
solution , then for well behaved the matrix Q(x, µ) with |H – | = -4
only inequality constraints has r1 eigenvalues that tend to = – (6 +4 + 8 + 12 .
∞ as μ → ∞, but the n – r1 eigenvalues, though varying
with μ, tend to finite limits. These limits turnout to be the This quadratic equation yields
eigenvalues of L( ) restricted to the tangent subspace M
of the active constraints. The other matrix = (3 + 2μ) ± ,
2μ with l equality constraints has = (3 + 2μ) - and = (3 + 2μ) + .
rank l. As μ → ∞, → , the matrix Q(x, µ) with only
equality constraints has l eigenvalues that approach Note that → ∞ as µ → ∞, while is finite; and, hence,
infinity while the n - l eigenvalues approach some finite the condition number of H approaches ∞ as µ → ∞.
limits. Consequently, we can expect a severely ill- Taking the ratio of the largest and the smallest
conditioned Hessian matrix for large values of μ. eigenvalue yields
Considering equation (2.10) with both equality and It should be clear that as μ → ∞, the limit of the
inequality constraints we have as → , is a local preceding ratio also goes to ∞. This indicates that as the
solution to the constrained minimization problem (P) and iterations proceed and we start to increase the value of μ,
that it satisfies the Hessian of the unconstrained function that we are
minimizing becomes increasingly ill-conditioned. This is a
h( ) = 0 and gA ( ) = 0 and gI( ) < 0, common situation and is especially problematic if we are
using a method for the unconstrained optimization that
where gA and gI, is the induced partitioning of g into r1 requires the use of the Hessian.
active and r2 inactive constraints, respectively. Assuming
that the l gradients of h and the r1 gradients of gA
evaluated at together are linearly independent, then is Unconstrained minimization techniques and penalty
said to be regular. It follows from this expression that, for function methods
large µ and for close to the solution of (P) the matrix Q
In this we mainly concentrate on the problems of
has l + , eigenvalues of the order of µ. Consequently, efficiently solving the unconstrained problems with a
we can expect a severely ill-conditioned Hessian matrix penalty method. The main difficulty as explained above is
for large values of μ. Since the rate of convergence of the the extremely unfavorable eigenvalue structure. Certainly
method of steepest descent applied to a functional is straight forward application of the method of steepest
determined by the ratio of the smallest to the largest descent is out of the question.
eigenvalues of the Hessian of that functional, it follows in
particular that the steepest descent method applied to
converges slowly for large . Newton’s method and penalty function methods
In examining the structure of Q is therefore; first, as µ is
increased, the solution of the penalty problem One method for avoiding slow convergence for the
approaches the solution of the original problem, and, problems is to apply Newton’s method (or one of its
hence, the neighborhood in which attention is focused for variations), since the order two convergence of Newton’s
convergence analysis is close to the true solution. This method is unaffected by the poor eigenvalue structure.
222 Afr. J. Math. Comput. Sci. Res.

In applying the method, however, special care must be use it here is:
devoted to the manner by which the Hessian is inverted,
since it is ill-conditioned. Nevertheless, if second order a. First, it is assumed that the objective and constraint
information is easily available, Newton’s method offers an functions be continuous and smooth (continuously
extremely attractive and effective method for solving differentiable). Experience has shown this to be a more
modest size penalty and barrier optimization problems. theoretical than practical requirement and this restriction
When such information is not readily available, or if is routinely violated in engineering design and in some
data handling and storage requirements of Newton’s facility location problems. Therefore it is better to develop
method are excessive, attention naturally focuses on zero a general code that solves both differentiable and non-
order or first order methods. differentiable problems.
b. The input of the derivative if it exists is tiresome for
problems with large number of variables. In spite of its
Conjugate gradients and penalty function methods advantages, Newton’s method for example is not
generally used in practice due to the following features of
According to Luenberger (1984) the partial conjugate the method:
gradient method for solving unconstrained problems is
ideally suited to penalty and barrier problems having only i. It requires the storing of the n n Hessian matrix of the
a few active constraints. If there are l active constraints, objective function,
then taking cycles of 1+1 conjugate gradient steps will ii. It becomes very difficult and sometimes, impossible to
yield a rate of convergence that is independent of µ. For compute the elements of the Hessian matrix of the
example, consider the problem having only equality objective function,
constraints: iii. It requires the inversion of the Hessian matrix of the
objective function at each step,
Minimize f(x) (P)
iv. It requires the evaluation of the product of inverse of
subject to h(x) = 0,
the Hessian matrix of the objective function and the
where x Rn, h(x) Rl, l < n. Applying the standard negative of the gradient of the objective function at each
quadratic penalty method, we solve instead the step.
unconstrained problem:
Because of the above reasons I do not prefer first and
minimize f(x) + µ , second order methods and I did not give more emphasis
on these methods and their algorithms.
for large µ. The objective function of this problem has a Finally, we should not use second-order gradient
Hessian matrix that has l eigenvalues that are of order µ methods (e.g., pure Newton's method) with the quadratic
in magnitude, while the remaining n – l eigenvalues are loss penalty function for inequality constraints, since the
close to the eigenvalues of the matrix LM, corresponding Hessian is discontinuous (Belegundu and, Chandrupatla,
to the primal problem (P). Thus, letting x µ+1 be 1999). To see this clearly, consider:
determined from xµ by making l + 1 steps of a
(nonquadratic) conjugate gradient method, and assuming Minimize f(x) = 100/x
xµ → , a solution to , the sequence {f(xµ)} converges subject to g = x -5 ≤ 0,
linearly to f( ) with a convergence ratio equal to
approximately with f(x) being a monotonically decreasing function of x.
At the optimum = 5, the gradient of p(x) is 2µmax(0, x –
β−α 2 5). Regardless of whether we approach from the left or
( )
β+α right, the value of at is zero. So, (x) is first-order
where and are, respectively, the smallest and largest differentiable. However, = 0 when approaching from
eigenvalues of LM( ). This is an extremely effective
technique when l is relatively small. The method can be the left while = 2µ when approaching from the right.
used for problems having inequality constraints as well Thus, the penalty function is not second-order
but it is advisable to change the cycle length, depending differentiable at the optimum.
on the number of constraints active at the end of the
previous cycle.
Here we will use Powell’s method which is the zero Powell’s method and penalty function methods
order method. Powell’s method is an extension of the
basic pattern search methods. It is the most widely used Powell’s method is a zero-order method, requiring the
direct search method and can be proved to be a method evaluation of f(x) only. If the problem involves n design
of conjugate directions. This is as effective as the first variables, the basic algorithm is (Kiusalaas, 2005):
order methods like the gradient method for solving
unconstrained optimization problems. The reason why we Choose a point x0 in the design space.
 Berhe 223

Choose the starting vectors vi , i = 1, 2, . . . , n(the usual monotonically increasing values of μk, let {μk}, k = 1, . . .
choice is v i = ei , where ei is the unit vector in the xi- be a sequence tending to infinity such that μk ≥ 0 and μk+1
coordinate direction). > μk. Now for each k we solve the problem
Cycle
do with i = 1, 2, . . . , n Minimize {θ(x, μk), x X}. (2.11)
Minimize f(x) along the line through xi−1 in the direction of
vi. Let the minimum point be xi. To obtain xk, the optimum it is assumed that problem
(2.11) has a solution for all positive values of μk. A simple
end implementation known as the sequential unconstrained
do v n+1 ← xn - x0 (this vector is conjugate to vn+1 produced minimization technique (SUMT) is given below.
in the previous loop).
Minimize f(x) along the line through x0 in the direction of Step 0: (Initialization) Select a growth parameter β > 1
vn+1. Let the minimum point be xn+1. and a stopping parameter ε > 0 and an initial value of the
if |xn+1 − x0| < ε exit loop penalty parameter μ0. Choose a starting point x0 that
do with i = 1, 2, . . . , n violates at least one constraint and formulate the
vi ← vi+1 (v1 is discarded, the other vectors are reused) augmented objective function θ(x, µk). Let k = 1.
end do end cycle.
Powell (1997) demonstrated that the vectors vn+1 Step 1: Iterative - Starting from xk-1, use an
produced in successive cycles are mutually conjugate, so unconstrained search technique to find the point that
that the minimum point of a quadratic surface is reached minimizes θ(x, μk–1) and call it xk .
in precisely n cycles. In practice, the merit function is
seldom quadratic, but as long as any function can be Step 2: Stopping Rule - If the distance between x k–1 and
approximated locally by quadratic function, Powell’s xk is smaller than ε, that is, || xk–1 – xk || < ε or the
method will work. Of course, it usually takes more than n difference between two successive objective function
cycles to arrive at the minimum of a non quadratic values is smaller than ε, that is, |f(xk-1) – f(xk)| < ε, stop
function. Note that it takes n line minimizations to with xk an estimate of the optimal solution otherwise, put
construct each conjugate direction. μk = βμk–1, and formulate the new θ(x, µk) and put k = k+1
Powell’s method does have a major flaw that has to be and return to the iterative step.
remedied; if f(x) is not a quadratic, the algorithm tends to
produce search directions that gradually become linearly
dependent, thereby ruining the progress towards the Considerations for implementation of the penalty
minimum. The source of the problem is the automatic function method
discarding of v1 at the end of each cycle. It has been
suggested that it is better to throw out the direction that Starting point x1
resulted in the largest decrease of f(x), a policy that we
adopt. It seems counter intuitive to discard the best First in the solution step is to select a starting point. A
direction, but it is likely to be close to the direction added good rule of thumb is to start at an infeasible point. By
in the next cycle, thereby contributing to linear depen- design then, we will see that every trial point, except the
dence. As a result of the change, the search directions last one, will be infeasible (exterior to the feasible region).
cease to be mutually conjugate, so that a quadratic form A reasonable place to start is at the unconstrained
is not minimized in n cycles any more. This is not a minimum. Always we should ensure that the penalty does
significant loss since in practice f(x) is seldom a quadratic not dominate the objective function during initial iterations
anyway. Powell suggested a few other refinements to of penalty function method.
speed up convergence. Since they complicate the
bookkeeping considerably, we did not implement them.
Selecting the initial penalty parameter (µ0)
General description of the penalty function method
algorithm The initial penalty parameter μ0 should be fixed so that
the magnitude of the penalty term is not much smaller
The detail of this and a MATLAB computer program for than the magnitude of objective function. If an imbalance
implementing the penalty method using Powell’s method exists, the influence of the objective function could direct
of unconstrained minimization is given in the appendix. the algorithm to head towards an unbounded minimum
even in the presence of unsatisfied constraints. Because
Algorithm 1 (Algorithm for the penalty function the exterior penalty method approach seems to work so
method) well, it is natural to conjecture that all we have to do is set
μ to a very large number and then optimize the resulting
To solve the sequence of unconstrained problems with augmented objective function θ(x, μk) to obtain the
224 Afr. J. Math. Comput. Sci. Res.

solution to the original problem. Unfortunately, this about the same magnitude. This scaling operation is
conjecture is not correct. First, “large” depends on the intended to ensure that no subset of the constraints has
particular model. It is almost always impossible to tell an undue influence on the search process. If some
how large μ must be to provide a solution to the problem constraints are dominant, the algorithm will steer towards
without creating numerical difficulties in the computations. a solution that satisfies those constraints at the expense
Second, in a very real sense, the problem is dynamically of searching for the minimum. In either case,
changing with the relative position of the current value of convergence may be exceedingly slow. Discussion on
x and the subset of the constraints that are violated. The how to normalize constraints is given on barrier function
third reason why the conjecture is not correct is methods.
associated with the fact that large values of μ create
enormously steep valleys at the constraint boundaries.
Steep valleys will often present formidable if not Test problems (Testing practical examples)
insurmountable convergence difficulties for all preferred
search methods unless the algorithm starts at a point As discussed in previous sections, a number of
extremely close to the minimum being sought. algorithms are available for solving constrained nonlinear
Fortunately, there is a direct and sound strategy that will programming problems. In recent years, a variety of
overcome each of the difficulties mentioned above. All computer programs have been developed to solve
that needs to be done is to start with a relatively small engineering optimization problems. Many of these are
value of μ. The most frequently used initial penalty complex and versatile and the user needs a good
parameters in the literature are 0.01, 0.1, 2, 5, and 10. understanding of the algorithms/computer programs to be
This will assure that no steep valleys are present in the able to use them effectively. Before solving a new
initial optimization of θ(x, μk ). Subsequently, we will solve engineering design optimization problem, we usually test
a sequence of unconstrained problems with the behavior and convergence of the algorithm/computer
monotonically increasing values of μ chosen so that the program on simple test problems. Eight test problems are
solution to each new problem is “close” to the previous given in this section. All these problems have appeared in
one. This will preclude any major difficulties in finding the the optimization and on facility location literature and
minimum of θ(x, μk ) from one iteration to the next. most of them have been solved using different
techniques.

Subsequent values of the penalty parameter

Example 1
Once the initial value of the μk is chosen, the subsequent
values of μk have to be chosen such that μk+1 > μk. Consider the optimization problem:

For convenience, the value of μk is chosen according to Minimize f(x) = +

the relation: subject to -4≤0
- ≤0
μk+1 = βμk. x1 + 2x2 - 4 ≤ 0
where β > 1. The value of β can be taken as in most x1 ≥ 0
literatures 2, 5, 10,100 etc. x2 ≥ 0
Various approaches to selecting the penalty parameter
sequence exist in the literature. The simplest is to keep it We consider the sequence of problems:
constant during all iterations and we consider here the
penalty parameter as same for all constraints. = f(x) + µ[
+ ] + +
]
Normalization of the constraints
Optimum solution point using Mathematica is x =
An optimization may also become ill-conditioned when (1.67244, 1.21942) and Optimum solution is at =
the constraints have widely different magnitudes and thus 34.1797
badly affect the convergence rate during the minimization Optimum solution point using MATLAB is x =
of θ-function. Much of the success of SUMT depends on (2.000000003129, 1.000000123435) and
the approach used to solve the intermediate problems, Optimum solution is at = 34.0000050125
which in turn depends on their complexity. One thing that
should be done prior to attempting to solve a nonlinear The graph of the feasible region and steps of a computer
programming using a penalty function method is, to scale program (Deumlich, 1996) with the contours of the
the constraints so that the penalty generated by each is objective function are shown in Figure 2.
 Berhe 225

-4 -2 2 4 6

-2

-4
Figure 2. The sequence of unfeasible results from outside the
feasible region. And the iteration step using MATLAB for penalty
and the necessary data are given.

Table 1. The iteration step using MATLAB.

µ xmin fmin augmin

1.00 (-10.000000000000,-10.000000000000) 481.00 485615721.72568649
10.00 (2.055936255098, 1.977625505064) 24.847007912 65.8104676667
100.00 (2.003470085541, 1.120258484331) 32.791068788 38.7633368822
1000.00 (2.000316074133, 1.012311179827) 33.875143422 34.4986676000
10000.00 (2.000031285652, 1.001234049950) 33.987473310 34.0500992010
100000.00 (2.000003125478, 1.000123433923) 33.998746923 34.0050122192
1000000.00 (2.000000312552, 1.000012343760) 33.999874687 34.0005012538
10000000.00 (2.000000031250, 1.000001234352) 33.999987469 34.0000501229
100000000.00 ( 2.000000003129, 1.000000123435) 33.999998747 34.0000050125

And the iteration step using MATLAB for penalty method subject to - ≤0
and the necessary data are given as follows:
- ≤0
Initial: -x1 + 2x2 - 2 ≤ 0
x1 = [2; 5]; We consider the sequence of problems:
µ = 1; beta = 10; = f(x) + µ[ +
tol = 1.0e-4; tol1 = 1.0e-6; h = 0.1;N = 10 (Table 1). ]
Optimum solution point using Mathematica is x =
Example 2 (1.33271, 1.7112) and Optimum solution is at =
8.26363
Consider the optimization problem: Optimum solution point using MATLAB is x =
(1.280776520285, 1.640388354297) and
Minimize f(x) = + Optimum solution is at = 8.5235020151.
226 Afr. J. Math. Comput. Sci. Res.

-4 -2 2 4 6
Figure 3. The sequence of unfeasible results.
-2

Table 2. The iteration step using MATLAB.

µ xmin fmin augmin

1.00 (10.00000000000, 10.00000000000) 85.000000000 8249.000000000
10.00 (1.762313021963, 2.438395531128) 3.9704775728 20.8446729268
100.00 (1.376888913083, 1.774431551334) 7.5876445202 12.0187470067
1000.00 (1.291940370698, 1.655287557366) 8.4151441359 8.9534555697
10000.00 (1.281912903137, 1.641896920522) 8.5124734059 8.5675584019
100000.00 (1.280890263154, 1.640539267595) 8.5223932351 8.5279146690
1000000.00 (1.280787794313, 1.640403311676) 8.5233871397 8.5239394231
10000000.00 (1.280777545189, 1.64038971403) 8.5234865508 8.5235417778
100000000.00 (1.280776520285, 1.64038835429) 8234964917 8.5235020150

The graph of the feasible region and steps of a computer Minimize f(x) = -ln( )
program (based on Mathematica) with the contours of the subject to ≥ 0
objective function are shown in Figure 3.
And the iteration step using MATLAB for penalty 2 +3 ≤6
method and the necessary data are given as follows:

Initial: Solution

x = [10; 10]; The -function of the corresponding unconstrained

µ = 1; beta = 10; problem is:
tol = 1.0e-3; tol1 = 1.0e-5; h = 0.1; N = 10 (Table 2).

= f(x) + µ[ ]
Example 3
The Exterior penalty function method, coupled with the
Consider the optimization problem: Powell method of unconstrained minimization and golden
 Berhe 227

2.5

2
Figure 4. The sequence of unfeasible results from outside the feasible region.

1.5
Table 3. The iteration step using MATLAB.

µ 1 xmin fmin augmin

1.00000 (100.0000000000, 100.0000000000) -2.9957322736 244033.0042677264
1.50000 (1.812414390011, 0.805517491028) -0.8081555566 -0.805586944500
2.25000 (1.812414390377, 0.805517490784) -0.8081555566 -0.804302638400
3.37500
0.5
(1.812414395786, 0.805517487178) -0.8081555566 -0.802376179003
5.06250 (1.803696121437, 0.801642712659) -0.8057446020 -0.804976156000

0.5
bracket and golden search method of one-dimensional 1 tol = 1.5 2 = 1.0e-3;
1.0e-9; tol1 2.5 h = 0.1;N
3 = 10;
3.5Table 3
search, is used to solve this problem.
Optimum solution point using Mathematica is x =
(1.80125, 0.800555) and Example 4
Optimum solution is at = -0.804892
Consider the optimization problem:
Optimum solution point using MATLAB is x =
(1.803696121437, 0.801642712659) and Minimize f(x) = - 5ln( )
Optimum solution is at = -0.8057446020 subject to + -4≤0
The graph of the feasible region and steps of a computer - ≤ 0.
program (Deumlich, 1996) with the contours of the
objective function are shown in Figure 4. And the iteration We consider the sequence of problems:
step using MATLAB for penalty method and the
necessary data are given as follows: = - 5ln( ) +

Initial: µ[ ].
We can solve this problem numerically. Since the
x1 = [100; 100]; function f is not convex we can expect local minimum
µ = 1; beta = 1.5; points depending on the choice of the initial point.
228 Afr. J. Math. Comput. Sci. Res.

Optimal
point

Figure 5. The sequence of unfeasible results from outside the feasible region.

Table 4. The iteration step using MATLAB.

µ xmin fmin augmin

1.0 (2 .000000000000, 3.000000000000) -8.0471895622 0.9528104378
10.0 (0.584383413070, 4.869701406514) -11.851026032 2.8191586927
100.0 (0.491210873054, 3.912290440909) -8.9702340739 -6.6115965813
1000.0 (0.478587312848, 3.786849015552) -8.5400072779 -8.2873616072
10000.0 (0.477272005299, 3.773806675110) -8.4944671131 -8.4690191379
100000.0 (0.477139883021, 3.772497114149) -8.4898858321 -8.4873391868
1000000.0 (0.477126692178, 3.772366078499) -8.4894274297 -8.4891727436
10000000.0 (0.477125354746, 3.772352991728) -8.4893815863 -8.4893561184

Optimum solution point using Mathematica is x = Example 5

(0.472776, 3.80538) and
Optimum solution is at = -8.52761 A new facility is to be located such that the sum of its
Optimum solution point using MATLAB is x = distance from the four existing facilities is minimized. The
(0.477125354746, 3.772352991728) and four facilities are located at the points (1, 2), (-2, 4), (2, 6),
Optimum solution is at = -8.4893815863 and (-6,-3). If the coordinates of the new facility are x 1
and x2, suppose that x 1 and x2 must satisfy the
The graph of the feasible region and steps of a computer restrictions x1 + x2 = 2, x1 ≥ 0, and x2 ≥ 0.
program (Deumlich 1996) with the contours of the
objective function are shown in Figure 5. And the iteration Formulate the problem
step using MATLAB for penalty method and the Solve the problem by a penalty function method using a
necessary data are given as follows: suitable unconstrained optimization technique.

Initial: Minimize f(x) = + + +

x1 = [2; 3];
µ = 1; beta = 10;
tol = 1.0e-4; tol1 = 1.0e-6; h = 0.1;N = 10 (Table 4). subject to x1 + x2 = 2
 Berhe 229

Optimal point
Figure 6. The sequence of unfeasible results from outside the feasible region.

Table 5. The iteration step using MATLAB.

µ xmin fmin augmin

0.1 (-100000.0000000,-100000.0000000) 565688.2534900 6000645688.6535
1.0 (-0.504816443491, 2.941129889320) 5.6534514664 16.0986605310
10.0 (-0.235317980540, 2.465609185550) 15.7843894418 16.8684753525
100.0 (-0.043496067432, 2.086197027712) 16.0316892765 16.4032172656
1000.0 (-0.004877163871, 2.009622311363) 16.1014887806 16.1477919328
10000.0 (0.000981076620, 2.000347630256) 16.1105064270 16.1281610467
100000.0 (0.003030519805, 1.997102981748) 16.1136901197 16.1154723862
1000000.0 (0.003236508228, 1.996776847061) 16.1140109620 16.1141893257

x1 ≥ 0 Example 6
x2 ≥ 0
A new facility is to be located such that the sum of its
The corresponding unconstrained optimization problem distance from the four existing facilities is minimized. The
four facilities are located at the points (1, 2), (-2, 4), (2, 6),
is: and (-6,-3). If the coordinates of the new facility are x 1
and x2, suppose that x 1 and x2 must satisfy the
restrictions x1 + x2 = 2, + ≤2, - - ≤-3, x1 ≥ 0,
= f(x) + µ[ ]. and x2 ≥ 0.
Optimum solution point using Mathematica is x
= Formulate the problem
Optimum solution is at = 16.0996. Solve the problem by a penalty function method using a
Optimum solution point using MATLAB is x = suitable unconstrained optimization technique.
(0.003236508228, 1.996776847061) and
Optimum solution is at = 16.1140109620. Minimize f(x) = + +
The graph of the feasible region and steps of a computer
+
program (Deumlich 1996) with the contours of the
objective function are shown in Figure 6. subject to x1 + x2 = 2
And the iteration step using MATLAB for penalty method + ≤2
and the necessary data are given as follows: - - ≤ -3
-x1 0
Initial: -x2 ≤ 0
x = [-100000; -100000]; µ = 0.1; beta = 10;
tol = 1.0e-6; tol1 = 1.0e-3; h = 0.1;N = 10 (Table 5). The corresponding unconstrained optimization
230 Afr. J. Math. Comput. Sci. Res.

Optimal point

𝑥12+𝑥22 = 2

−𝑥12 − 2𝑥22 = -3
Figure 7. The sequence of unfeasible results from outside the feasible region.

Table 6. The iteration step using MMATLAB.

µ xmin fmin augmin

0.01 (-100.00000000000,-100.0000000000) 568.62244178640 4000376.7024418
0.10 (-0.204074540511, 2.488615905901) 15.7748455724 17.5805067419
1.00 (-0.005676255570, 1.87710924673) 16.2384983466 18.5763296826
10.0 (0.208853398412, 1.519794991416) 16.7514115057 18.7366197410
100.0 (0.541258580758, 1.337453329403) 17.2205864696 19.3598462433
1000.0 (0.774045018932, 1.191529581681) 17.7152204663 19.2571015793
10000.0 (0.894859224075, 1.097008320463) 18.0550540479 18.8928463977
100000.0 (0.951465363246, 1.046723818830) 18.2363926667 18.6484042954
1000000.0 (0.977561265017, 1.022043406130) 18.3250429734 18.5208297859
10000000.0 (0.989607061234, 1.01030727942) 18.3670763153 18.4588852580

problem is: µ = 0.01; beta = 10;

tol = 1.0e-6; tol1 = 1.0e-3; h = 0.1; N = 10; Table 6
=f(x)+ [ +

. Example 7
Optimum solution point using Mathematica is x =
{0.624988, 1.28927} and The detail of this location problem is given in example 1
Optimum solution is at = 17.579. of the barrier method.
Optimum solution point using MATLAB is x =
(0.989607061234, 1.010307279416) and Minimize
Optimum solution is at = 18.3670763153.
The graph of the feasible region and steps of a computer 3600 + 2500 +
program (Deumlich 1996) with the contours of the
+ 2200 +
objective function are shown in Figure 7.
+
And the iteration step using MATLAB for penalty method
and the necessary data are given as follows: + +
+ +
Initial:

x1 = [-100; -100]; subject to + ≤ 25

 Berhe 231

Table 7. The iteration step using MMATLAB.

µ xmin fmin augmin

10.0 (-100.000000,-100.0000000) 4201299.48370 3994823869.4837
100.0 (5.18664531200, 5.613234562710) 217725.1471 335935.8823
1000.0 (3.874734526, 3.94912345612000) 243622.3686 306218.6111
10000.0 (4.28341223, 2.646232345145123) 256672.3825 399517.2275
100000.0 (4.175642351, 0.46232345145100) 293524.5216 342434.4427
1000000.0 (4.01825125, 0.047812341001001) 302445.1718 307672.3810
10000000.0 (4.001831234, 0.00479123410010) 303387.2635 303913.3990
100000000.0 (4.000181234, 0.00047912341001) 303481.9797 303534.6277
1000000000.0 (4.000018123, 0.00004791234100) 303491.4564 303496.7216
10000000000.0 (4.0000018123, 0.0000047912340) 303492.4042 303492.9307
100000000000.0 (4.0000001812, 0.0000004790123) 303492.4989 303492.5516
100000000000.0 (4.000000018, 0.00000004812331) 303492.5084 303492.5137
10000000000000.0 (4.0000000018, 0.0000000048123) 303492.5100 303492.5099
100000000000000.0 (4.000000000185, 0.00000000048) 303492.5095 303492.5095
1000000000000000.0 (4.00000000002, 0.00000000005) 303492.5095 303492.5095

x1 + x2 = 4 (δ), and side constraints. It has four design variables

x1 – x2 = 4
-x1 ≤ 0
-x2 ≤ 0
Design vector: =
The corresponding unconstrained optimization problem
is:
Objective function: f (x) = 1.10471x 1x2 + 0.04811x3x4
= f(x) + + (14.0 + x2)
+ ].
Optimum solution point using Mathematica is x = Constraints:
(4, ) g1(x) = η(x) − ηmax ≤ 0
Optimum solution is at = 303493.0. g2(x) = ζ(x) − ζmax ≤ 0
Optimum solution point using MATLAB is x = g3(x) = x1 − x4 ≤ 0
(4.000000000018, 0.000000000048) and g4(x) = 0.10471x1 + 0.04811x3x4 (14.0 + x2) − 5.0 ≤ 0
Optimum solution is at = 303492.50947. g5 (x) = 0.125 − x1 ≤ 0
And the iteration step using MATLAB for penalty method g6 (x) = δ(x) − δmax ≤ 0
and the necessary data are given as follows: g7 (x) = P − Pc (x) ≤ 0
g8 (X) to g11 (x): 0.1 ≤ xi ≤ 2.0, i = 1, 4
Initial: g12 (x) to g15 (x): 0.1 ≤ xi ≤ 10.0, i = 2, 3
where
x = [-100; -100];
µ = 10; beta = 10;
tol = 1.0e-6; tol1 = 1.0e-6; h = 0.1; N = 20 (Table 7).

Example 8 ,
,
Here, we test the well studied welded beam design
problem, which has been solved by using a number of ,
classical optimization methods and by using Genetic
Algorithms [Deb, 128 to 129]. The welded beam is ,
designed for minimum cost subject to constraints on
shear stress in weld (η), bending stress in the beam (ζ),
buckling load on the bar (Pc), end deflection of the beam
232 Afr. J. Math. Comput. Sci. Res.

Table 8. The iteration step using MMATLAB.

µ xmin fmin augmin

0.01 (2.00000, 3.00000, 0.100000, 0.050000) 13.2606093500 10160035228635306
0.02 (0.3634872, 2.7826082, 10.558957, 0.232105300) 2.3849380 2.39159576770
0.04 (0.3634872, 2.7826082, 10.55895716, 0.2321053) 2.3849380 2.39825371460
0.08 (0.3738517, 2.8145296, 10.10980684, 0.2340900) 2.3490380 2.35157651770
0.16 (0.375852754, 2.8212375, 10.0249324, 0.234488) 2.3426482 2.34679527470

P = 6000 lb, ηmax =13,600 psi, ζmax = 30,000 psi, and δmax For the types of penalty functions considered thus far,
= 0.25 in. we have seen that we need to make the penalty
parameter infinitely large in a limiting sense to recover an
Starting and optimum solutions: optimal solution. This can cause numerical difficulties and
ill-conditioning effects. To alleviate the computational
xstart = = , fstart = 5.398 and X* = and
difficulties associated with having to take the penalty
parameter to infinity in order to recover an optimal
= $2.3810 solution to the original problem, we present below two
penalty functions that possess this property and are
Optimum solution point given by Rao (2009) is x = known as exact penalty functions. These are exact
(0.2444, 6.2177, 8.2915, 0.2444) and Optimum solution absolute value (l1 penalty function) and augmented
Lagrangian penalty function method.
is at = 2.3810. Optimum solution point using MATLAB
is x = (0.375852754, 2.8212375, 10.0249324, 0.234488)
The exact absolute value or l1 penalty function
and Optimum solution is at = 2.3467952747.
An attractive approach to nonlinear programming is to
And the iteration step using MATLAB for penalty method attempt to determine an exact penalty function by which
and the necessary data are given as follows: is meant a function defined in terms of the objective
function and constraints. This holds out the possibility
Initial:
that the solution can be found by a single application of
x1 = [2; 3; 0.1; 0.05]; an unconstrained minimization technique to , as against
µ = 0.01; beta = 2; the sequential processes described above cannot be
tol = 1.0e-2; tol1 = 1.0e-6; h = 0.1; N = 5 (Table 8). used. Consider problem (P) to minimize f(x) subject to
≤ 0, i = 1, . . . , m, and = 0, i = 1, . . . , l, and a
Using other starting point we have different solution but penalty parameter μ > 0.
the difference is not significant as given below. Roughly speaking, an exact penalty function for
problem (P) is a function , where μ > 0 is the
Initial: penalty parameter, with the property that there exists a
lower bound > 0 such that for μ ≥ any local minimizer
x1 = [0.4; 6; 0 .01; 0.05];
of (P) is also a local minimizer of the penalty problem.
µ = 0.1; beta = 2;
Exact penalty functions can be divided into two classes:
tol = 1.0e-2; tol1 = 1.0e-6; h = 0.1;N = 30 (Table 9).
continuously differentiable and non-differentiable exact
penalty functions. Continuously differentiable exact
penalty functions were introduced by Fletcher (1987) for
EXACT PENALTY FUNCTION METHODS equality constrained problems and by Gland and Polak
(1979) for problems with inequality constraints; further
In this chapter, we analyze two important extensions of contributions have been assumed in Di Pillo. Non-
the transformation methods, which are called exact differentiable exact penalty functions were introduced by
penalty functions and have been most frequently used. In Zangwill (1967); Pietrgykowski (1969). The most
these methods a single unconstrained minimization frequently used type of exact penalty function is the l1
problem, with a reasonable sized penalty parameter can exact penalty function. This function has been
yield an optimum solution to the original problem. This researched widely, for example by Pietrgykowski (1969);
suggests an algorithm which attempts to locate the Coleman and Conn (1982); in nonlinear programming
optimum value of whilst keeping µ finite and so avoids applications amongst others. Unfortunately the many
the ill-conditioning in the limit µ goes to infinity that we effective techniques for smooth minimization cannot
face in penalty function methods. adequately be used because of its non-differentiability
 Berhe 233

and the best way of using this penalty function is functions and that , i = 1, . . . , l are affine functions.
currently being researched. A more realistic approach is Then, for μ ≥ maximum { , i I, | |, i = 1, . . . , l}, also
to use this function as a criterion function to be used in minimizes the exact l1 penalized objective function
conjunction with other iterative methods for nonlinear defined by (4.1).
programming. The most satisfactory approach of all is to
apply methods of non-smooth optimization.
A class of non-differentiable exact penalty functions
n Proof
associated to (P) for X = R was analyzed by
Charalambous in 1978. It is assumed by
Since is a KKT point to (P), it is feasible to (P) and
satisfies
(x, ) = f(x) q ,

where q ≥ 1, , > 0, i = 1, . . . , m and i = 1, . . . , l. For f( ) + + = 0, ≥0 for i ∈ I

q = 1 and considering all the penalty parameters equal to (.4.2) (Moreover solves (P).)
µ; we have the l1 penalty function, introduced by
Pietrgykowski (1969), Now, consider the problem of minimizing over x
∈ Rn. This can equivalently be restated as follows, for
= f(x) μ , (4.1) any μ ≥ 0:

Where: Minimize f(x) + μ (4.3a)

p(x) =
= , is the penalty
subject to ≥ (x) and ≥ 0 for i = 1, . . . , m (4.3b)
function.

Pietrgykowski (1969), has shown that function (4.1) is ≥ (x) and ≥ (x) for i = 1, . . . , l. (4.3c)
exact in the sense that there is a finite µ > 0 such that
any regular local minimizer of (P) is also a local minimizer The equivalence follows easily by observing that for any x
of the penalized unconstrained problem. In 1970, ∈ Rn, the maximum value of the objective function in
Luenberger showed that, under convex assumptions, (4.3a), subject to (4.3b) and (4.3c), is realized by taking
there is a lower bound for µ, equal to the largest = maximum {0, } for i = 1, . . . , m and = | |
Lagrange multiplier in absolute value, associated to the for i = 1, . . . , l. In particular, given , define =
nonlinear problem. In 1978, Charalambous generalized maximum {0, } for i = 1, …, m and = | | 0 for i
the result of Luenberger for the l1 penalty function (4.1), = 1, …, l.
assuming the second-order sufficient conditions for (P).
The following result shows that, under suitable convexity
assumptions, there does exist a finite value of μ that will Note that, of the inequalities ≥ (x), i = 1, . . . , m, only
recover an optimum solution to (P) via the minimization those corresponding to i ∈ I are binding, while all the
of . Alternatively, it can be shown that if satisfies the other inequalities in (4.3) are binding at ( ). Hence,
second-order sufficiency conditions for a local minimum for ( ) to be a KKT point for (4.3), we must find
of (P) (the Hessian is positive definite). Then, for μ at Lagrangian multipliers , , i = 1, . . . , m, and , , i
least as large as the theorem below, will also be a local = 1, . . . , l, associated with the respective pairs of
minimum of . constraints in (4.3b) and (4.3c) such that

f( ) + + = 0,
Theorem 4 μ = 0 for i = 1, . . . , m,
μ = 0 for i = 1, . . . , l,
Consider the following primal problem: , ≥ 0 for i = 1, . . . , m,
Minimize f(x) , ) ≥ 0 for i = 1, . . . , l, = 0 for i I.
subject to g(x) 0
h(x) = 0. (P) Assumed that μ ≥ , i ∈ I, | |, i = 1, . . . , l},
we then have, using (4.2), that = for all i ∈ I, =0
Let be a KKT point with Lagrangian multipliers , i ∈ I , for i ≠ I, =μ- for all i = 1, . . . , m, and =
and , i = 1, . . . , l associated with the inequality and
equality constraints, respectively, where I = {i ∈ {1, . . . , and = for i = 1, . . . , l satisfy the forgoing KKT
m} : ( ) = 0} is the index set of active constraints. conditions. By stated convexity assumptions, it follows
that ( ) solves (4.3), and, so, minimizes . This
Furthermore, suppose that f and , i I are convex completes the proof. We proof it as follows in detail:
234 Afr. J. Math. Comput. Sci. Res.

Lemma 5 Example 1

Suppose (P) is a convex program for which the Karush- Minimize +

Kuhn-Tucker conditions are necessary. Suppose that subject to + -1 = 0

p(x) = + . = is the KKT point with the Lagrangian multiplier

associated with this point is found as:
Then as long as µ is chosen sufficiently large, the sets of
optimal solutions of and (P) coincide. In fact, it + = 0, which follows that 2 + = 0 and 2
suffices to choose µ > maximum , i = 1, . . . , + = 0.
l}, where ( , ) is a vector of Karush-Kuhn-Tucker
multipliers. Equating the two we have = -2 = -2( ) = -1.
The function defined by (4.1) for μ ≥ 0 is:
+ + μ| + -1|.
Proof
If μ = 0, is minimized at (0, 0). For μ > 0,
Suppose solves (P). For any x Rn we have: minimizing is equivalent to:

(x, µ) = f(x) + µ Minimizing + + μz

≥ f(x) + + subject to -z + + - 1 ≤ 0 ( )
≥ f(x) + + -z – – +1≤0

For ( , ) to be a KKT point for ( ) above, we must

≥f(x)+ + find Lagrange multipliers , associated with the
) respective constraints such that:
= f(x) + ( )
= f(x) - ( ) ≥ f( ) = ( )
= f( ) + µ =
µ =0
( , µ). (-z + + -1) = 0
Thus ( , µ) ≤ (x, µ) for all x, and therefore solves (-z – – + 1) = 0
(x, µ). and, moreover, optimality dictates that z = | + -1|.
Next suppose that solves . Then if solves (P),
we have: Now let us consider the cases,

f( )+µ ≤f( )+ Case 1: if ( + ) < 1, then

= f( ) (-z + + -1) = 0, from this since -z + +
and so -1 < 0.
f( ) ≤ f( ) - µ .( ) And, hence ( ) is:

However, if is not feasible for (P), then =

μ= =
f( ) f( ) ( ) 2 – and
= f( ) - )- ) 2 - =0
≥ f( ) + - ) +
It follows that = = .
= f( ) + -
> f( ) - , This is a KKT point, provided that 0 ≤ μ < 1.

which contradicts ( ). Thus is feasible for (P). That Case 2: if + = 1, then z = | + -1| = 0.
being the case,
By ( )
f( ) ≤ f( ) - µ = f( )
=
from (4.3.1.) and so solves (P). Therefore they have the
same optimal value. μ = 0, then
 Berhe 235

2 = =2 = = = . are not only practically important in their own right, but

have also served as the starting point for a chain of
From this we have; research developments centering around the use of
penalty functions, Lagrange multiplier iterations, and
=μ–[ ] with and = . Newton's method for solving the system of necessary
This is a KKT point, provided that μ 1. optimality conditions. Again, the motivation here is to
avoid the ill-conditioning difficulties encountered by the
Case 3: if ( + ) > 1, so that ( ) is: classical approach as the penalty parameter approaches
to infinity.
+ = = 0, For simplicity, let us begin by discussing the case with
only equality constraints, for which augmented
Lagrangians are first introduced, and then readily extend
which implies that = = , and the discussion to include inequality constraints as well.
that + = -μ > 1, a
contradiction to μ > 0. Consequently, as μ increases from ALAG penalty function for equality constrained
0, the minimum of occurs at ( ) until μ reaches the problems
value 1, after which it remains at ( , ), which is the
Consider Problem (P) of minimizing f(x) subject to hi(x) =
optimum to the original problem.
0 for i = 1, . . . , l. we have seen if we employ the
quadratic penalty function problem to minimize f(x) +
Augmented lagrangian penalty function (ALAG) , then we typically need to µ to obtain a
constrained minimum for (P). We might then be curious
As we have seen in the above discussion, most “smooth” whether, if we were to shift the origin of the penalty term
penalty functions (such as quadratic penalty function) to = ( , i = l, . . . , l) and consider the penalized
never generate exact solutions to the constrained objective function f(x) + with respect to
minimization problem. Therefore, we would need to solve the problem in which the constraint right-hand sides are
the (penalized) unconstrained problems with very large perturbed to from 0, it can be shown (Theorem 4.1
values of the constant µ in order to obtain solutions that below) that if the Lagrange multipliers are fixed at their
are close to being feasible and optimal. (In theory, we optimum values , the minimization of (x, v, µ)
need to let µ → ∞ to obtain a solution.) This is gives the solution of the original problem (P) in one step
unfortunate, since the unconstrained optimization for any value of µ. In such a case there is no need to
problems one encounters in implementing penalty minimize the function for an increasing sequence of
methods tend to become ill-conditioned when µ values of µ. In expanded form, this latter objective
increases, and therefore, it will be hard to solve each of function is
the unconstrained problems required by the algorithm.
Alternatively, one could employ an exact penalty method, f(x) – + + .
that is, a method that guarantees termination at an Denoting = - for i = 1, . . . , l and dropping the
optimal solution provided that the value of µ is sufficiently final constant term (independent of x), this can be written
large (but finite). As we have established, linear penalty as
function is an exact penalty function; unfortunately, it is
not differentiable at points at the boundary of the feasible
(x, v) = f(x) + + , (4.4)
region, and therefore poses difficulties in solving
corresponding unconstrained problems. l
Motivated by our discussion of exact penalty functions, where v R is some vector of multipliers, that can be
it is natural to raise the question whether we can design a either kept constant or updated as we proceed with the
penalty function that not only recovers an exact optimum penalty algorithm. (Compare this to the usual Lagrangian
for finite penalty parameter values but also enjoys the function L(x, v) = f(x) + .) The usage of this
property of being differentiable. The Augmented function as a penalty function can be partially motivated
Lagrangian Penalty Function (ALAG), also known as the by the following observation: suppose that is the
multiplier penalty function, is one such exact penalty optimal solution of (P), and is the vector of
function. This approach uses both a Lagrangian multiplier corresponding multipliers. Taking the (partial) gradient of
term and a penalty term in the auxiliary function. This the function , we obtain
approach was independently proposed by Hestenes
(1969); Powell (1997). The original proposal of this = 2µ =0 (4.5)
method may be viewed as a significant milestone in the
recent history of the constrained optimization area. As For all values of ; whereas this was not necessary the
described by Hestenes, augmented Lagrangian methods case with the quadratic penalty function, unless was
236 Afr. J. Math. Comput. Sci. Res.

itself zero. Hence, whereas we need to take µ to ]

recover in a limiting sense using the quadratic penalty = +2 (4.8)
function, it is possible that we only need to make µ large
enough (under suitable regularity conditions as where is the Hessian of the Lagrangian function
enunciated below) for the critical point of (., ) to for (P) with a multiplier vector at x = . From the
turn out to be its (local) minimizer. In this respect, the last second-order sufficiency conditions, we know that
term in (4.5) turns out to be a local convexifier of the is positive definite on the cone
overall function.
Observe that the function (4.5) is the ordinary C = {d ≠ 0 : d = 0 for i = 1, . . . , l}.
Lagrangian function augmented by the quadratic penalty
term; hence the name augmented Lagrangian penalty Now, on the contrary, if there does not exist a such that
function. Accordingly, (4.5) can be viewed as the usual G( ) is positive definite for μ ≥ , then it must be the case
quadratic penalty function with respect to the following that, given any = k, k = 1, . . . , l, there exists a with
problem that is equivalent to (P): || || = 1 such that
Minimize {f(x) + : 1, l}. (4.6) G( ) = + 2k ≤ 0. (4.9)
Alternatively, (4.4) can be vied as a Lagrangian function Since, || || = 1 for all k, there exists a convergent
for the following problem which is equivalent to (P):
subsequence for { } with limit point , where || || = 1.
Over this subsequence, since the first term in (4.9)
Minimize {f(x) + : 1, l}. (4.7)
approaches L( ) , a constant, we must have
= 0 for all i = 1, . . . , l for (4.9) to hold for all k.
inclusion of a “multiplier based term” in the quadratic
Hence, C. Moreover, since ≤ 0 for all k by
penalty objective function; it is also sometimes called a
multiplier penalty function. These view points lead to a (4.9), we have ≤ 0. This contradicts the
reach theory and algorithmic felicity that is not present in second-order sufficiency conditions. Consequently, G( )
the pure quadratic penalty function. is positive definite for µ exceeding some value , and so,
The following result provides the basis by virtue of is a strict local minimum for FALAG(., ).
which the ALAG penalty function can be classified as an Finally, suppose that f is convex and hi are affine, and
exact penalty function. Namely, if the vector of multipliers is optimal to (P). There exists a set of Lagrange
is known, one can hope that under some regularity multipliers such that ( ) is a KKT solution. As before,
assumptions, the point is the local minimizer of FALAG(x, we have = 0, and since for FALAG(. , ) is
) for large (but finite) values of µ. convex for any µ ≥ 0, this completes the proof.
We remark here that without the second-order
sufficiency conditions of Theorem 4.3, there might not
Theorem 6 (ALAG Theorem) exist any finite value µ that will recover an optimum for
problem (P), and it might be that we need to take µ
Consider problem (P) to minimize f(x) subject to hi(x) = 0 for this to occur. The following example from (1987)
illustrates this point.
for i = 1, . . . , l, and let the KKT solution ( ) satisfy the
second‐ order sufficiency conditions for a local minimum
(the Hessian is positive definite.) Then, there exists a
Example 2
such that for μ ≥ , FALAG(., ) also achieves a strict local
minimum at . In particular, if f is convex and hi are affine, Consider the following optimization problem:
then any minimizing solution for (P) also minimizes
FALAG(., ) for all μ ≥ 0. Minimize f(x) = 4
+
subject to =0

Proof = is the optimal solution. From the KKT

conditions, + = 0 and we get = 0 as the
Since ( ) is a KKT solution, we have, from (4.5), that unique Lagrange multiplier. Note that:
= 0.
L( ) = f( ) + = f( ). Then,
Furthermore, letting G( ) denote the Hessian of FALAG( . , = and
) at x = , we have
=
G( ) = + + 2µ + = =H
 Berhe 237

The eigenvalues of H are found by solving: |H- | = 0, subject to x + = 10

With Or . Therefore is indefinite.
This shows the second-order sufficiency condition does using KKT conditions, it is easy to compute the optimal
not hold at ( ). Now, consider solutions as follows (This is computed using
Mathematica):
( ) = ( ) = f( ) + +
KTSolution[ + x ,{ x + -10= 0},{x,y}];
4 2
= + +0+µ . Lagrangian→ + x + [-10+ x + ]
Note that for any µ > 0 Valid KT point(s)
f → 50
,
x → 10
vanishes at = and = . Furthermore, y→0 → -10
Optimum: x = 10, y = 0 Lagrange multiplier, v = -10
= and = ,
In the augmented Lagrangian approach, the
unconstrained function is defined by adding exterior
is indefinite and, hence, is not a local minimizer for any penalty term to the Lagrangian of the original problem.
µ > 0. Hence worth it is assumed that second order Thus we have the following unconstrained function.
sufficient conditions hold and µ is sufficiently large.
+ x + v(x + -10) + µ
However,
The necessary conditions for the minimum of this function
= , give the following equations:

The eigenvalues of are all positive for µ > 0

which shows that is positive definite, and is
in fact the minimizer of for all µ > 0. Moreover, as µ If v is set to the optimum value of the Lagrange multiplier,
→ ∞, approaches the constrained minimum for problem we get the following equations:
(P).
It is demonstrated in the following examples that if the
optimum Lagrange multipliers are known, then the
solution of this unconstrained problem corresponds to the
solution of the original problem regardless of the value of The second equation can be written as follows:
the penalty parameter.
y( ) = 0.

Example 3 Thus, y = 0 satisfies this equation for any value µ.

Substituting y = 0 in the first equation, we get
Consider the optimization problem (P) in example 4.1.
= , with = -1 is the unique KKT point and
optimum for this problem. Furthermore, = is or
positive definite, and thus second-order sufficiency ( )(1 + 2µ) = 0.
condition holds at ( , ). Moreover from equation (4.4),
2 2 Thus, x = 10 satisfies this equation. Thus, the Lagrangian
( )= + –( + -1) + penalty function has the property that the optimum
= + + + , solution of the original problem is recovered, if we know
the optimum values of the Lagrange multipliers.
which is clearly uniquely minimized at = for all µ
Therefore, in this sense it is an exact penalty function.
≥ 0. Hence, both assertions of Theorem 4.3 are verified. Obviously, when we are solving a problem we don’t
know the optimum Lagrange multipliers. (If they were
known we wouldn’t need to spend time in developing new
Example 4 algorithms. We could simply use them with the KKT
Consider the following optimization problem: conditions to get a solution). However, the presence of
Lagrange multipliers makes the choice of penalty
parameter less critical. In a computational procedure
Minimize f(x) = +x
based on the augmented Lagrangian penalty function
238 Afr. J. Math. Comput. Sci. Res.

method, we start with arbitrary values of Lagrange Although, there are different algorithms to solve this kind
multipliers and develop a procedure that moves the of problems the algorithm due to Powell (1997) is given
Lagrange multipliers closer to their optimum values. below and ensures global convergence. The outline of
Thus, near the optimum, the function is not as sensitive such an algorithm is as follows.
to the value of and the procedure converges to the true
optimum.
Therefore, to make use of the above result, one Algorithm 1: Algorithm for ALAG with equality
attempts to estimate the multipliers by updating the constraints
vector v after solving each (or some) unconstrained
minimizations of FALAG. The outline of such an algorithm Initialization: Select some initial Lagrangian multipliers v
is given in the following section. = usually 0 and positive values μ1, . . . , μl for the
penalty parameters. Let x o be a null vector, and denote
VIOL(xo) = ∞, where for any x ∈ Rn, VIOL(x) =
Schema of an algorithm using augmented Lagrangian
maximum{|hi(x)| : i = 1, . . . , l} is a measure of constraint
penalty functions violations. Put k = 1 and proceed to the "inner loop" of the
algorithm.
Method of multipliers
Inner loop (Penalty function minimization): Solve
The method of multipliers is an approach for solving n
minimize FALAG(x, ) subject to x ∈ R and let xk denote
nonlinear programming problems by using the
the optimal solution obtained. If VIOL (xk) = 0, stop with xk
augmented Lagrangian penalty function in a manner that
as a KKT point. (Practically, one would terminate if VIOL
combines the algorithmic aspects of both Lagrangian
(xk) is less than some tolerance > 0). Otherwise, if VIOL
duality methods and penalty function methods. However,
(xk) ≤ 0.25VIOL (xk‐ 1), proceed to the outer loop. On the
this is accomplished while gaining from both these
other hand, if VIOL (xk) > 0.25VIOL (xk‐ 1) then, for each
concepts without being impaired by their respective
constraint i = 1, . . . , l for which |hi(xk)| >0.25VIOL(xk‐ 1),
shortcomings. The method adopts a dual ascent step
replace the corresponding penalty parameter μi by 10μi
similar to the sub-gradient optimization scheme for
and repeat this inner loop step.
optimizing the Lagrangian dual; but, unlike the latter
approach, the overall procedure produces both primal
and dual solutions. The primal solution is produced via a
Outer loop (Lagrange multiplier update): Replace
penalty function minimization; but because of the
properties of the ALAG penalty function, this can usually by ,
be accomplished without having to make the penalty Where,
parameter infinitely large and, hence, having to contend
with the accompanying ill-conditioning effects. Moreover, = +2 for i = 1, . . . , l. (4.10)
we can employ efficient derivative based methods in
minimizing the penalized objective function. The Increment k by 1, and return to the inner loop.
fundamental scheme of this algorithm is as follows.
The inner loop of the forgoing method is concerned with
the minimization of the augmented Lagrangian penalty
Schema of the algorithm for equality constraints function. For this purpose, we can use xk‐ 1 (for k ≥ 2) as a
starting solution and employ Newton’s method (with line
Consider the problem of minimizing f(x) subject to the searches) in case the Hessian is available, or else use a
equality constraints hi(x) = 0 for i = 1, . . . , l. (The quasi-Newton method if only gradients are available, or
extension to include inequality constraints is relatively use some conjugate gradient method for relatively large-
straight forward and is addressed in the following scale problems. If VIOL (xk) = 0, then xk is feasible, and,
subsection). Below, we outline the procedure first, and moreover,
then provide some interpretations, motivations, and
implementation comments. As is typically the case, the = + +2 =0
augmented Lagrangian function employed is of the form (4.11)
(4.4), except that each constraint is assigned its own
specific penalty parameter , instead of a common implies that a KKT point. Whenever the revised
parameter µ. Hence, constraint violations, and iterate of the inner loop does not improve the measure
consequent penalizations, can be individually monitored. for constraint violations by selected factor 0.25, the
Accordingly, we replace (4.4) by penalty parameter is increased by a factor of 10. Hence,
the outer loop will be visited after a finite number of
(x, v) = f(x) + + iterations when the tolerance is used in the inner
 Berhe 239

loop, since, as in Theorem 2.2 we have → 0 as Example 4.5

→ ∞ for i = 1, . . . , l.
Observe that the forgoing argument holds regardless of Consider the optimization problem of Example 4.1. Given
the dual multiplier update scheme used in the outer loop, any v, the inner loop of the method of multipliers
and that it is essentially related to using the standard evaluates = ,
2 2
quadratic penalty function approach on the equivalent Where = + + v( + -1) +
problem (4.6). In fact, if we adopt this view point, then the .
Lagrange multiplier estimate associated with the Solving = 0 yields
constraints in (4.6) is assumed by 2 for i = 1, . . . , x(v) = . (The KKT point)
l, as (2.8). Since the relationship between the Lagrange The outer loop then updates the Lagrangian multiplier
multipliers of the original problem (P) and its primal according to
equivalent from (4.6) with v = is that the Lagrange = +2
multiplier vector for (P) equals plus the Lagrange
which gives = v + 2µ[ (v) + (v) -1] = . Note
multiplier vector for (4.6), equation (4.10) then gives the
corresponding estimate for the Lagrange multiplier that, as µ → ∞, → -1, the optimal Lagrange
associated with the constraints for (P). multiplier value.
This observation can be reinforced more directly by the Hence, if we start the algorithm with = 0, and µ = 1, the
following interpretation. Note that having inner loop will determine
minimized , we have (4.11) holding true.
However, for and to be a KKT solution, we want x(0) = = with VIOL[x(0)] = + –1= -1
= 0, where L(x, v) = f(x) + is the = ,
Lagrangian function for (P). Hence, we can choose to
revise to in a manner such that and the outer loop will find = 0 + 2(1)( ) = . Next,
at the second iteration, the inner loop solution will be
+ = 0. obtained as

Super imposing this identity on (4.11), we get

x(v) = = =
= , x( ) = with VIOL(x( )) = h[x( )] = >

which follows that ( )( ) with VIOL(x(1)) > VIOL(x(0)).

+ +2 = + Hence we will increase µ to 10, and recompute the

revised

by eliminating like terms, we get x( ) = = = with VIOL(x( )) = .

The outer loop will then revise the Lagrange multiplier =
=
to = + 2(10) ( ) = .
From this,
The iteration will progress in this fashion, using the
forgoing formulas, until the constraint violation at the
= + 2 which is the update scheme in inner loop solution is acceptably small.
(4.10).

Hence, from the view point of problem (4.6), convergence ALAG penalty function for problems with mixed
is obtained above in one of the two ways. First, we might constraints
finitely determine a KKT point as is frequently the case.
Alternatively, viewing the forgoing algorithm as a one of Consider problem (P) to minimize f(x) subject to the
applying the standard quadratic penalty function constraints gi(x) ≤ 0 for i = 1, . . . , m and hi(x) = 0 for i =
approach, in sprit, to the equivalent sequence of 1, . . . , l (Bhatti (2000). The extension of the forgoing
problems of the type (4.6), each having particular theory of augmented Lagrangians and the method of
estimates of the Lagrangian multipliers in the objective multipliers to this case, which also includes inequality
function, convergence is achieved by letting the penalty constraints, is readily accomplished by equivalently
parameters approach infinity. In the latter case, the inner writing the inequalities as the equations gi(x) + = 0 for i
loop problems become increasingly ill-conditioned and = 1, . . . , m. Now suppose that is a KKT point for
second-order methods become imperative. problem (P)
240 Afr. J. Math. Comput. Sci. Res.

with optimal Lagrange multipliers , i = 1, . . . , m, and , 2( )(2 ) = ) = 0, i = 1, . .

i = 1, . . . , l, associated with the inequality and equality
. , m.
constraints, respectively, and such that the strict
complementary slackness condition holds, namely, that
gi( ) = 0 for all i = 1, . . . , m, with > 0 for each i ( ) These conditions state that either
= {i : gi( ) = 0}. Furthermore, suppose that the second-
order sufficiency condition holds at ( ), namely, that = 0 Or = 0 which follows
is positive definite over the cone = 0 Or =-( ) ≥ 0.

C = {d ≠ 0 : = 0 for i
= 1, . . . , l}. Using this can be written as:

Then it can be verified that the conditions of Theorem 4.3

are satisfied for problem to minimize f(x) subject to =
gi(x) + = 0 for i = 1, . . . , m, and hi(x) = 0 for i = 1, . . . , = , say. (4.14)
l, at the solution ( , ), where = - gi(x) for i = 1, . . .
, m. Hence, for µ large enough, the solution ( , ) will turn Similar to (4.14), the function is sometimes
out to be a strict local minimizer for the following ALAG referred to as the ALAG penalty function itself in the
penalty function at (u, v) = ( , ): presence of both equality and inequality constraints. In
particular, in the context of the method of multipliers, the
f(x) + + inner loop evaluates , measures the constraint
µ . (4.12) violations, and revises the penalty parameter(s) in an
identical fashion as before.
The augmented Lagrangian function defined in (4.12) In order the augmented Lagrangian penalty function to
includes slack variables for inequality constraints. Their solve constrained optimization problems, we need to
presence increases the number of variables in the determine a procedure that, starting from arbitrary values,
problem. It is possible to remove these variables by leads to near optimum values of Lagrange multipliers. A
writing the necessary conditions for the minimum with simple procedure is based on comparing the necessary
respect to s. Before proceeding it is convenient to conditions for the minimum of the Lagrangian function
combine the two terms involving the slack variables by and the augmented Lagrangian penalty function for
noting that; problem only with equality constraint. In the presence of
inequality constraints, the above analysis does not work
µ =µ . out as clearly as for the equality case. In practice, the
following rule based on similarity with the equality is
Rearranging the terms gives, adopted:

µ + ( )=µ - . = + maximum .
If minimizes (4.14), then the sub-gradient component
For a given penalty parameter µ > 0, let represent to at (u, v) = ( , ) is found at
the minimum of (4.12) over (x, s) for any given set of = 2µ – 2
Lagrange multipliers (u, v). Now let us rewrite (4.12) more and is
conveniently as follows:
- .

f(x) + µ - + + Adopting the fixed step length of 2µ along this sub-

gradient direction as for the equality constraint case
]. (4.13) revises to
= + 2µ[ - ]
Hence, in computing , we can minimize (4.13) over
(x, s) by first minimizing [ ] over in terms = 0 + maximum
= + maximum for i = 1, . . . , m. (4.15)
of x for each i = 1, . . . , m and then minimizing the
resulting expression over the x Rn. The former task is
To start the process, arbitrary values, usually zero, are
accomplished by writing the necessary conditions for the
assigned to all multipliers. Also, the multiplier updating is
minimum of with respect to the slack variables, we get done only after a substantial decrease in constraint
violation is achieved. The following algorithm from [6] is
= 0 implies that used in most literatures.
 Berhe 241

Algorithm 2: Algorithm for ALAG with mixed SUMMARY AND CONCLUSION

constraints The intent of this section is to point out some of the key
points discussed in the previous chapters; and based on
Set iteration counter k = 0. Set multipliers = 0, i = 1, …, that it aims to draw some conclusions.
m and = 0, i = 1,…,l. Set multiplier update counter l = 0. As discussed in the previous sections all algorithms for
Choose a penalty parameter µ and a factor > 1 to constrained optimization are unreliable to a degree. This
increase the penalty parameter value during iterations. fact also holds true in the penalty and function methods.
Typically µ = 10 and = 2. Penalty methods are among the most powerful class of
Set up the unconstrained minimization problem. algorithms available for attacking general nonlinear
optimization problems. This statement is supported by
the fact that these techniques will converge to at least a
local minimum in most cases, regardless of the convexity
= characteristics of the objective function and constraints.
They work well even in the presence of cusps and similar
use a suitable unconstrained minimization problem to find anomalies that can stymie other approaches. Penalty
the minimum. The derivative of with respect to are methods approximate a constrained problem that assigns
evaluated according to the following: high cost to points that are far from the feasible region.
As the approximation is made more exact (by letting) the
= + 2µ + 2µ + + penalty parameter μ tend to infinity) the solution of the
2µ , j = 1, . . . , n. unconstrained penalty problem approaches the solution
to the original constrained problem from outside of the
active constraints. This method is not used in cases
Check for convergence: A simple convergence criterion is
where feasibility must be maintained, for example, if the
to stop if all constraints are satisfied and the objective
objective function is undefined or ill-conditioned outside
function is not changing much between successive
the feasible region.
iterations. Thus, stop if the following two criteria are
Penalty methods are quite different than other
satisfied. Otherwise continue to step (iv).
algorithms that they are not iterative in nature. The
definition of in no way depends on that of . From
If Abs[( )] <
this point of view, if one decides to terminate the
VIOL(xk) < sequence at the Nth term corresponding to μN, obtaining
VIOL(xk) = maximum{Abs(hi(x), i = 1, . . . , l), xN, the calculation of the previous vectors x 1, x2, …, xN-1 is
} is the maximum constraint violation. irrelevant, since x N could have been calculated directly by
Update the multiplier and the penalty parameter: solving a single unconstrained problem. Indeed, this is
the generally the manner that penalty functions are
If VIOL(xk) ≤ 0.25VIOL(xk‐ 1) then update the multipliers employed; one selects a large value of μ, solves the
= + maximum i = 1, …, m unconstrained problem, and takes the resulting solution
= +2 as the final approximate answer to the original problem. It
Set l = l + 1. is sometimes recognized, however, that selecting a single
Else update the penalty parameter large value of μ can lead to difficulty. First, exactly what is
a large value relative to a given problem may not be
= 10μk . known in advance and consequently an initial trial may
produce a solution point that is not close enough to the
(v) Update the iteration counter k = k +1 and go back to
feasible region in which case μ must be increased.
step (ii).
Second, large values of μ yield, as shown in the above
The ALAG has several advantages. As stated earlier, the sections, ill-conditioned Hessians which in turn imply slow
penalty parameter need not be increased to infinity for convergence for many algorithms.
convergence. The starting design vector, x 0, need not be A partial remedy to these difficulties is obtained by
feasible. Finally, it is possible to achieve = 0 and noting that the search for can be initiated from , a
= 0 precisely and the nonzero values of the starting point that may be fairly close to . Solution of
Lagrange multipliers ≠ 0) identify the active the k +1th problem will then probably require less time
constraints automatically. It is to be noted the function than if the search were initiated from an arbitrary point x0.
, assumed by (4.14), is continuous and has For this reason the penalty methods are often regarded
continuous first derivatives but has discontinuous second as truly iterative algorithms. It has never been
determined, however, that solving a sequence of
derivatives with respect to x at = - . Hence, a
unconstrained problems for increasing value of μ leads to
second-order methods cannot be used to minimize the a computational saving over just solving the
function (Rao, 2009). corresponding to the largest value of μ directly. Indeed,
242 Afr. J. Math. Comput. Sci. Res.

indications are that it does not. The Hessian matrix of (x, Also, his warmest and honorable thanks also go to his
) becomes increasingly ill-conditioned as µ → ∞ and the best friend, Abreha Hailezgi who motivated and told him
minimization becomes more difficult. That's why the about his potential, and contributed a lot for the success
parameter µ should not be increased too quickly and the of this research paper.
previous iterate should be used as a starting point. As μ Finally, he thanks all his friends who have helped him
→ ∞ the Hessian (at the solution) is equal to the sum of directly or indirectly in this endeavor, especially those
L, the Hessian of the Lagrangian associated with the who love him more.
original constrained problem, and a matrix of rank r that
tends to infinity (where r is the number of active
constraints). This is the fundamental property of these Some notations
methods.
Though penalty functions are old methods for solving The following notations are frequently appearing in this
constrained optimization problems, it is, nevertheless, research:
worthy of noticing to recognize the wrong assumption and
generalization that everything which is old method is µ = Penalty parameter.
nonsense. We have to be very careful not to trivialize old x = (x1, x2, x3, …, xn) is n-dimensional vector.
methods for solving constrained optimization problems θ(x, µ) = Unconstrained representation of the primal
and erroneously assume it to be as synonymous to problem (P).
backwardness, as some might misconceive it. In fact, this θ(µ) = is the infimum of θ(x, µ) with respect to x.
sequential methods needs to be modified in one way or xµ = A minimum point of θ(µ).
another so that they would serve for the ever-changing X = A nonempty set in Rn.
and growing demands of algorithms for certain M(f, S) = Set of minimum points of the constrained
optimization problems. Though these methods suffer optimization problem (P).
from some computational disadvantages, in the absence M(f, X) = Set of minimum points of the unconstrained
of alternative software especially for no-derivative optimization problem θ(x, µ).
problems they are still recommended. They work well for FALAG = Augmented Lagrangian Penalty Function.
zero order methods like Powell’s method with some p(x) = Penalty function.
modifications and taking different initial points and LM( ) = L restricted to the subspace M that is tangent to
monotonically increasing parameters. the constraint surface.
Finally, In spite of their great initial success, their slow
rates of convergence due to ill-conditioning of the
REFERENCES
associated Hessian led researchers to pursue other
approaches. With the advent of interior point methods for Bazaraa MS, Sherali HD, Shetty CM (2006). Nonlinear Programming:
linear programming, algorithm designers have taken a Theory and Algorithms, Second Edition, John Wiley & Sons, New
fresh look at penalty methods and have been able to York. pp. 469-500.
Belegundu AD, Chandrupatla TR (1999). Optimization concepts and
achieve much greater efficiency than previously thought Applications in Engineering 2nd edition, Pensylvania State University,
possible (Nash and Sofer, 1993). pp. 278-290.
Exact transformation methods are newer and less well- Bhatti MA (2000). Practical Optimization Methods with Mathematica
established as sequential transformation methods and Applications, Department of Civil and Environmental Engineering
University of Iowa, Springer-Verlag New York, Inc. pp. 512-680.
are called the newly established modern penalty Charalambous CA (1978). Lower Bound for the controling parametres
methods. Exact transformation methods avoid this long of exact panalty functions, Mathematical Programming, 15:278-290.
sequence by constructing penalty functions that are exact Coleman TF, Conn AR (1982). Nonlinear Programming Via an exact
in the sense that the solution of the penalty problem penalty function: Asymptotic analysis, Mathematical programming,
pp.123-136
yields the exact solution to the original problem for a finite Deumlich R (1996). A course in Mathematica, Addis Ababa University,
value of the penalty parameter. However, it can be shown Faculty of science, Department of Mathematics. pp.1-140
that such exact functions are not differentiable in most Fiacco AV, McCormick GP (1968). Extensions of SUMT for nonlinear
cases. Great consideration should be assumed to the programming: Equality constraints and extrapolation. Manage. Sci.
12(11):816-828.
convexity assumption and second-order conditions in Fletcher R (1987). Practical Methods of Optimization, Second Edition,
using these methods. John Wiley & Sons, New York. pp. 277-318.
Gland ST, Polak E (1979). A multiplier method with Authomatic
Limitation of penalty growth. Math. Programming,17:140-155
Hestenes MR (1969). Multiplier and gradient methods. J. Optim. Theory
ACKNOWLEDGEMENTS Appl. 4(5):123-136.
Himmelblau DH (1972). Applied Nonlinear Programming, New York,
First, the author wishes to acknowledge his family, McGraw-Hill, pp. 342-355.
especially his parents for their unconditional love and Kiusalaas J (2005). Numerical Methods in Engineering with MATLAB,
the Pennsylvania State University, and Cambridge University Press
faith in him since his birth, without whose support and Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore,
encouragement, this would not have been a reality. São Paulo. pp. 391-404.
 Berhe 243

Luenberger DG (1974). A combined penalty function and Gradient Zangwill WI (1967). Nonlinear programming via Penalty Functions.
projection method for nonlinear programming. J. Opt. Appl. 14:5. Manage. Sci. 13(5):344-358.
Luenberger DG (1984). Linear and Nonlinear Programming, 2nd ed.,
Addison-Wesley Publishing Company, Reading, MA. pp. 401-430.
Nash SG, Sofer A (1993). Linear and Nonlinear Programming, McGraw
Hill, New York. pp. 469-765.
Pietrgykowski T (1969). An exact potential method for constrained
maxima, SIAM J. Num. Anal. 6:217-238.
Powell MJD (1997). A fast algorithm for nonlinearity constrained
optimization calculations, in Lecture Notes in Mathematics, Watson
GA et al., Eds., Springer-Verlag, Berlin. pp. 343-357.
Rao SS (2009). Engineering Optimization: Theory and Practice, Fourth
Edition, John Wiley & Sons, Inc. pp. 248-318.
244 Afr. J. Math. Comput. Sci. Res.

Appendix k=64746.022*(1-0.0282346*x(3))*x(3)*x(4).^3;
m =(2.1952./(x(3).^3*x(4)));
General description of the penalty function algorithm t1=(6000./(sqrt(2)*x(1)*x(2)));
t2=6000*(14+0.5*x(2))*sqrt(0.25*(x(2).^2+(x(1)+x(3)).^2));
The SUMT iteration involves updating the penalty t3=2*(0.707*x(1)*x(2)*((x(2).^2/12)+0.25*(x(1)+x(3)).^2));
parameters and initial design vector and calling the t=t2./t3;
unconstrained problem again. In the algorithm Powell’s T=sqrt((t1).^2+(t).^2+((x(2)*t1*t)./sqrt(0.25*(x(2).^2+(x(1)+
method (which is the zero order method) together with x(3)).^2))));
golden-bracket and golden-section method for line c=[T-13600;l-30000;x(1)-x(4);6000-k;m-0.25;-
minimization is used. The program expects the following x(1)+0.125;x(1)-10;-x(2)+0.1;x(2)-10;...
files to be available in the path -x(3)+0.1;x(3)-10;-x(4)+0.1;x(4)-10];
ceq =[]; % no equality constraints.
i. Objective function,
ii. Equality and inequality constraints together, function z = unconweldedbeam(x,miw) %
iii. Unconstrained function, The corresponding unconstrained problem
iv. The flines function (for a line search). l=(504000./((x(3)).^2*x(4)));
k=64746.022*(1-0.0282346*x(3))*x(3)*x(4).^3;
For each an iteration of the penalty method there is an m =(2.1952./(x(3).^3*x(4)));
inner iteration of the Powell’s method. t1=(6000./(sqrt(2)*x(1)*x(2)));
The program uses global statements to communicate t2=6000*(14+0.5*x(2))*sqrt(0.25*(x(2).^2+(x(1)+x(3)).^2));
penalty parameters, initial point, search direction (V), t3=2*(0.707*x(1)*x(2)*((x(2).^2/12)+0.25*(x(1)+x(3)).^2));
whereas the initial penalty parameters, initial design t=t2./t3;
variable, the number of iterations for penalty method, T=sqrt((t1).^2+(t).^2+((x(2)*t1*t)./sqrt(0.25*(x(2).^2+(x(1)+
tolerances for the penalty and Powell’s method are given x(3)).^2))));
by user automatically. z=obweldedbeam(x)+miw*(max(0,T-13600)).^2
Several parameters are coded into the program, +miw*(max(0,l-30000)).^2+...
especially those needed for golden bracket and golden miw*(max(0,x(1)-x(4))).^2 +miw*(max(0,6000-
section methods. k)).^2+miw*(max(0,m-0.25)).^2 +miw*(max(0,-
Step 0: (Initialization) Choose x 0, number of SUMT x(1)+0.125)).^2+...
iterations (N), penalty parameter (µ), and penalty miw*(max(0,x(1)-10)).^2 +miw*(max(0,-
x(2)+0.1)).^2+miw*(max(0,x(2)-10)).^2 +miw*(max(0,-
multiplier ( ), tolerance for the penalty method (tol1) and
for the Powell’s method (tol). x(3)+0.1)).^2+...
miw*(max(0,x(3)-10)).^2 +miw*(max(0,-
k = 1 (SUMT iteration counter)
Step 1: Start the Powell’s method to minimize f(x, µ) x(4)+0.1)).^2+miw*(max(0,x(4)-10)).^2;
Output xk.
Step 3: Convergence of exterior penalty method. The MATLAB Code for Penalty Function Method:
Stopping criteria:
= - , = - . Function penaltyfunction
If ≤ : stop (they have approximately the same % Penalty function method for minimizing f(x1,x2, ..., xn).
solution) % Example for Logarithmic function on Example 8.
else if ≤ : stop (design not changing)
else if k = : stop (max SUMT iteration reached) % input:
continue % tol and tol1 are error tolerances for Powell’s method
k k+1 and penalty method respectively.
% x = starting point (vector).
= % µ = the penalty parameter.
go to step 2 % beta = the penalty multiplier.
% N = number of iterations for the penalty method, we
Input for the welded beam example given in example 8 of choose it depending on the problem.
penalty function method. % h = initial step size used in search for golden bracket.
% output:
function f = obweldedbeam(x) % objective function % xmin = minimum point.
f=1.10471*x(1).^2*x(2)+0.04811*x(3)*x(4)*(14+x(2)); % objmin = miminum value of objective function.
% augmin = minimum of the corresponding unconstrained
function [c,ceq] = conwelededbeam(x) % constraints problem

l=(504000./((x(3)).^2*x(4))); % globals (must be declared global in calling program).

 Berhe 245

% V = search direction, the same as the unit vectors in method in the Kth iteration
the coordinate directions. µ=beta*µ;
sqrt(dot(f - obj,(f- obj));
% Starting of the program. if sqrt(dot(f - obj, - obj)) < tol1
clc; % clears the screen. return
clear all; % clears all values of variables for memory end
advantage. end % end of SUMT iteration.
global x µ V
x = [0.4; 6; 0.01; 0.05]; % f in the direction of coordinate axes.
µ = 0.1; beta = 2;
tol = 1.0e-2; tol1 = 1.0e-6; h = 0.1;N = 30; function z = flines(s) % f in the search direction V
if size(x,2) > 1; x = x'; end % x must be column vector global x µ V
n = length(x); % Number of design variables z = feval(@unconweldedbeam,x+s*V,µ);
df = zeros(n,1); % Decreases of f stored here % Start of golden bracketing for the minimum.
u = eye (n); % Columns of u store search directions V
disp(sprintf(' µ xmin objmin augmin ')) function [a,b] = goldbracket(func,x1,h)
disp(sprintf(' ------ ------------------ ------------ --------------- ')) % Brackets the minimum point of f(x).
for k=1:N % loop for the penalty function method % Usage: [a,b] = goldbracket(func,xstart,h)
[c,ceq]= conwelededbeam(x);
obj= obweldedbeam(x); % input:
f= unconweldedbeam(x,µ); % func = handle of function that returns f(x).
disp(sprintf('%1.5f (%3.12f,%3.12f) %2.10f %2.10f ',µ,x, % x1 = starting value of x.
obj,f)) % h = initial step size used in search.
for j = 1:30 % Allow up to 30 cycles for Powell’s method % c = a constant factor used to increase the step size h
xold = x;
fold = feval(@unconweldedbeam,xold,µ); % output:
% First n line searches record the decrease of f % a, b = limits on x at the minimum point.
for i = 1:n c = 1.618033989;
V = u(1:n,i); f1 = feval(func,x1);
[a,b] = goldbracket(@fline,0.0,h); x2 = x1 + h; f2 = feval(func,x2);
[s,fmin] = goldsearch(@fline,a,b); % Determine downhill direction and change sign of h if
df(i) = fold - fmin; needed.
fold = fmin; if f2 > f1
x = x + s*V; h = -h;
end x2 = x1 + h; f2 = feval(func,x2);
% Last line search in the cycle % Check if minimum is between x1 - h and x1 + h
V = x - xold; if f2 > f1
[a,b] = goldbracket(@fline,0.0,h); a = x2; b = x1 - h; return
[s,fmin] = goldsearch(@fline,a,b); end
x = x + s*V; end
if sqrt(dot(x-xold,x-xold)/n) < tol % Search loop for the minimum
y = x; % assign the solution to y for i = 1:100
end h = c*h;
% Identify biggest decrease of f & update search x3 = x2 + h; f3 = feval(func,x3);
directions if f3 > f2
imax = 1; dfmax = df(1); a = x1; b = x3; return
for i = 2:n end
if df(i) > dfmax x1 = x2; x2 = x3; f2 = f3;
imax= i; dfmax = df(i); end
end error('goldbracket did not find a minimum please try
end another starting point')
for i = imax:n-1
u(1:n,i) = u(1:n,i+1); % Start of golden search for the minimum.
end function [xmin,fmin] = goldsearch(func,a,b,tol2)
u(1:n,n) = V; % Golden section search for the minimum of f(x).
end % end of Powell’s method % The minimum point must be bracketed in a <= x <= b.
x=y; % y is the minimum point found using Powell’s % usage: [fmin,xmin] = goldsearch(func,xstart,h).
246 Afr. J. Math. Comput. Sci. Res.

% input: else
% func = handle of function that returns f(x). b = x2; x2 = x1; f2 = f1;
% a, b = limits of the interval containing the minimum. x1 = R*a + C*b;
% tol2 = error tolerance used in golden section. f1 = feval(func,x1);
end
% output: end
% fmin = minimum value of f(x). if f1 < f2; fmin = f1; xmin = x1;
% xmin = value of x at the minimum point. else
if nargin < 4; tol2 = 1.0e-6; end fmin = f2; xmin = x2;
nIter = ceil(-2.078087*log(tol2/abs(b-a))); end
R = 0.618033989; % R is called golden ratio.
C = 1.0 - R;
% First telescoping
x1 = R*a + C*b;
x2 = C*a + R*b;
f1 = feval(func,x1);
f2 = feval(func,x2);
% Main loop
for i =1:nIter
if f1 > f2
a = x1; x1 = x2; f1 = f2;
x2 = C*a + R*b;
f2 = feval(func,x2);

View publication stats