FENG 346

NUMERICAL METHODS FOR ENGINEERS II

CONSTRAINED AND NONLINEAR
OPTIMIZATION PROBLEMS
𝑓 (𝑥 , 𝑦)

Asst. Prof. M. ÖZGÜN GÜLEÇ

Izmir University of Economics
Mechatronics Engineering

0507 705 8644

[email protected]
OUTLINE

•Equality Constraints
 Substitution Method

 Lagrange Multipliers Method

•Inequality Constraints
 The Variate Transformation Method

 Karush-Kuhn-Tucker (KKT) Necessary Conditions

2
Constrained Optimization Problems

In most optimization problems, the optimization parameters (design

parameters) must provide additional conditions while making objective
function extremum. These conditions are given to the optimization problem
as constrained equations.

Constrained equations can be in equality or inequality form, and these are

called equality constraints and inequality constraints, respectively.

3
Constrained Optimization Problems

A feasible region is a region that satisfies all the constraints of the problem.

Equality constraint Inequality constraint

4
Equality Constrains

• In the single variable design parameter case, the boundary conditions are easy to check because there
are only two boundaries.
• In multivariable design parameter case, there are infinitely many solutions on the boundary.
Remember that the feasible region is only on the boundary because the equality constraint restricts it
5
 Equality Constrains

Suppose that is the objective function and dependent on number of design variables. Hence, it is
degrees optimization problem.

Then, there are number of equality constraints, expressed via the function .
Remember that .

Several methods can be used for equality constrained optimization problems, and the method is selected
according to the type of problem.

6
 Substitution Method

In some problems, equality constraint equations consist of dependent variables. Then,

one variable can be written as a function of the other variable. This function can also be
embedded in the objective function to eliminate the related constraint from the
optimization problem.

Hence, the equality constrained optimization problem is converted to an unconstrained

one.

7
Substitution Method

Example : Find the maximum area of the rectangle inside a triangle as a function of , as shown in the
figure below.

Objective function : Maximize

Equality constraint function :

After solving Eqn (2) for , it is substituted to Eqn (1). Hence, both the
Number of equality constraints = 1
variable and constraint are eliminated from the optimization problem Degrees of the optimization problem
and the constraint is guaranteed.

So, the problem is converted to the unconstrained optimization one. 8

Substitution Method

Stationary Points; 

Substitute into Eqn 2. 

İt is non-zero and negative  it is a local maximum

9
Lagrange Multipliers Method

In case some variables cannot be written as a function of others (means that

variables are independent from each other) it will be impossible to use
substitution method. Hence, the Lagrange Multipliers method becomes an
alternative way.

The method is not only used in optimization problems but also in

constrained problems of dynamic.

10
Lagrange Multipliers Method

There is a point where the gradient of the objective function

and the gradient of the equality constraint function are parallel.
That point is also the extremum of the function on the
boundary.

In other words, at the extremum point on the boundary created

by the constraint, the gradient vector of the objective function
and gradient vector of the constraint equation become parallel.

∇ 𝑓 /¿ ∇ 𝑔
Notations : The constraint equation notations are in some books while it is
or in others. 11
Lagrange Multipliers Method

Although these two gradients are parallel, they may not be in same direction or size. Hence, a
constant / scaler multiplier can define their linear relationship. This multiplier is also called
Lagrange Multiplier.

There exists n number of equations with n number of variables, so there are finite number of
solutions. Then the global extremum (minimum or maximum) can be find easily. 12
Lagrange Multipliers Method

13
Lagrange Multipliers Method
Example 1
∇ 𝑓 /¿ ∇ 𝑔
Objective Constraint
function g

Number of equality constraints = 1

Degrees of the optimization problem
14
 Lagrange Multipliers Method
Objective Constraint
function g

∇ 𝑓 =¿ ∇ 𝑔=¿

(1)

(2) 3 Eqns. 3 Unkonws

(3) (constraint equation)

Substitute eqn. 2 into eqn. 1

  not an ellipse

 Substitute it into Eqn 3 

15
 Lagrange Multipliers Method
Objective
function Minimums

 Max.
 Min.
 Min.
 Max.

Maximums

16
 Number of equality constraints = 1
Lagrange Multipliers Method Degrees of the optimization problem

Example 2 : Find the maximum area of the rectangle inside a triangle as a function of , as shown in the
figure below. (This problem was solved via substitution method)

Objective function : Maximize

Equality constraint function :

If . We can define a new function 

Solve eqn. 4 for y. And, solve eqn.

5 for x. Then substitute them into
eqn. 2 (constraint equation)

17
 Lagrange Multipliers Method
Rectangular Prism Volume

Example 3 :Find the dimensions of the rectangular prism with the largest
volume that can be fitted into the ellipsoid

Optimization Parameters / Design variables

𝑥 , 𝑦 , 𝑧 : 𝐶𝑜𝑜𝑟𝑑𝑖𝑛𝑎𝑡𝑒𝑠 𝑜𝑓 𝑡h𝑒 𝑝𝑜𝑖𝑛𝑡 𝑃
Objective Function : Maximize (Merit Function)
• Single Objective Function
¿ S=min ⁡(− 8 𝑥𝑦𝑧 ) • Nonlinear
• Constrained (equality)
Constraints : equality
Number of equality constraints = 1
Degrees of the optimization problem
18
Lagrange Multipliers Method
Rectangular Prism Volume

4 equations, 4 unknowns

Add them up  

19
1
Lagrange Multipliers Method
Rectangular Prism Volume

Solve Eqn 5 for ,

then substitute 
into eqn. 8.

Objective Function : Maximize (Merit Function)

20
 • Single Objective Function
Lagrange Multipliers Method • Nonlinear
• Constrained (equality)
Example 4 :Find the closest and the farthest points and their distances between the origin of the Oxyz and the
cross-section of the ellipsoid and surface

Optimization Parameters / Design variables

𝑥 , 𝑦 , 𝑧 :𝐶𝑜𝑜𝑟𝑑𝑖𝑛𝑎𝑡𝑒𝑠 𝑜𝑓 𝑡h𝑒 𝑝𝑜𝑖𝑛𝑡 𝑃

Objective Function : Maximize and minimize

Constraints : 2 equality constraints

Number of equality constraints = 2

Degrees of the optimization problem
21
Lagrange Multipliers Method

5 equations
5 unknowns



22
Lagrange Multipliers Method

23
 • Single Objective Function
Lagrange Multipliers Method • Nonlinear
• Constrained (equality)
Example 5 :Minimize the function
Subject to an equality constraint
Number of equality constraints = 1
Degrees of the optimization problem 2

24
Lagrange Multipliers Method

25
 • Single Objective Function
Lagrange Multipliers Method • Nonlinear
Cylindrical Tank Design • Constrained (equality)

Example 6 :The tank is closed at both ends and is required to have volume The radius and height are
selected as design variables. It is desired to design the tank having minimum surface area.

Number of equality constraints = 1

Degrees of the optimization problem 2

26
Lagrange Multipliers Method
Cylindrical Tank Design

27
Inequality Constraints

Suppose that is the objective function and dependent on number of design variables. Hence, it is
degrees optimization problem.

Then, there are number of inequality constraints, expressed via the function .

Several methods can be used for inequality constrained optimization problems, and the method is
selected according to the type of problem.

28
 Inequality Constraints

Variate Transformation Method

If the inequality constraints are simple expressions, the variate transformation method can eliminate
them from the optimization problem. Thus, the problem is transformed into an equality constraint or
unconstrained form.
Some Example Variate Transformations for Inequality Constraints

29
 Inequality Constraints
Variate Transformation Method

Example : Minimize the function

Subject to the inequality constraints

30
Inequality Constraints
Variate Transformation Method

31
Inequality Constraints
Variate Transformation Method

32
Inequality Constraints
Variate Transformation Method

33
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

Standard form of the Method.

All objective functions must All constraints are in the

be minimization problem. form of inequality and all
The maximization problems inequality constraints must John Forbes Nash Jr. is a
are transformed to be transformed to friend of Harold W. Kuhn
minimization problems. constraints and doctoral student of
Albert W. Tucker.
Let's add a slack-variable to the left side of the constraints to convert them to
equality constraints.
34
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

All inequality constraints are transformed to the equality constraints. Now, it is possible to continue with
Lagrange Multipliers method.

35
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

both sides

Solve Eqn. 2 for . Then Substitute it into Eqn. 3

36
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

1) If Constraint is active

The inequality constraint restricts the region in which there

would be an extremum if the related constraint did not exist. In
other words, there are an extremum but the constaint restricts it.
In this case, the solution occurs on the baundary of the infeasible
region.

2) If Constraint is inactive

The constraint already restricts a region in which the extremum will not be exist. In this case, the solution
is not on the baundary of the infeasible region.
37
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

Final additional necessary condition of the method

Proof
both sides and sum

the equations from

This results with;

However. We know that, as the values becomes higher, the

feasible region will be extended.It means that, the
probability of finding lower increases. Thus, these are
inversely proportional. Hence, we can directly write that; 38
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

In Summary, The Karush- Kuhn-Tucker Necessary Conditions

Step 1 - Find the possible solutions : These

are the steps of the method and gives the
possible solution set.

Step 2 - Conditions Check : Those who pass the

condition tests form the optimum solution set
consisting of local minimums.

Step 3 – Global Minimum: Compare the results of the optimum solutions in the
objective function. Then find the global minimum one.
39
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

Example : Let’s solve same problem with KKT – Necessary Conditions Method
Minimize the function subject to the inequality constraints
Reminder for KKT
Necessary Conditions

Transform the
problem into
standard form

40
 Reminder for KKT
Inequality Constraints Necessary Conditions
Karush-Kuhn-Tucker (KKT) Necessary Conditions

Step 1 - Find the possible solutions

Step 2 - Conditions Check

41
 Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

1) µ1= µ2 =µ 3= 0 Assume that all constraints are inactive. It gives Then condition 6 gives . The
option is elected via condition 6. It means that at least one constraint is active. The minimum is not out of
all 3 constraints. No solution.

2) µ1 ≠ 0 , µ2 =µ 3= 0 Assume that the first constraint is active and others are inactive. It gives Then
condition 6 gives . The option is elected via condition 6. It means that not only first constraint is active. The
minimum is not on only first constraint’s baundary. No solution 42
 Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

3) µ3 ≠ 0 , µ1 =µ 2= 0 Assume that the only third constraint is active and others are inactive. It gives;

They pass all The solution is

 necessary  only on the 
conditions third boundary.

43
 Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

4) µ2 ≠ 0 , µ1 =µ 3= 0 Assume that the second constraint is active and others are inactive. It gives Then
condition 6 gives . The option is elected via condition 6. It means that not only second constraint is active.
The minimum is not on only first constraint’s baundary. No solution
5) µ2 ≠ 0 , µ3 ≠ 0 , µ1= 0 Assume that only first constraint is inactive and others are active. It gives . The
option is elected via condition 8 because of negative Lagrange Multiplier. It means that constraints 2 and 3
are not active together. The minimum is not on both second and third constraint’s baundaries. No solution
44
 Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

6) µ1 ≠ 0 , µ3 ≠ 0 , µ 2= 0 Assume that only second constraint is inactive and others are active. It gives . The
option is elected via condition 7 because of negative Lagrange Multiplier. It means that constraints 1 and 3
are not active together. The minimum is not on both first and third constraint’s baundaries. No solution

7) µ1 ≠ 0 , µ2 ≠ 0 , µ 3= 0 Assume that only third constraint is inactive and others are active. It gives Then
condition 6 gives . The option is elected via condition 6. It means that constraints 1 and 3 are not active
together. The minimum is not on both first and second constraint’s baundaries. No solution
45
 Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

8) µ1 ≠ 0 , µ2 ≠ 0 , µ 3 ≠ 0 Assume that all constraints are active. It gives which is not possible. No solution

Hence, only solution exists in 3. option. There is

only one minimum. It means that it is a global
minimum of the problem

 Global Minimum
46
Inequality Constraints
Karush-Kuhn-Tucker (KKT) Necessary Conditions

Graphical Representation of the Problem

As shown in graph, only third constraint is active.
µ𝟑 ≠ 𝟎 , µ 𝟏= µ𝟐 =𝟎

The result is also same with the result of the Variate

Transformation method. KKT is also longer way for this
problem. However, Variate Transformation method cannot
be adaptable to all types of inequality constraint problems.

Global Minimum :
47