1

UNIT – I
Introduction to Optimization:

Statement of an Optimization problem: design vector, design constraint,

constraint surface, objective function, objective function surfaces,
Classification of Optimization problems, Classical Optimization Techniques:
Single variable Optimization, Multivariable Optimization.

2
 BOOKS
TEXT BOOKS:

1) “Engineering Optimization- Theory and Practice” by Third Edition, SINGIRESU S. RAO, A Wiley-
Interscience Publication John Wiley & Sons, Inc. 1996

2) "Optimal State Estimation Kalman, H- infinity, and Nonlinear Approaches" by Dan Simon, John
Wiley & Sons,2006.

3) Xin-She Yang, “Nature-Inspired Optimization Algorithms”, Elsevier, 1st edition, 2014.

4) "Introduction to Metaheuristics" by Fred Glover and Gary A. Kochenberger

5) "Introduction to Operations Research" by Frederick S. Hillier and Gerald J. Lieberman

REFERENCE BOOKS:

1) "Optimal Estimation of Dynamic Systems" by John L. Crassidis and John L. Junkins, CRC Press, 2004.

2) "Operations Research: An Introduction" by Taha Hamdy A.

3) "Numerical Optimization" by Jorge Nocedal and Stephen J. Wright

4) "Convex Optimization" by Stephen Boyd and Lieven Vandenberghe

5) "Machine Learning Algorithms" by Giuseppe Bonaccorso , 2017

3
INTRODUCTION TO OPTIMIZATION
• Optimization is the act of obtaining the best result under given circumstances.
• In design, construction, and maintenance of any engineering system,
engineers have to take many technological and managerial decisions at
several stages.
• The ultimate goal of all such decisions is either to minimize the effort
required or to maximize the desired benefit.
• Since the effort required or the benefit desired in any practical situation can
be expressed as a function of certain decision variables, optimization can be
defined as the process of finding the conditions that give the maximum or
minimum value of a function.
• It can be seen from Fig. 1.1 that if a point
x* corresponds to the minimum value of
function f(x), the same point also
corresponds to the maximum value of the
negative of the function, —f(x).
• Thus, without loss of generality,
optimization can be taken to mean
minimization since the maximum of a
function can be found by seeking the
minimum of the negative of the same
function.
• There is no single method available for
solving all optimization problems
efficiently.
• Hence a number of optimization methods
have been developed for solving different
types of optimization problems.
• The optimum seeking methods are also known as mathematical programming techniques and are generally studied as a part of
operations research. Operations research is a branch of mathematics concerned with the application of scientific methods and
techniques to decision making problems and with establishing the best or optimal solutions.
• Mathematical programming techniques are useful in finding the
minimum of a function of several variables under a prescribed set of
constraints.
• Stochastic process techniques can be used to analyze problems
described by a set of random variables having known probability
distributions.
• Statistical methods enable one to analyze the experimental data and
build empirical models to obtain the most accurate representation of
the physical situation.
 ENGINEERING APPLICATIONS OF OPTIMIZATION
• Design of aircraft and aerospace structures for minimum weight
• Finding the optimal trajectories of space vehicles
• Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost
• Minimum-weight design of structures for earthquake, wind, and
other types of random loading
• Design of water resources systems for maximum benefit
• Optimal plastic design of structures
• Optimum design of linkages, cams, gears, machine tools, and other
mechanical components
• Selection of machining conditions in metal-cutting processes for
minimum production cost
• Design of material handling equipment such as conveyors, trucks,
and cranes for minimum cost
• Design of pumps, turbines, and heat transfer equipment for
maximum efficiency
• Optimum design of electrical machinery such as motors, generators,
and transformers
• Optimum design of electrical networks
• Shortest route taken by a salesperson visiting various cities during
one tour
• Optimal production planning, controlling, and scheduling.
12

Statement of an optimization problem

 x1 
 
To find X=
 x2  which maximizes f(X)
 . 
 
 . 
x 
 n 

Subject to the constraints

gi(X) <= 0 , i = 1, 2,….,m

lj(X) = 0 , j = 1, 2,….,p
13

Statement of an optimization problem…

where
 X is an n-dimensional vector called the design vector
 f(X) is called the objective function
 gi(X) and lj(X) are inequality and equality constraints, respectively.
This type of problem is called a constrained optimization problem.

Optimization problems can be defined without any constraints as well

Unconstrained optimization problems.
15

Introduction - Preliminaries
Basic components of an optimization problem :
 An objective function expresses the main aim of the model which is
either to be minimized or maximized.

 Set of unknowns or variables which control the value of the objective

function.

 Set of constraints that allow the unknowns to take on certain values

but exclude others.

Optimization problem is then to:

Find values of the variables that minimize or maximize the objective

function while satisfying the constraints.
16

• Many optimization problems have a single objective function

• The two interesting exceptions are:

• No objective function: User does not particularly want to optimize anything

so there is no reason to define an objective function. Usually called a
feasibility problem.

• Multiple objective functions. In practice, problems with multiple objectives

are reformulated as single-objective problems by either forming a weighted
combination of the different objectives or by treating some of the objectives
by constraints.
19

Variables and Constraints

 Variables

 These are essential. If there are no variables, we cannot define the

objective function and the problem constraints.

 Constraints

 Even though constraints are not essential, it has been argued that

almost all problems really do have constraints.

 In many practical problems, one cannot choose the design variable

arbitrarily. Design constraints are restrictions that must be satisfied

to produce an acceptable design.
20

Constraints (contd.)
• Constraints can be broadly classified as :

• Behavioral or Functional constraints : Represent limitations on the

behavior and performance of the system.

• Geometric or Side constraints : Represent physical limitations on

design variables such as availability, fabricability, and transportability.
21

Constraint Surfaces
 Consider the optimization problem presented earlier with only inequality

constraints gi(X) . Set of values of X that satisfy the equation gi(X) forms a

boundary surface in the design space called a constraint surface.

 Constraint surface divides the design space into two regions: one with gi(X) < 0

(feasible region) and the other in which gi(X) > 0 (infeasible region). The points

lying on the hyper surface will satisfy gi(X) =0.

22

The figure shows a hypothetical two-dimensional design

space where the feasible region is denoted by hatched lines

Behavior
Infeasible constraint
region g2  0

Side
constraint Feasible region
Behavior
g3 ≥ 0 .
constraint
. g1 0
Bound
acceptable point.

Free acceptable point

Free unacceptable Bound
point unacceptable
point.
23

Objective Function Surface

• If the locus of all points satisfying f(X) = a constant c is considered, it can
form a family of surfaces in the design space called the objective function
surfaces.

• When drawn with the constraint surfaces as shown in the figure we can
identify the optimum point (maxima).

• Possible graphically only when the number of design variable is two.

• When we have three or more design variables because of complexity in the

objective function surface we have to solve the problem as a mathematical
problem and this visualization is not possible.
24

Objective function surfaces to find the

optimum point (maxima)
36

Stationary Points: Functions of

Single and Two Variables
37

Stationary points
➢ For a continuous and differentiable function f(x) a stationary
point x* is a point at which the function vanishes,

i.e. f ’(x) = 0 at x = x*. x* belongs to its domain of definition.

➢ Stationary point may be a minimum, maximum or an

inflection (saddle) point
 Stationary points…

f (x ) = 0
f (x )  0,
f (x )  0 f (x )  0
f (x )  0

f (x )  0,
f (x ) = 0, f (x )  0 f (x )  0 f (x )  0
f (x ) = 0 f (x ) = 0

(a) Inflection point (b) Minimum (c) Maximum

39

Relative and Global Optimum

• A function is said to have a relative or local minimum at x = x* if

for all sufficiently small positive and negative values of h, i.e. in the near
f ( x *
)  f ( x + h)
vicinity of the point x.

• A point x* is called a relative or local maximum if

for all values of h sufficiently close to zero.

f ( x* )  f ( x + h)
• A function is said to have a global or absolute minimum at x = x* if for all x in the
domain over which f(x) is defined.

( x )  isf (said
*
• Af function x) to have a global or absolute maximum at x = x* if for all x in the
domain over which f (x) is defined.

f ( x* )  f ( x )
40

Relative and Global Optimum…

A1, A2, A3 = Relative maxima
A2 = Global maximum
B1, B2 = Relative minima
B1 = Global minimum

.
A2
Relative minimum is
also global optimum
f(x)

. .
A3
f(x)

A1
.
B2

. .
B1
x
a b x a b
41

Functions of a single variable

➢ Consider the function f(x) defined for a  x  b

➢ To find the value of x*  [ a, b] such that x* maximizes f(x) we need to

solve a single-variable optimization problem.

42

Functions of a single variable …

➢ Necessary condition : For a single variable function f(x) defined for x
 [ a, b] which has a relative maximum at x = x* , x* [ a, b] if the
derivative f ‘(x) = df(x)/dx exists as a finite number at x = x* then
f ‘(x*) = 0.
➢ Above theorem holds good for relative minimum as well.
➢ Theorem only considers a domain where the function is continuous and
derivative.
➢ It does not indicate the outcome if a maxima or minima exists at a point
where the derivative fails to exist. This scenario is shown in the figure
below, where the slopes m1 and m2 at the point of a maxima are unequal,
hence cannot be found as depicted by the theorem.
43

Functions of a single variable ….

Salient features:
⚫ Theorem does not consider if the
maxima or minima occurs at the
end point of the interval of
definition.
⚫ Theorem does not say that the
function will have a maximum or
minimum at every point where
f ’(x) = 0, since this condition
f ’(x) = 0 is for stationary points
which include inflection points
which do not mean a maxima or
a minima.
44

Sufficient condition
• For the same function stated above let f ’(x*) = f ”(x*) = . . . = f (n-1)(x*) =

0, but f (n)(x*) ≠ 0, then it can be said that f (x*) is

• (a) a minimum value of f (x) if f (n)(x*) > 0 and n is even

• (b) a maximum value of f (x) if f (n)(x*) < 0 and n is even

• (c) neither a maximum or a minimum if n is odd

45

Example 1
Find the optimum value of the function f ( x) = x 2 + 3x − 5
and also state if the function attains a maximum or a
minimum.
Solution
f '( x) = 2 x + 3 = 0
For maxima or minima,
or x* = -3/2
f ''( x*) = 2
is positive .
Hence the point x* = -3/2 is a point of minima and the function attains a
minimum value of -29/4 at this point.
46

Example 2
Find the optimum value of the function f ( x) = ( x − 2) 4 and also state if
the function attains a maximum or a minimum

Solution:
f '( x) = 4( x − 2)3 = 0 or x = x* = 2 for maxima or minima.

f ''( x*) = 12( x * −2) 2 = 0 at x* = 2

f '''( x*) = 24( x * −2) = 0 at x* = 2

( )
f  x * = 24 at x* = 2

Hence fn(x) is positive and n is even hence the point x = x* = 2 is a point of

minimum and the function attains a minimum value of 0 at this point.

More examples can be found in the lecture notes

Thank you