Introduction to Optimization

g
a rin
rjy ee
at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
Rajib Kumar Bhattacharjya

.R m
of ion
Pr zat

PT
Professor
i
im
Department of Civil Engineering
pt
O

IIT Guwahati
Email: [email protected]

14 July 2021 R.K. Bhattacharjya/CE/IITG 1

7/14/2021
O
pt
im
i
Pr zat
of ion
.R m
N ajib eth
PT o
EL Ku ds
, I ma fo
IT r B r C
G h iv
Objective function is defined as

uw at il
ah tac eng
at ha in
i rjy ee
a rin
g
Minimization/Maximization f(x)
Single variable optimization

2
 Single variable optimization

g
Stationary points

a rin
rjy ee
at ha in
For a continuous and differentiable function 𝑓(𝑥), a stationary point 𝑥 ∗ is a

ah tac eng
uw at il
point at which the slope of the function is zero, i.e.𝑓 ′ 𝑥 = 0 at 𝑥 = 𝑥 ∗ ,

G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
.R m
of ion
Pr zat

PT
i
im
pt
O

Minima Maxima Inflection point

7/14/2021 R.K. Bhattacharjya/CE/IITG 3

 Global minimum and maximum

A function is said to have a global or absolute

minimum at 𝑥 = 𝑥 ∗ if 𝑓(𝑥 ∗ ) ≤ 𝑓(𝑥) for all 𝑥 in the

g
a rin
rjy ee
domain over which 𝑓(𝑥) is defined.

at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
.R m
of ion
A function is said to have a global or absolute

Pr zat

PT
maximum at 𝑥 = 𝑥 ∗ if 𝑓(𝑥 ∗ ) ≥ 𝑓(𝑥) for all 𝑥 in
i
im
pt
O
the domain over which 𝑓(𝑥) is defined.

7/14/2021 R.K. Bhattacharjya/CE/IITG 4

 Global minimum and maximum
Global Maxima

Local Maxima
Local Maxima

g
a rin
rjy ee
Local Maxima

at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
Local Minima

o
N ajib eth
Local Minima Local Minima

.R m
Global Minima

of ion
Pr zat

PT
Global Minima

i
im
pt
O

1 𝑥
𝑓= 2𝑥 4 − 7𝑥 2 − 𝑥+6 𝑓 = sin 𝑥 𝑒𝑥𝑝 −
2 5
−1.5 ≤ 𝑥 ≤ 1.5 −3.2𝜋 ≤ 𝑥 ≤ 3.2𝜋

7/14/2021 R.K. Bhattacharjya/CE/IITG 5

7/14/2021
O
pt
im
i
Pr zat
of ion
.R m
N ajib eth
PT o
EL Ku ds
, I ma fo
f

IT r B r C
G h iv
uw at il
ah tac eng
at ha in
i rjy ee
a rin
g
X
Global optima
Introduction to optimization

Local optima

Local optima

Local optima
Local optima

6
 Necessary and sufficient conditions for optimality

Necessary condition
If a function 𝑓(𝑥) is defined in the interval 𝑎 ≤ 𝑥 ≤ 𝑏 and has a relative minimum at 𝑥 = 𝑥 ∗ , Where
𝑎 ≤ 𝑥 ∗ ≤ 𝑏 and if 𝑓 / 𝑥 exists as a finite number at 𝑥 = 𝑥 ∗ , then 𝑓 / 𝑥 ∗ = 0

g
a rin
rjy ee
at ha in
ah tac eng
Proof

uw at il
G h iv
IT r B r C

i
, I ma fo
𝑓 𝑥 ∗ + ℎ − 𝑓(𝑥 ∗ )

EL Ku ds
𝑓/ 𝑥∗ = lim

o
ℎ

N ajib eth
ℎ→0

.R m
Since 𝑥 ∗ is a relative minimum 𝑓(𝑥 ∗ ) ≤ 𝑓 𝑥 ∗ + ℎ

of ion
Pr zat

PT
i
im
For all values of ℎ sufficiently close to zero, hence
pt
O

𝑓 𝑥 ∗ + ℎ − 𝑓(𝑥 ∗ )
≥0 if ℎ ≥ 0
ℎ
𝑓 𝑥 ∗ + ℎ − 𝑓(𝑥 ∗ )
≤0 if ℎ ≤ 0
ℎ
7/14/2021 R.K. Bhattacharjya/CE/IITG 7
 Necessary and sufficient conditions for optimality
Thus
𝑓/ 𝑥∗ ≥ 0 If ℎ tends to zero through +ve value

g
𝑓/ 𝑥∗ ≤ 0

a rin
If ℎ tends to zero through -ve value

rjy ee
at ha in
ah tac eng
Thus only way to satisfy both the conditions is to have

uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
/ ∗
𝒇 𝒙 =𝟎

.R m
of ion
Pr zat

PT
i
im
Note: O
pt
• This theorem can be proved if 𝑥 ∗ is a relative maximum
• Derivative must exist at 𝑥 ∗
• The theorem does not say what happens if a minimum or maximum occurs at an end point of
the interval of the function
• It may be an inflection point also.
R.K. Bhattacharjya/CE/IITG
 Sufficient conditions for optimality

Sufficient condition
Suppose at point 𝑥 ∗ , the first derivative is zero and first nonzero higher

g
a rin
rjy ee
derivative is denoted by 𝑛, then

at ha in
ah tac eng
uw at il
1. If 𝑛 is odd, 𝑥 ∗ is an inflection point

G h iv
IT r B r C

i
, I ma fo
𝑓′ 𝑥∗ = 0

EL Ku ds
o
N ajib eth
𝑓 ′′ 𝑥 ∗ = 0

.R m
2. If 𝑛 is even, 𝑥 ∗ is a local optimum

of ion
Pr zat

PT
𝑓3 𝑥∗ = 0
 If the derivative is positive, 𝑥∗
is a local minimum
i
im
pt
 If the derivative is negative, 𝑥 ∗ is a local maximum
O
𝑓4 𝑥∗ = 0

𝑓𝑛 𝑥∗ ≠ 0

7/14/2021 R.K. Bhattacharjya/CE/IITG 9

 Sufficient conditions for optimality
Proof Apply Taylor’s series

ℎ2 ′′ ∗ ℎ𝑛−1 ℎ 𝑛
𝑓 𝑥 ∗ + ℎ = 𝑓 𝑥 ∗ + ℎ𝑓 ′ 𝑥 ∗ + 𝑓 𝑥 + ⋯+ 𝑓 𝑛−1 𝑥 ∗ + 𝑓 𝑛 𝑥 ∗

g
2! 𝑛−1 ! 𝑛!

a rin
rjy ee
at ha in
Since 𝑓 ′ 𝑥 ∗ = 𝑓 ′′ 𝑥 ∗ = ⋯ = 𝑓 𝑛−1 𝑥 ∗ = 0

ah tac eng
uw at il
G h iv
IT r B r C
ℎ𝑛 𝑛 ∗

i
, I ma fo
𝑓 𝑥∗ + ℎ − 𝑓 𝑥∗ = 𝑓 𝑥

EL Ku ds
𝑛!

o
N ajib eth
ℎ𝑛

.R m
When 𝑛 is even ≥0

of ion
𝑛!

Pr zat

PT
i
im
Thus if 𝑓 𝑛 𝑥 ∗ is positive
pt 𝑓 𝑥 ∗ + ℎ − 𝑓 𝑥 ∗ is positive Hence it is local minimum
O

Thus if 𝑓 𝑛 𝑥 ∗ negative 𝑓 𝑥 ∗ + ℎ − 𝑓 𝑥 ∗ is negative Hence it is local maximum

ℎ𝑛
When 𝑛 is odd, changes sign with the change in the sign of ℎ.
𝑛!

Hence it is an inflection point

R.K. Bhattacharjya/CE/IITG
 Sufficient conditions for optimality
Example

𝑓 𝑥 = 𝑥 3 − 10𝑥 − 2𝑥 2 − 10

g
a rin
Apply necessary condition

rjy ee
at ha in
ah tac eng
𝑓 ′ 𝑥 = 3𝑥 2 − 10 − 4𝑥 = 0

uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
Solving for 𝑥 𝑥 ∗ = 2.61 𝑎𝑛𝑑 − 1.28

o
N ajib eth
.R m
of ion
These two points are stationary points

Pr zat

PT
i
Apply sufficient condition im
𝑓 ′′ 𝑥 = 6𝑥 − 4
pt
O

𝑓 ′′ 2.61 = 11.66 positive and n is odd 𝑓 ′′ −1.28 = −11.68 negative and n is odd

𝑥 ∗ = 2.61 is a minimum point 𝑥 ∗ = −1.28 is a maximum point

7/14/2021 R.K. Bhattacharjya/CE/IITG 11

 Multivariable optimization without constraints

𝑥1
𝑥2
Minimize 𝑓(𝑋) Where 𝑋 = ⋮
𝑥𝑛

g
a rin
rjy ee
at ha in
ah tac eng
Necessary condition for optimality

uw at il
G h iv
IT r B r C

i
, I ma fo
If 𝑓(𝑋) has an extreme point (maximum or minimum) at 𝑋 = 𝑋 ∗ and if the first partial

EL Ku ds
o
N ajib eth
Derivatives of 𝑓 𝑋 exists at 𝑋 ∗ , then

.R m
of ion
Pr zat

PT
i
𝜕𝑓 𝑋 ∗ 𝜕𝑓 𝑋 ∗ im
𝜕𝑓 𝑋 ∗
pt
O

= =⋯= =0
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛

R.K. Bhattacharjya/CE/IITG
 Multivariable optimization without constraints

Sufficient condition for optimality

The sufficient condition for a stationary point 𝑋 ∗ to be an extreme point is that the
matrix of second partial derivatives of 𝑓(𝑋) evaluated at 𝑋 ∗ is

g
a rin
rjy ee
at ha in
(1) positive definite when 𝑋 ∗ is a relative minimum

ah tac eng
uw at il
(2) negative definite when 𝑋 ∗ is a relative maximum

G h iv
IT r B r C

i
, I ma fo
(3) neither positive nor negative definite when 𝑋 ∗ is neither a minimum nor a maximum

EL Ku ds
o
N ajib eth
Proof Taylor series of two variable function

.R m
of ion
Pr zat

PT
𝜕𝑓 𝜕𝑓 1 𝜕2𝑓 𝜕2𝑓 𝜕2𝑓

i
2 2

im
𝑓 𝑥 + ∆𝑥, 𝑦 + ∆𝑦 = 𝑓 𝑥, 𝑦 + ∆𝑥 + ∆𝑦 + ∆𝑥 + 2∆𝑥∆𝑦 + ∆𝑦 +⋯
pt𝜕𝑥 𝜕𝑦 2! 𝜕𝑥 2 𝜕𝑥𝜕𝑦 𝜕𝑦 2
O

𝜕𝑓 𝜕2𝑓 𝜕2𝑓
𝜕𝑥 1 𝜕𝑥 2 𝜕𝑥𝜕𝑦 ∆𝑥
𝑓 𝑥 + ∆𝑥, 𝑦 + ∆𝑦 = 𝑓 𝑥, 𝑦 + ∆𝑥 ∆𝑦
𝜕𝑓 + 2!
∆𝑥 ∆𝑦 +⋯
𝜕2𝑓 𝜕 2 𝑓 ∆𝑦
𝜕𝑦 𝜕𝑥𝜕𝑦 𝜕𝑦 2
R.K. Bhattacharjya/CE/IITG
 Multivariable optimization without constraints

1 𝑇
𝑓 𝑋 ∗ + ℎ = 𝑓 𝑋 ∗ + ℎ𝑇 𝛻𝑓 𝑋 ∗ + ℎ 𝑯ℎ + ⋯
2!
Another test: Evaluation of determinants
Since 𝑋 ∗ is a stationary point, the necessary

g
condition gives that 𝛻𝑓 𝑋 ∗ = 0

a rin
𝐴1 = 𝑎11

rjy ee
at ha in
Thus

ah tac eng
∗ ∗
1 𝑇 𝑎11 𝑎12
𝑓 𝑋 +ℎ −𝑓 𝑋 = ℎ 𝑯ℎ + ⋯ 𝐴2 = 𝑎

uw at il
G h iv
2! 21 𝑎22

IT r B r C

i
, I ma fo
𝑎11 𝑎12 𝑎13

EL Ku ds
Now, 𝑋 ∗ will be a minima, if ℎ𝑇 𝑯ℎ is positive

o
N ajib eth
𝐴3 = 𝑎21 𝑎22 𝑎23

.R m
𝑋 ∗ will be a maxima, if ℎ𝑇 𝑯ℎ is negative 𝑎31 𝑎32 𝑎33

of ion
Pr zat

PT
𝑎11 𝑎12 𝑎13 𝑎14 ⋯ 𝑎1𝑛
ℎ𝑇 𝑯ℎ will be positive if 𝑯 is a positive definite matrix

i
im
pt 𝑎21 𝑎22 𝑎23 𝑎24 ⋯ 𝑎2𝑛
𝐴𝑛 = ⋮
O
⋮ ⋮ ⋮ ⋯ ⋮
ℎ𝑇 𝐻ℎ will be negative if 𝑯 is a negative definite matrix 𝑎𝑛1 𝑎𝑛2 𝑎𝑛3 𝑎𝑛4 ⋯ 𝑎𝑛𝑛

A matrix 𝐻 will be positive definite if all the

eigenvalues are positive, i.e. all the 𝜆 values are  A matrix 𝑨 will be positive definite if any only if all the
positive which satisfies the following equation values 𝐴1 , 𝐴2 , 𝐴3 , …, 𝐴𝑛 are positive.
 The matrix will be negative definite is and only if the
𝑨 − 𝜆𝑰 = 0 R.K. Bhattacharjya/CE/IITG sign of 𝐴𝑗 is −1 𝑗 for 𝑗 = 1, 2, 3, … , 𝑛
 Multivariable optimization without constraints

Example
𝑓 𝑥1 , 𝑥2 = 𝑥1 − 10 2 + 𝑥2 − 10 2

Necessary condition

g
𝜕𝑓

a rin
= 2 𝑥1 − 10 = 0 ⇒ 𝑥1 = 10

rjy ee
𝜕𝑥1

at ha in
ah tac eng
𝜕𝑓 10
= 2 𝑥2 − 10 = 0 ⇒ 𝑥2 = 10 𝑋 ∗ =

uw at il
G h iv
𝜕𝑥2 10

IT r B r C

i
, I ma fo
Sufficient condition

EL Ku ds
o
N ajib eth
𝜕2𝑓 𝜕2𝑓

.R m
𝜕𝑥 2 𝜕𝑥𝜕𝑦

of ion
2 0
𝐻= =

Pr zat

PT
𝜕2𝑓 𝜕2𝑓 0 2

i
𝐴1 = 𝑎11 = 2
im
𝜕𝑥𝜕𝑦 𝜕𝑦 2 pt
O
𝑎11 𝑎12 2 0
𝐴2 = 𝑎 = =4
𝐻 − 𝜆𝐼 =
2−𝜆 0
= 2−𝜆 2−𝜆 =0 21 𝑎22 0 2
0 2−𝜆
Thus the eigenvalues of the matrix 𝐻 are 2, and 2. Both 𝐴1 and 𝐴2 are positive
The 𝐻 is a positive definite matrix and the 𝑋 ∗ is a The 𝐻 is a positive definite matrix
relative minimum Hence 𝑋 ∗ is a relative minimum
7/14/2021 R.K. Bhattacharjya/CE/IITG 15
 Unimodal and duality principle

30

25

20 Optimal solution 𝑥 ∗ = 0

g
a rin
rjy ee
15

at ha in
Optimal solution 𝑥 ∗ = 0

ah tac eng
10

uw at il
G h iv
5

IT r B r C

i
, I ma fo
0

EL Ku ds
o
-5 -3 -1 1 3 5

N ajib eth
.R m
0

of ion
Pr zat

PT
-5 -3 -1 1 3 5
-5

i
im
pt
O
-10

-15 Minimization 𝑓 𝑥 = Maximization −𝑓 𝑥

-20

-25

-30
7/14/2021 R.K. Bhattacharjya/CE/IITG 16
 7/14/2021
O
pt
im
i
Pr zat
of ion
.R m
N ajib eth

R.K. Bhattacharjya/CE/IITG
PT o
EL Ku ds
, I ma fo
IT r B r C
G h iv
uw at il
ah tac eng
at ha in
i rjy ee
a rin
g
17