Optimization in

Engineering Design
SECTION 3: MULTI VARIABLE OPTIMIZATION

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 1

 Introduction
➢Algorithms for optimizing functions having multiple
design or decision variables
Multi variable

Optimization in Engineering Design

➢Single variable function optimization algorithms are
used in some of these algorithms to perform a
unidirectional search along a desired direction A. Direct Search
1. Evolutionary Optimization
➢Algorithms presented for minimization problems and can Method
be used for maximization problems using duality 2. Hooke Jeeves Pattern
3. Powell’s Conjugate Direction
principle
➢Algorithms can be broadly classified as
B. Gradient Based
➢Direct Search method: Only function values at different 1. Cauchy’s steepest descent
points are used to constitute a search 2. Newton’s method
➢Gradient Based method: Derivative information is used to 3. Marquardt method
constitute a search

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 2

Function & First Derivative
➢Definition of a local, global or an inflection point remains the same but optimality criteria is
different
➢Gradient of a function will not be a scalar quantity: instead it is a vector quantity

Optimization in Engineering Design

➢Assume an objective function of N design variables
➢Obj function = 𝑓(𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 … … … … 𝑥𝑁 )
➢Gradient vector is given by
𝜕𝑓
𝜕𝑥1
𝜕𝑓
𝜕𝑥2
∇𝑓 𝑥Ԧ = 𝜕𝑓
𝜕𝑥3
…..
….
𝜕𝑓
𝜕𝑥𝑁
10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 3
Gradient

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 4
Gradient Vector

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 5
Gradient

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 6
Second Derivative: Hessian Matrix
➢Second derivatives in multi variable functions form a symmetric matrix[N ×N]
(Better known as Hessian Matrix (H(𝒙))

Optimization in Engineering Design

𝜕 𝜕𝑓 𝜕𝑓 𝜕𝑓
ቈ ⋯ ቃ 𝜕2 𝑓 𝜕2 𝑓 𝜕2 𝑓
𝜕𝑥1 𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑁 ⋯
𝜕𝑥12 𝜕𝑥1 𝜕𝑥2 𝜕𝑥1 𝜕𝑥𝑁
𝜕 𝜕𝑓 𝜕𝑓 𝜕𝑓 𝜕2 𝑓 𝜕2 𝑓 𝜕2 𝑓
ቈ ⋯ ቃ ⋯
∇2 𝑓 𝑥Ԧ = 𝜕𝑥2 𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑁 = 𝜕𝑥1 𝜕𝑥2 𝜕𝑥22 𝜕𝑥2 𝜕𝑥𝑁

⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
𝜕2 𝑓 𝜕2 𝑓 𝜕2 𝑓
𝜕 𝜕𝑓 𝜕𝑓 𝜕𝑓 ⋯ 2
ቈ ⋯ ቃ 𝜕𝑥1 𝜕𝑥𝑁 𝜕𝑥2 𝜕𝑥𝑁 𝜕𝑥𝑁
𝜕𝑥𝑁 𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑁

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 7

Optimality Criteria
➢A point 𝑥Ԧ = 𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 … … … … 𝑥𝑁 is a stationary point if 𝜵𝒇 𝒙 = 𝟎.

Optimization in Engineering Design

Furthermore, the point is a minimum, a maximum or an inflection point if
𝜵𝟐 𝒇 𝒙 is positive definite, negative definite or otherwise
➢A matrix ∇2 𝑓 𝑥Ԧ is defined to be positive definite if for any point 𝑦 & 𝑦 ≠ 0 in
the search space the quantity 𝒚𝑻 𝜵𝟐 𝒇 𝒙 𝒚 > 𝟎
➢A matrix ∇2 𝑓 𝑥Ԧ is defined to be negative definite if for any point 𝑦 in the
search space the quantity 𝒚𝑻 𝜵𝟐 𝒇 𝒙 𝒚 < 𝟎
➢If at some point in the search space the quantity 𝒚𝑻 𝜵𝟐 𝒇 𝒙 𝒚 > 𝟎 (Positive)
and at some point 𝒚𝑻 𝜵𝟐 𝒇 𝒙 𝒚 < 𝟎 (Negative) then the matrix 𝜵𝟐 𝒇 𝒙 is
neither positive definite nor negative definite

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 8

Unidirectional Search
➢Many multi variable optimization techniques use
(2,5)
successive unidirectional search techniques to

Optimization in Engineering Design

find the minimum point along a particular
direction
➢Unidirectional search is one dimensional search
performed by comparing function values only (6.207,11.517)
(10,10)
along a specified direction
➢Unidirectional search performed from a point 𝑥
and in a specified direction 𝒔
➢An arbitrary point on that line can be expressed
as
𝑥 𝛼 = 𝑥 + 𝛼𝑠
➢𝜶 specifies the relative measure of distance of
the point 𝑥 𝛼 from 𝑥

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 9

Example 3.1
Check for optimality of Himmelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2
+
𝑥1 + 𝑥22 − 7 2 at point (0, 0)

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 10
Example 3.1
Check for optimality of Himmelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2
+
𝑥1 + 𝑥22 − 7 2 at point (0, 0)

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 11
Example 3.1

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 12
Example 3.2
Check for optimality of Himmelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2
+
𝑥1 + 𝑥22 − 7 2 at point (3, 2)

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 13
Example 3.2
Optimal??? Check for definiteness using the following points
1 −1 −1 1

Optimization in Engineering Design

𝑦= , , ,
1 −1 1 −1

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 14

Example 3.2

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 15
Example 3.2

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 16
Test for Definiteness

Optimization in Engineering Design

Positive
Definiteness

𝑦 𝑇 ∆2 𝑓(𝑥)𝑦:
Ԧ Leading Principal Eigen Values:
Positive Minors: Positive Positive

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 17

Sub Matrices & Minors
A symmetric matrix A (𝑁 × 𝑁) will be:

Optimization in Engineering Design

➢Positive Definite: if all its leading principal minors are Positive 𝐴𝑘 >0
➢Negative Definite: if all its leading principal minors alternates in signs
(−1)𝑘 𝐴𝑘 > 0: 𝐴1 < 0, 𝐴2 > 0, 𝐴3 < 0, 𝐴4 > 0 … . .
For matrix A (𝑁 × 𝑁), the kth order sub matrix is obtained by deleting any n-k
rows and any n-k columns, and its determinant is called a minor.
Eg: Matrix ‘A’ (3 × 3)
𝑎11 𝑎12 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 18

First Order Sub Matrix: k=1
➢Number of rows and columns to be deleted = n-k = 3-1 =2
➢Example: Deleting row (1 & 3) and column (2 & 3)

Optimization in Engineering Design

𝑎11 𝑎22 𝑎23
𝐴 = 𝑎21 𝑎22 𝑎23 → 𝐴1 = 𝑎21
𝑎31 𝑎32 𝑎33
➢Similarly deleting various combinations of rows and columns: One can obtain
the following sub matrices
𝑎11 , 𝑎12 , 𝑎23
𝒂𝟐𝟏 , 𝑎22 , 𝑎23
𝑎31 , 𝑎32 , 𝑎33
➢ We get nine 1st order submatrices of [A]

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 19

Second Order Sub Matrix: k=2
➢Number of rows and columns to be deleted = n-k = 3-2 =1
➢Example: Deleting row (1) and column ( 3)

Optimization in Engineering Design

𝑎11 𝑎12 𝑎13
𝑎21 𝑎22
𝐴 = 𝑎21 𝑎22 𝑎23 → 𝐴2 = 𝑎 𝑎32
31
𝑎31 𝑎32 𝑎33
➢Similarly deleting various combinations of rows and columns: One can obtain the
following sub matrices
𝑎11 𝑎12 𝑎12 𝑎13 𝑎11 𝑎13
𝑎21 𝑎22 , 𝑎22 𝑎23 , 𝑎21 𝑎23
𝒂𝟐𝟏 𝒂𝟐𝟐 𝑎22 𝑎23 𝑎21 𝑎23
𝒂𝟑𝟏 𝒂𝟑𝟐 , 𝑎32 𝑎33 , 𝑎31 𝑎33
𝑎11 𝑎12 𝑎12 𝑎13 𝑎11 𝑎13
𝑎31 𝑎32 , 𝑎32 𝑎33 , 𝑎31 𝑎33
➢We get nine 2nd order submatrices of [A]
10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 20
Third Order Sub Matrix: k=3
➢Number of rows and columns to be deleted = n-3 = 3-3 =0
➢Example: Deleting 0 row and column

Optimization in Engineering Design

𝑎11 𝑎12 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33
➢We get one 3rd order submatrix of [A] which is the same matrix itself
➢Corresponding Determinants are called minors

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 21

Principal Sub Matrices
For matrix A (𝑁 × 𝑁), the kth order principal sub matrix is obtained by deleting
the any n-k rows and the same n-k columns, and its determinant is called a

Optimization in Engineering Design

principal minor.
Eg: Matrix ‘A’ (3 × 3)
𝑎11 𝑎12 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

𝑎11 𝑎12 𝑎13

𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 22

First Order Principal Sub Matrix: k=1
➢Number of rows and columns to be deleted = n-k = 3-1 =2

Optimization in Engineering Design

➢Example: Deleting row (1 & 3) and column (1 & 3)
𝑎11 𝑎22 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23 → 𝐴1 = 𝑎22
𝑎31 𝑎32 𝑎33
➢Similarly deleting various combinations of rows and columns: One can obtain
the following sub matrices
𝑎11 , 𝒂𝟐𝟐 , 𝑎33
➢ We get three 1st order submatrices of [A]

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 23

Second Order Principal Sub Matrix: k=2
➢Number of rows and columns to be deleted = n-k = 3-2 =1

Optimization in Engineering Design

➢Example: Deleting row (1) and column ( 1)
𝑎11 𝑎12 𝑎13
𝑎22 𝑎23
𝐴 = 𝑎21 𝑎22 𝑎23 → 𝐴2 = 𝑎 𝑎33
32
𝑎31 𝑎32 𝑎33
➢Similarly deleting various combinations of rows and columns: One can obtain the
following sub matrices
𝑎11 𝑎12 𝒂𝟐𝟐 𝒂𝟐𝟑 𝑎11 𝑎13
𝑎21 𝑎22 , 𝒂𝟑𝟐 𝒂𝟑𝟑 , 𝑎31 𝑎33
➢We get three 2nd order submatrices of [A]

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 24

Third Order Principal Sub Matrix: k=3
➢Number of rows and columns to be deleted = n-3 = 3-3 =0
➢Example: Deleting 0 row and column

Optimization in Engineering Design

𝑎11 𝑎12 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33
➢We get one 3rd order submatrix of [A] which is the same matrix itself
➢Corresponding Determinants are called principal minors

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 25

Leading Principal Sub Matrices
For matrix A (𝑁 × 𝑁), the kth order leading principal sub matrix is obtained by
deleting the last n-k rows and the last n-k columns, and its determinant is

Optimization in Engineering Design

called a leading principal minor 𝑨𝒌 .
Eg: Matrix ‘A’ (3 × 3)
𝑎11 𝑎12 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

𝑎11 𝑎12 𝑎13

𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 26

First Order Leading Principal Sub Matrix: k=1
➢Number of rows and columns to be deleted = n-k = 3-1 =2

Optimization in Engineering Design

➢Example: Deleting row (2 & 3) and column (2 & 3)
𝑎11 𝑎22 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23 → 𝐴1 = 𝑎11
𝑎31 𝑎32 𝑎33
➢One can obtain the following sub matrices
𝒂𝟏𝟏
➢ We get one 1st order submatrix of [A]

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 27

Second Order Leading Principal Sub Matrix: k=2
➢Number of rows and columns to be deleted = n-k = 3-2 =1

Optimization in Engineering Design

➢Example: Deleting row (3) and column ( 3)
𝑎11 𝑎12 𝑎13
𝑎11 𝑎12
𝐴 = 𝑎21 𝑎22 𝑎23 → 𝐴2 = 𝑎 𝑎22
21
𝑎31 𝑎32 𝑎33
➢One can obtain the following sub matrix
𝒂𝟏𝟏 𝒂𝟏𝟐
𝒂𝟐𝟏 𝒂𝟐𝟐
➢We get one 2nd order submatrices of [A]

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 28

Third Order Leading Principal Sub Matrix: k=3
➢Number of rows and columns to be deleted = n-3 = 3-3 =0

Optimization in Engineering Design

➢Example: Deleting 0 row and column
𝑎11 𝑎12 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33
➢We get one 3rd order submatrix of [A] which is the same matrix itself
➢Corresponding Determinants are called leading principal minors
➢LPS are constructed starting from the leftmost top corner & proceeding along
leading diagonal top to bottom and left to right

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 29

Example 3.3
Using Principal minors, check if the point (3, 2) is optimal for Himmelblau
function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2 + 𝑥1 + 𝑥22 − 7 2

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 30
Example 3.3

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 31
Box’s Evolutionary Optimization Method
➢Simple Optimization technique
developed by G.E.P Box in 1957

Optimization in Engineering Design

➢The algorithm requires 𝟐𝑵 + 𝟏
function evaluation of which 2𝑁 are corner
points of an N-dimensional hypercube
➢The function values at these points are
compared & the best point is identified
➢In the next iteration another hypercube
is formed around this best point
➢If at iteration, an improved point is not
found, the size of the hypercube is
reduced
➢Process continues until the hypercube
becomes very small
10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 32
Algorithm: Box’s Evolutionary Method
Step 1: Initialization (Assumption)
◦ Initial point 𝑥Ԧ (0) arbitrarily

Optimization in Engineering Design

◦ Size reduction parameters, ∆= ∆1 , ∆2 , ∆3 , … … . . ∆𝑁
◦ Maximum error 𝜀 → 0+ for termination check
◦ Set 𝑥Ԧ ∗ = 𝑥Ԧ (0)
Step 2:
◦ If ∆ < 𝜀
◦ Print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥 ∗ & 𝑺𝑻𝑶𝑷
∆𝑖
◦ Else create 2𝑁 points by adding & subtracting for each variable at point 𝑥Ԧ ∗ &
2
Goto Step 3

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 33

Algorithm: Box’s Evolutionary Method
Step 3:
◦ Compute function values at all (2𝑁 +1) points

Optimization in Engineering Design

◦ Find the point having minimum function value & set as 𝑥Ԧ ∗
Step 4:
◦ If (𝑥Ԧ ∗ ) = (𝑥Ԧ (0) )
∆𝑖
◦ Reduce size parameters ∆𝑖 = & Goto Step 2
2
◦ Else set 𝑥Ԧ (0) = 𝑥 ∗ & Goto Step 2

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 34

Example 3.4
Minimize Himmelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2 + 𝑥1 + 𝑥22 − 7 2 using
Box’s Evolutionary Search Method in the interval 0 ≤ 𝑥1 , 𝑥2 ≤ 5. Assume initial

Optimization in Engineering Design

1 2
point 𝑥Ԧ (0) = , size reduction parameter ∆= with a desired accuracy
1 2
𝜀=0.001

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 35

Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 36
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 37
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 38
Hooke Jeeves Pattern Search Method
➢The search method works by creating a set of search directions iteratively

Optimization in Engineering Design

➢The created search directions should be such that they completely span
the search space
➢In a N-dimensional problem, this requires at least N linearly independent
search directions
➢In the Hooke-Jeeves method, a combination of exploratory moves and
heuristic pattern moves is made iteratively
▪ An exploratory move is performed in the vicinity of the current point
systematically to find the best point around the current point
▪ Two such points are used to make a pattern move

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 39

Exploratory Move
➢Search for the better possible solution in the vicinity of the current point 𝑥Ԧ =
𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 … … … 𝑥𝑁

Optimization in Engineering Design

➢Move in ‘i’ direction with step size of ∆𝑖
➢In each direction, ‘i’, move one step ahead & back
➢Take one variable 𝒙𝒊 at a time to move ±∆𝑖
➢Compare the function values at these points

𝑓 = 𝑓(𝑥𝑖 ) 𝑓 + = 𝑓(𝑥𝑖 + ∆𝑖 )
𝑓 − = 𝑓(𝑥𝑖 − ∆𝑖 )
-∆𝑖 +∆𝑖
𝑥𝑖 − ∆𝑖 𝑥𝑖 𝑥𝑖 + ∆𝑖

➢If𝑥Ԧ (𝑘) ≠ 𝑥,
Ԧ move is success

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 40

Pattern Move
➢Move along a straight line connecting current best point 𝑥Ԧ (𝑘) and previous best
point 𝑥Ԧ (𝑘−1)

Optimization in Engineering Design

➢New point after pattern move is given as 𝑥Ԧ (𝑘+1) = 𝑥Ԧ (𝑘) + (𝑥Ԧ (𝑘) − 𝑥Ԧ (𝑘−1) )

𝑥2

Pattern Move

Exploratory Move

𝑥1

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 41

Algorithm: Hooke Jeeves Method
Step 1: Initialization (Assumption)
◦ Initial point 𝑥Ԧ (0) arbitrarily

Optimization in Engineering Design

◦ Variable increments along each ‘i’ directions ∆= ∆1 , ∆2 , ∆3 , … … . . ∆𝑁
◦ Step reduction factor: 𝛼 > 1
◦ Maximum error 𝜀 → 0+ for termination check
◦ Set iteration, k=0
Step 2:
◦ Do Exploratory Move (EM) with 𝑥Ԧ (𝑘) as current point & let outcome be 𝑥Ԧ
◦ If EM is success i.e. 𝑥Ԧ ≠ 𝑥Ԧ (𝑘)
◦ Set 𝑥Ԧ (𝑘+1) = 𝑥Ԧ & goto Step 4
◦ Else goto Step 3

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 42

Algorithm: Hooke Jeeves Method
Step 3:
◦ If ∆ < 𝜀
◦ Print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥 (𝑘) & 𝑺𝑻𝑶𝑷

Optimization in Engineering Design

∆
◦ Else set ∆𝑖 = 𝑖 for i=1, 2,…… N & Goto Step 2
𝛼
Step 4:
◦ Set k=k+1 & do pattern move (PM)
(𝑘+1)
𝑥Ԧ𝑝 = 𝑥Ԧ (𝑘) + (𝑥Ԧ (𝑘) − 𝑥Ԧ (𝑘−1) )
Step 5:
(𝑘+1)
◦ Do EM using 𝑥Ԧ𝑝 as base point.
◦ Let outcome of EM be 𝑥Ԧ (𝑘+1)
Step 6:
◦ If 𝑓(𝑥Ԧ (𝑘+1) ) < 𝑓(𝑥Ԧ (𝑘) )
◦ Goto Step 4 for next PM
◦ Else Goto Step 3

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 43

Example 3.4
Minimize Himelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2 + 𝑥1 + 𝑥22 − 7 2 using
0
Hooke Jeeves Pattern Search Method. Assume initial point 𝑥Ԧ (0) = , increment

Optimization in Engineering Design

0
0.5
∆= , step reduction factor 𝛼 = 2 and desired accuracy 𝜀=0.1
0.5

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 44

Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 45
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 46
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 47
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 48
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 49
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 50
Example 3.4

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 51
Example 3.4
S# 𝒙𝟏 𝒙𝟐 𝒌 𝒙(𝒌) 𝒇(𝒙(𝒌) )
1 0 0 0 𝒙(𝟎) 170

Optimization in Engineering Design

2 0.5 0
3 0.5 0.5 1 𝒙(𝟏) 144.15
4 1 1 (2)
𝑥𝑝
5 1.5 1
6 1.5 1.5 2 𝒙(𝟐) 63.12
7 2.5 2.5 (3)
𝑥𝑝
8 3 2.5
9 3 2 3 𝒙(𝟑) 0
10 4.5 2.5 (4)
𝑥𝑝
11 4 2.5
12 4 2 4 𝒙(𝟒) 50

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 52

Gradient Based Method: Multivariable
➢For Discrete/ Discontinuous function: Exact Derivatives not possible
➢Approximate derivatives obtained using Central Difference Method

Optimization in Engineering Design

𝜕𝑓 𝑓 𝑥𝑖 +∆𝑥𝑖 −𝑓(𝑥𝑖 −∆𝑥𝑖 )
=
𝜕𝑥𝑖 2∆𝑥𝑖
𝜕2 𝑓 𝑓 𝑥𝑖 +∆𝑥𝑖 −2𝑓 𝑥𝑖 +𝑓(𝑥𝑖 −∆𝑥𝑖 )
=
𝜕2 𝑥𝑖2 (∆𝑥𝑖 )2
▪ Where 𝑓 𝑥𝑖 + ∆𝑥𝑖 = 𝑓 𝑥1 , 𝑥2 , … … . . 𝑥𝑖 + ∆𝑥𝑖 … … . 𝑥𝑁
➢Here all other variable will be held constant except 𝑥𝑖
𝜕2 𝑓 1
➢ = × ൣ 𝑓 𝑥𝑖 + ∆𝑥𝑖 , 𝑥𝑗 + ∆𝑥𝑗 − 𝑓 𝑥𝑖 + ∆𝑥𝑖 , 𝑥𝑗 − ∆𝑥𝑗 − ൛𝑓൫𝑥𝑖 −
𝜕𝑥𝑖 𝜕𝑥𝑗 4∆𝑥𝑖 ∆𝑥𝑗
∆𝑥𝑖 , 𝑥𝑗 + ∆𝑥𝑗 ൯ − 𝑓 𝑥𝑖 − ∆𝑥𝑖 , 𝑥𝑗 − ∆𝑥𝑗 ൟ൧
▪ Where, 𝑓 𝑥𝑖 + ∆𝑥𝑖 , 𝑥𝑗 + ∆𝑥𝑗 = 𝑓 𝑥1 , 𝑥2 , … … . . 𝑥𝑖 + ∆𝑥𝑖 , … … 𝑥𝑗 + ∆𝑥𝑗 … … . 𝑥𝑁

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 53

Gradient Vector

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 54
Descent Direction
➢A search direction 𝑑Ԧ(𝑘) , is a descent
direction at point 𝑥Ԧ (𝑘) if the condition

Optimization in Engineering Design

𝜵𝒇 𝒙 𝒌 . 𝒅(𝒌) ≤ 𝟎 is satisfied in the vicinity
of the point 𝑥Ԧ (𝑘)
➢Along 𝑑Ԧ(𝑘) : 𝑓 𝑥Ԧ ↓⟹ 𝑓 𝑥Ԧ (𝑘+1) < 𝑓(𝑥Ԧ (𝑘) )
➢For a current point 𝑥Ԧ (𝑘) , the next point
𝑥Ԧ (𝑘+1) is taken along the direction 𝑑Ԧ(𝑘) for a
scalar 𝛼 representing relative distance from
𝑥Ԧ (𝑘) , then
𝑥Ԧ (𝑘+1) = 𝑥Ԧ (𝑘) + 𝛼. 𝑑Ԧ(𝑘)

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 55

Cauchy’s Steepest Descent Method
➢The search direction used in Cauchy’s Method is the negative of the gradient
at any particular point 𝑥Ԧ (𝑘)

Optimization in Engineering Design

𝑑Ԧ(𝑘) = −𝛻𝑓 𝑥Ԧ 𝑘
➢Since it gives the maximum descent in function values, also known as
Steepest descent method
➢At every iteration, the derivative 𝛻𝑓 𝑥Ԧ 𝑘 is computed at the current point
𝑥Ԧ (𝑘) & a unidirectional search is performed in the negative to this direction to
find the minimum point 𝑥Ԧ (𝑘+1) along that direction
➢The minimum point 𝑥Ԧ (𝑘+1) becomes the current point & search is continued
➢The process continues until the minimum point of the search space is
obtained

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 56

Algorithm: Steepest Descent Method
Step 1: Initialization (Assumption)

Optimization in Engineering Design

◦ Initial point 𝑥Ԧ (0) arbitrarily
◦ Max. Number of iterations = M
◦ Two termination parameters, (𝜀1 , 𝜀2 → 0)
◦ Set iteration, k=0
Step 2:
𝑘
◦ Calculate the 𝛻𝑓 𝑥Ԧ at current point 𝑥Ԧ (𝑘)
Step 3:
◦ If 𝛻𝑓 𝑥Ԧ 𝑘 ≤ 𝜀1 , print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘)
◦ Else if 𝑘 ≥ 𝑀, print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘) & STOP

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 57

Algorithm: Steepest Descent Method
Step 4:

Optimization in Engineering Design

◦ Do unidirectional search to find 𝛼 (𝑘) using 𝜀2 such that
𝑥Ԧ 𝑘+1 = 𝑥Ԧ 𝑘 − 𝛼 (𝑘) 𝛻𝑓 𝑥Ԧ 𝑘 is minimum
𝑘+1 𝑘
◦ Termination Criteria 𝛻𝑓 𝑥Ԧ . 𝛻𝑓 𝑥Ԧ ≤ 𝜀2
Step 5:
◦ Check for termination
𝑥Ԧ 𝑘+1 −𝑥Ԧ 𝑘
◦ Is ≤ 𝜀1
𝑥Ԧ 𝑘

◦ If yes, print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘) & STOP

◦ Else 𝑘 = 𝑘 + 1 & Goto Step 2

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 58

Example 3.5
Minimize Himelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2 + 𝑥1 + 𝑥22 − 7 2 using
0
Cauchy’s Method. Assume initial point 𝑥Ԧ (0) = , Maximum number of

Optimization in Engineering Design

0
iterations = 100, desired accuracy 𝜀=0.1

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 59

Example 3.5

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 60
Example 3.5

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 61
Example 3.5

Optimization in Engineering Design

(1.778, 2.794)

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 62

Gradient Based Methods
Cauchy’s

Optimization in Engineering Design

steepest
Descent: Start
point far away

Gradient Newton’s ➢The search direction used in

Based Method: Start Newton’s method uses second
Multivariable point near order derivative at any particular
point 𝑥Ԧ (𝑘) when compared to
Marquardts Cauchy’s Method
Algorithm:
Start point −1
anywhere 𝑑Ԧ(𝑘) = − 𝛻 2 𝑓 𝑥Ԧ 𝑘 . 𝛻𝑓 𝑥Ԧ 𝑘

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 63

Algorithm: Newton’s Method
Step 1: Initialization (Assumption)

Optimization in Engineering Design

◦ Initial point 𝑥Ԧ (0) arbitrarily
◦ Max. Number of iterations = M
◦ Two termination parameters, (𝜀1 , 𝜀2 → 0)
◦ Set iteration, k=0
Step 2:
𝑘
◦ Calculate the 𝛻𝑓 𝑥Ԧ at current point 𝑥Ԧ (𝑘)
Step 3:
◦ If 𝛻𝑓 𝑥Ԧ 𝑘 ≤ 𝜀1 , print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘)
◦ Else if 𝑘 ≥ 𝑀, print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘) & STOP

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 64

Algorithm: Newton’s Method
Step 4:
◦ Do unidirectional search to find 𝛼 (𝑘) using 𝜀2 such that

Optimization in Engineering Design

𝑘+1 𝑘 (𝒌) 𝟐 𝒌 −𝟏 𝑘
𝑥Ԧ = 𝑥Ԧ −𝜶 𝜵 𝒇 𝒙 𝛻𝑓 𝑥Ԧ is minimum
Step 5:
◦ If 𝒇(𝒙 𝒌+𝟏 ) < 𝒇(𝒙 𝒌 ) Goto Step 6
◦ Else Goto Step 1
Step 6:
◦ Check for termination
𝑥Ԧ 𝑘+1 −𝑥Ԧ 𝑘
◦ Is ≤ 𝜀1
𝑥Ԧ 𝑘

◦ If yes, print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘) & STOP

◦ Else 𝑘 = 𝑘 + 1 & Goto Step 2
10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 65
Example 3.6
Minimize Himelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2 + 𝑥1 + 𝑥22 − 7 2 using
0
Newton’s Method. Assume initial point 𝑥Ԧ (0) = , Maximum number of

Optimization in Engineering Design

0
iterations = 100, desired accuracy 𝜀=0.1

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 66

Example 3.6

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 67
Example 3.6

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 68
Example 3.6

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 69
Example 3.6

Optimization in Engineering Design

(-0.334, -0.85023)

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 70

Marquardt’s Method
➢Initially runs like Cauchy’s Method
➢Later on, once it approaches optimal point, it runs like Newton’s Method

Optimization in Engineering Design

➢Transition from Cauchy to Newton is Adaptive & depends on intermediate
points
➢Next point: 𝑥Ԧ (𝑘+1) = 𝑥Ԧ (𝑘) + 𝛼. 𝑑Ԧ(𝑘)

Sl. No. Method Descent Direction (𝑑Ԧ(𝑘) )

1 Cauchy 𝑘
−𝛻𝑓 𝑥Ԧ
2 Newton −1
− 𝛻 2 𝑓 𝑥Ԧ 𝑘 . 𝛻𝑓 𝑥Ԧ 𝑘

3 Marquardt −1
− 𝛻 2 𝑓 𝑥Ԧ 𝑘
+ 𝜆(𝑘) 𝐼 . 𝛻𝑓 𝑥Ԧ 𝑘

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 71

Algorithm: Marquardt’s Method
Step 1: Initialization (Assumption)

Optimization in Engineering Design

◦ Initial point 𝑥Ԧ (0) arbitrarily
◦ Max. Number of iterations = M
◦ Two termination parameters, (𝜀1 , 𝜀2 → 0)
◦ Set iteration, k=0
Step 2:
𝑘
◦ Calculate the 𝛻𝑓 𝑥Ԧ at current point 𝑥Ԧ (𝑘)
Step 3:
◦ If 𝛻𝑓 𝑥Ԧ 𝑘 ≤ 𝜀1 , print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘)
◦ Else if 𝑘 ≥ 𝑀, print 𝑥Ԧ𝑚𝑖𝑛 = 𝑥Ԧ (𝑘) & STOP

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 72

Algorithm: Marquardt’s Method
Step 4:
◦ Do unidirectional search to find 𝛼 (𝑘) using 𝜀2 such that

Optimization in Engineering Design

𝑘+1 𝑘 𝒌 𝟐 𝒌 −𝟏
𝑥Ԧ = 𝑥Ԧ − 𝜶 𝜵 𝒇 𝒙 + 𝝀 𝑰 . 𝛻𝑓 𝑥Ԧ 𝑘 𝒌
is minimum
◦ Termination Criteria 𝛻𝑓 𝑥Ԧ 𝑘+1 . 𝛻𝑓 𝑥Ԧ 𝑘 ≤ 𝜀2
◦ Next point can also be 𝑥Ԧ 𝑘+1 = 𝑥Ԧ 𝑘 + 𝑑Ԧ
Step 5:
◦ If 𝑓(𝑥Ԧ 𝑘+1 ) < 𝑓(𝑥Ԧ 𝑘
) Goto Step 6
◦ Else Goto Step 7
Step 6:
𝜆(𝑘)
◦ Set 𝜆(𝑘+1) = ,𝑘 = 𝑘 + 1, Goto Step 2
2
Step 7:
◦ Set 𝜆(𝑘) = 2𝜆(𝑘) , Goto Step 4

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 73

Example 3.7
Minimize Himelblau function 𝑓 𝑥 = 𝑥12 + 𝑥2 − 11 2 + 𝑥1 + 𝑥22 − 7 2 using
0
Marquardt Method. Assume initial point 𝑥Ԧ (0) = , Maximum number of

Optimization in Engineering Design

(0)
0
iterations = 100, desired accuracy 𝜀=0.1, 𝜆 = 100

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 74

Example 3.7

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 75
Example 3.7

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 76
Example 3.7

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 77
Example 3.7

Optimization in Engineering Design

10/29/2021 DEPARTMENT OF MECHANICAL ENGINEERING, NITT 78