Engineering Design and Optimization

PPU19160

Optimization algorithm
Part I – Unconstrained Optimization
EDO Chapter 4.1 – 4.5

Gauti Asbjörnsson
Department of Industrial and Materials Science
Chalmers University of Technology
SE-412 96 Gothenburg
e-mail: [email protected]
 Lecture objective

Apply numerical optimization techniques and computer tools to solve

optimization problems

Key points
Gradient based algorithms
Optimality conditions
Line search
Search direction

2 Division of Product Development, IMS 2024-09-04

 Solving the problem!!

Define Formulate Model Explore Solve

3 Division of Product Development, IMS 2024-09-04

 Optimization problem classification

4 Division of Product Development, IMS 9/4/2024

2024-09-04
 Optimization algorithm classification

5 Division of Product Development, IMS 9/4/2024

2024-09-04
 Optimization algorithm selection

6 Division of Product Development, IMS 9/4/2024

2024-09-04
7Division of Product Development, IMS 2024-09-04
2024-09-04
 Single-variable optimality conditions
1. First-order necessary condition
𝜕𝑓
If 𝑓(𝑥) is differentiable and 𝑥∗ is a local minimum, then (𝑥∗ ) = 0.
𝜕𝑥

Note: this is not sufficient - 𝑥∗ could be a local

maximum or saddle point (rather than a minimum)

2. Second-order sufficient condition zero

slope
𝜕𝑓 𝜕2 𝑓
If (𝑥∗ ) = 0 and 𝑥∗ > 0, then 𝑥∗ is a local minimum
𝜕𝑥 𝜕𝑥 2

Note: this is not necessary - 𝑥∗ could be a local

minimum without satisfying the former criteria

8 Division of Product Development, IMS 2024-09-04

 Single-variable optimality conditions

9 Division of Product Development, IMS 2024-09-04

 Multiple variables

𝜕𝑓 𝜕𝑓 𝜕𝑓
Gradient: 𝛻𝑓(𝐱) ≜ , ,…,
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛

𝜕2 𝑓 𝜕2 𝑓
⋯
𝜕𝑥12 𝜕𝑥1 𝑥𝑛

Hessian: 𝐇(𝐱) ≜ ⋮ ⋮

𝜕2 𝑓 𝜕2 𝑓
⋯ 2
𝜕𝑥𝑛 𝑥1 𝜕𝑥𝑛

10 Division of Product Development, IMS 2024-09-04

 Multi-variable optimality conditions
1. First-order necessary condition
If 𝑓(𝐱) is differentiable and 𝐱 ∗ is a local minimum, then 𝛁𝑓(𝐱 ∗ ) = 𝟎.

2. Second-order sufficient condition

If 𝛁𝑓(𝐱 ∗ ) = 𝟎 and 𝐇 𝐱 is positive-definite, then 𝐱 ∗ is a local minimum

11 Division of Product Development, IMS 2024-09-04

 Hessian
A Hessian is positive-definite if 𝜕𝐱 𝑇 𝐇 𝐱∗ 𝜕𝐱 > 0 for all 𝜕𝐱 ≠ 𝟎.
A matrix is positive-definite if and only if any of these hold:
1. All of its eigenvalues are positive.
2. All determinants of its leading principle minors are positive. Note: 𝜕𝐱 = 𝐱 − 𝐱 ∗
3. All the pivots are positive when the matrix is reduced to row-echelon form.

Other matrix classifications

Replace positive-definite with: Replace positive with: Nature of x*

negative-definite negative local maximum

positive-semidefinite non-negative probable valley

negative-semidefinite non-positive probable ridge

indefinite have both + & - saddle point

12Division of Product Development, IMS 2024-09-04
2024-09-04
13Division of Product Development, IMS 2024-09-04
2024-09-04
 Interpretation of derivatives
• The gradient is always orthogonal to the contour lines

14Division of Product Development, IMS 2024-09-04

 Convexity
A set is convex if a line segment connecting any two points in the set contains only points within that set

Convex
Non-convex

More rigorously, a set 𝑆 is convex if, for every point 𝐱1 , 𝐱 2 ∈ 𝑆, the point

𝐱 𝜆 = 𝜆𝐱 2 + 1 − 𝜆 𝐱1 , 0≤𝜆≤1

also belongs to 𝑆.

If the Hessian matrix is positive-definite at the stationery point then that x is a local minimum. A
differentiable function is convex if the Hessian matrix is positive-semidefinite in its entire convex domain
15Division of Product Development, IMS 2024-09-04
2024-09-04
 Find the optimal solution
Applying algorithms to find solution(s)
• Basic approaches
• Nonlinear, gradient-based
• Nonlinear, gradient-free

Define Formulate Model Explore Solve

16 Division of Product Development, IMS 2024-09-04

 Basic nonlinear problem

How do we know it’s a

minimizer?

17 Division of Product Development, IMS 2024-09-04

 Basic nonlinear problem

Positive definite

Positive definite

Indefinite

18 Division of Product Development, IMS 2024-09-04

 Gradient-based algorithms
• Line search (4.3)
• Search direction (4.4)
• Steepest descent (4.4.1)
• Conjugate Gradient (4.4.2)
• Newton method (4.4.3)
• Quasi-Newton methods (4.4.4)
• Trust region (4.5)

19 Division of Product Development, IMS 2024-09-04

 Gradient-based algorithms

20Division of Product Development, IMS 2024-09-04

2024-09-04
 Search Direction
• The optimization algorithms are named after the method used to find the search
direction, pk

21Division of Product Development, IMS 2024-09-04

2024-09-04
 Line Search
• The line search method determines how far to step in the chosen direction α and is
usually independent of the method for choosing the search direction

22Division of Product Development, IMS 2024-09-04

2024-09-04
 Line Search

23Division of Product Development, IMS 2024-09-04

2024-09-04
 Sufficient Decrease and Backtracking
If an iteration doesn't take you down enough, backtracking makes you take smaller steps until
you start making good progress again
Sufficient decrease condition - represents the expected decrease of the function, assuming
the function continued at the same slope
Backtracking - starts with a maximum step and successively reduces the step by a constant
ratio 휌 until it satisfies the sufficient decrease condition

24Division of Product Development, IMS 2024-09-04

2024-09-04
 Steepest Decent
The gradient points in the direction of steepest increase, so − 𝛁𝑓 𝐱 𝑖 points in the direction of
steepest descent

One major issue with the steepest descent is that, in general, the entries in the gradient and
its overall scale can vary greatly depending on the magnitudes of the objective function and
design variables.
The search direction is often better posed as a normalized direction

25Division of Product Development, IMS 2024-09-04

2024-09-04
 Steepest Descent

26Division of Product Development, IMS 2024-09-04

2024-09-04
 Steepest Descent
The gradient does not provide enough information to inform a good guess of the initial step
size for the line search
𝐱 𝑘 = 𝐱 𝑘−1 − 𝛼𝛁𝑓 𝐱 𝑘−1
We can make a guess of the step size for a given line search based on the result of the
previous one

Assuming that we will obtain a decrease in objective function at the current line search that is
comparable to the previous one.

28 Division of Product Development, IMS 2024-09-04

 Steepest Decent

Each search direction is orthogonal to the previous one

29Division of Product Development, IMS 2024-09-04

2024-09-04
 Steepest Descent example
Problem:

Gradient &
Hessian:

Algorithm:

0.27 
f
=  0.58 
f
0.77 

31 Division of Product Development, IMS 2024-09-04

 Conjugate Gradient
For a positive β, the result is a new direction somewhere between the current steepest
descent and the previous search direction

32Division of Product Development, IMS 2024-09-04

2024-09-04
 Conjugate Gradient

33Division of Product Development, IMS 2024-09-04

2024-09-04
 Conjugate Gradient example
Problem:

Gradient &
Hessian:

f k f kT f k f k
Algorithm: pk = k = T pk = − +  pk −1
f k f k −1f k −1 f k

k x Delta f pi beta alfa

0 [1;1;1] [7;15;20] [-0.27;-0.58;-0.77] 0 1

1 [0.73;0.42;0.23] [3;5.97;7.05] [0.31;0.63;0.74] 0.14 1

34 2 [0.38;-0.27;-0.60]
Division of Product Development, IMS 2024-09-04
 Newton’s Method
Newton’s method uses second-order (curvature) information to get better estimates for
search directions.

−1 𝛁𝑓
𝐱 𝑘+1 = 𝐱 𝑘 − 𝐇(𝐱 𝑘 ) 𝐱0

For a quadratic function using an exact line search and Newton’s method. Unsurprisingly, only one
iteration is required.
35Division of Product Development, IMS 2024-09-04
2024-09-04
 Newton´s method

37Division of Product Development, IMS 2024-09-04

2024-09-04
 Newton’s method example
Problem:

Gradient &
Hessian:

Algorithm:

38 Division of Product Development, IMS 2024-09-04

 Quasi-Newton methods
The second derivative can be difficult to compute. The basic idea is that we can use first-
order information (gradients) along each step in the iteration path to build an approximation of
the Hessian

Example update function: BFGS

Or estimate the inverse directly

39 Division of Product Development, IMS 2024-09-04

 Quasi-Newton methods example
Problem:

Gradient &
Hessian:

Algorithm: H

 1.91 2.07 2.98   0.94 −0.13 −0.18

H =  2.07 5.66 6.73  V =  −0.13 0.71 −0.41
 2.98 6.73 10.71  0.18 −0.41 0.40 

40 Division of Product Development, IMS 2024-09-04

 Quasi-Newton methods

41 Division of Product Development, IMS 2024-09-04

 Trust region
Instead of relying on line search, we can define a region around the current point with
quadratic approximation (restricted-step methods)
The main motivation for trust region methods is to address the issues with Newton’s
method as it does not guarantee a positive definite-Hessian approximation

42 Division of Product Development, IMS 2024-09-04

 Trust region
The trust region can grow, shrink, or remain the same, depending on how well the model
predicts the actual function decrease
The metric we use to assess the model is the actual function decrease divided by the expected
decrease.

𝑎𝑐𝑡𝑢𝑎𝑙
=
𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑

An r value close to 1 means that the model agrees well with the actual function. Value larger
than 1 means that the actual decrease was even greater than expected. A negative value means
that the function actually increased at the expected minimum, and therefore the model is not
suitable.
43 Division of Product Development, IMS 2024-09-04
 Trust region

44 Division of Product Development, IMS 2024-09-04

 Gradient based Methods

45Division of Product Development, IMS 2024-09-04

2024-09-04
 Gradient based Methods

46Division of Product Development, IMS 2024-09-04

2024-09-04
 Engineering Design Optimization
• https://www.youtube.com/@BYUFLOWLab

47Division of Product Development, IMS 2024-09-04

2024-09-04
 Next lecture(s)
Wednesday at 10:00 Tuesday
X1 - reflection
Design for Additive Manufacturing (L6)

Wednesday at 13:15
Material selection and DOE workshop (W3 and W4)

Wednesday at 15:15
Material Selection in Design (L7)

48 Division of Product Development, IMS 2024-09-04

 Product Design and Optimization

Q&A

49 Division of Product Development, IMS 2024-09-04