Engineering Optimization Concepts and Applications

Engineering Optimization
Concepts and Applications

Fred van Keulen
Matthijs Langelaar
CLA H21.1
[email protected]
Engineering Optimization Concepts and Applications
Recap / overview
Optimization problem
Definition
Checking
Negative null form Model
Special topics
Sensitivity analysis
Topology optimization
Linear / convex problems
Solution methods
Optimality criteria
Optimization algorithms
Unconstrained problems
Optimality criteria
Optimization algorithms
Constrained problems
Engineering Optimization Concepts and Applications
Summary optimality conditions
Conditions for local minimum of unconstrained problem:
First Order Necessity Condition:
Second Order Sufficiency Condition: H positive definite
0 = Vf
For convex f in convex feasible domain:
condition for global minimum:
Sufficiency Condition:
0 = Vf
Engineering Optimization Concepts and Applications
Stationary point nature summary
Definiteness H Nature x*

Positive d. Minimum

Positive semi-d. Valley

Indefinite Saddlepoint

Negative semi-d. Ridge

Negative d. Maximum
Hy y
T
i

0 >
0 >
0 =
0 s
0 <
Engineering Optimization Concepts and Applications
Complex eigenvalues?
Question: what is the nature of a stationary point when
H has complex eigenvalues?
Answer: this situation never occurs, because H is
symmetric by definition. Symmetric matrices have real
eigenvalues (spectral theory).
Engineering Optimization Concepts and Applications
Nature of stationary points
Nature of initial position depends on load (buckling):
F

k
1
k
2
l

2

5 . 9 , 10
2 , 6
2 1
= =
= =
k k
l F
2 1
cos cos l l dz =
Fdz k k + = H
2
2 2
2
1 1
2
1
2
1

0 =
)
`

= H V
1 2 2 2
2 1 1 1
cos sin
cos sin


Fl k
Fl k
(

= AH
2 1 2 2 1
2 1 2 1 1
cos cos sin sin
sin sin cos cos


Fl k Fl
Fl Fl k
)
`

=
0
0

= AH
Fl k
Fl k
2
1
0
0
|
.
|

\
|
=
l
k
l
k
F
crit
2 1
, min
75 . 4 =
crit
F
Engineering Optimization Concepts and Applications
Nature of stationary points (2)
75 . 3 = F 75 . 4 = F 75 . 5 = F 75 . 6 = F 75 . 7 = F
2

H
Engineering Optimization Concepts and Applications
Unconstrained optimization
algorithms
Single-variable methods
0
th
order (involving only f )
1
st
order (involving f and f )
2
nd
order (involving f, f and f )
Multiple variable methods
Engineering Optimization Concepts and Applications
Why optimization algorithms?
Optimality conditions often cannot be used:
Function not explicitly known (e.g. simulation)
Conditions cannot be solved analytically
Example:
1 2
2
2 1
2
1 ` 2 1
x x
e x e x x x x f

+ + + + =
Stationary points:

= + =
c
c
= + + =
c
c
= V


0 2 1
0 2 1
1 2
1 2
2 2 1
2
2
2 1 1
1
x x
x x
e x e x x
x
f
e x x e x
x
f
f 0
Engineering Optimization Concepts and Applications
0
th
order methods: pro/con
Strengths:

No derivatives needed
Work also for
discontinuous / non-
differentiable functions
Easy to program
Robust
Weaknesses:

(Usually) less efficient
than higher order
methods (many function
evaluations)
Engineering Optimization Concepts and Applications
Minimization with one variable
Why?
Simplest case: good starting point
Used in multi-variable methods during line search
x x x t s
x f
x
s s . .
) ( min
Setting:
f

x

Model
Optimizer
Iterative process:
Engineering Optimization Concepts and Applications
Termination criteria
Stop optimization iterations when:
Solution is sufficiently accurate (check optimality criteria)
Progress becomes too slow:

Maximum resources have been spent
The solution diverges
Cycling occurs
f k k x k k
x f x f x x c c s s

) ( ) ( ,
1 1
x
a
x
b
Engineering Optimization Concepts and Applications
Brute-force approach
Simple approach: exhaustive search





Disadvantage: rather inefficient
f

x

L
0
0
2
1
n
L L
n
=
+
n points:
Final interval size =
L
n
Engineering Optimization Concepts and Applications
Basic strategy of 0
th
order
methods for single-variable case
1. Find interval [a
0
, b
0
] that contains the minimum
(bracketing)
2. Iteratively reduce the size of the interval [a
k
, b
k
]
(sectioning)
3. Approximate the minimum by the minimum of a simple
interpolation function over the interval [a
N
, b
N
]
Sectioning methods:
Dichotomous search
Fibonacci method
Golden section method
Engineering Optimization Concepts and Applications
Bracketing the minimum
f

x

x
1
x
2
= x
1
+A x
3
= x
2
+A x
4
= x
3
+
2
A
[a
0
, b
0
]
Starting point x
1
, stepsize A, expansion parameter : user-defined
Engineering Optimization Concepts and Applications
Unimodality
Bracketing and sectioning methods work best for
unimodal functions:
An unimodal function consists of exactly one
monotonically increasing and decreasing part
Engineering Optimization Concepts and Applications
L
0

a
0
b
0

Dichotomous search
Main Entry: di chot o mous
Pronunciation: dI-'kt-&-m&s also d&-
Function: adjective
: dividing into two parts
Conceptually simple
idea:
Try to split interval in half in each step
o << L
0

L
0
/2
( ) ( ): > f f
Engineering Optimization Concepts and Applications
Dichotomous search (2)
Interval size after 1 step (2 evaluations):
( ) o + =
0 1
2
1
L L
Interval size after m steps (2m evaluations):
|
.
|

\
|
+ =
m m
m
L
L
2
1
1
2
0
o
Proper choice for o :
m
m m
m
m
L L L L
L
2 10 10 10 2
0
ideal
0
ideal

= s < = o
L
0

Engineering Optimization Concepts and Applications
Dichotomous search (3)
Example: m = 10
1024 2
0
10
0
ideal
10
L L
L = =
10240 10
0
ideal
10
L L
= = o
|
|
.
|

\
|
0
10
log
L
L
m
Ideal
interval
reduction
10240
0
L
= o
m
Engineering Optimization Concepts and Applications
Sectioning - Fibonacci
Situation:
minimum
bracketed
between x
1

and x
3
:
x
1
x
3
Fibonacci,
1180?-1250?
x
2
Test new points and reduce interval
x
4
x
4
Optimal point placement?
Engineering Optimization Concepts and Applications
Optimal sectioning
Fibonacci method: optimal sectioning method
Given:
Initial interval [a
0
, b
0
]
Predefined total number of evaluations N, or:
Desired final interval size c
Engineering Optimization Concepts and Applications
Fibonacci sectioning - basic idea
Start at final interval and use symmetry and maximum
interval reduction:
I
N-1
= 2I
N
I
N-2
= 3I
N
I
N-3
= 5I
N
I
N-4
= 8I
N
I
N-5
= 13I
N
o << I
N

I
N

N j j N
I F I
1 +
=
Fibonacci number
=
j
F
1 2
=
k k k
I I I
j N j N j N
I I I

+ =
1 2
Engineering Optimization Concepts and Applications
Sectioning Golden Section
For large N, Fibonacci fraction | converges to golden
section ratio | (0.618034):
N
N
F
F
1
Golden section method
uses this constant interval
reduction ratio |
|
|
1
1
Engineering Optimization Concepts and Applications
Sectioning - Golden Section
Origin of golden section:
I
1

I
3
= |I
2
I
2
= |I
1
I
2
= |I
1
3 2 1
I I I + =
1
2
1 1
I I I | | + =
2
5 1
0 1
2 , 1
2

= = + | | |
618034 . 0
2
1 5
~

= |
Final interval:
1
I I
N
N
| =
Engineering Optimization Concepts and Applications
|
|
.
|

\
|
0
10
log
L
L
m
Ideal dichotomous
interval reduction
Fibonacci
Golden
section
Evaluations
Comparison sectioning methods
N
Dichotomous 12
Golden section 9
Fibonacci 8
(Exhaustive 99)
Example:
reduction to 2% of
original interval:
Conclusion: Golden section simple and near-optimal
Engineering Optimization Concepts and Applications
Quadratic interpolation
Three points of the bracket define interpolating quadratic
function:
c bx ax x f + + =
2
) (
~
a
i
b
i New point evaluated at
minimum of parabola:
a
b
x b ax f
new
2
0 2 '
~
= = + =
x
new
For minimum: a > 0!
Shift x
new
when very close to existing point
a
i+1
b
i+1
Engineering Optimization Concepts and Applications
Unconstrained optimization
algorithms
Single-variable methods
0
th
order (involving only f )
1
st
order (involving f and f )
2
nd
order (involving f, f and f )
Multiple variable methods
Engineering Optimization Concepts and Applications
Cubic interpolation
Similar to quadratic interpolation, but with 2 points and
derivative information:
d cx bx ax x f + + + =
2 3
) (
~
a
i
b
i
Engineering Optimization Concepts and Applications
Bisection method
Optimality conditions: minimum at stationary point
Root finding of f
Similar to sectioning methods, but uses derivative:
f
f
Engineering Optimization Concepts and Applications
Secant method
Also based on root finding of f
Uses linear interpolation
f
Engineering Optimization Concepts and Applications
Unconstrained optimization
algorithms
Single-variable methods
0
th
order (involving only f )
1
st
order (involving f and f )
2
nd
order (involving f, f and f )
Multiple variable methods
Engineering Optimization Concepts and Applications
Newtons method
Again, root finding of f
Basis: Taylor approximation of f :

2
'( ) '( ) ''( ) ( ) f x h f x f x h o h + = + +
Linear
approximation
) ( "
) ( '
x f
x f
h =
New guess:
) ( "
) ( '
1
k
k
k k k k
x f
x f
x h x x = + =
+
Engineering Optimization Concepts and Applications
Newtons method
Best convergence of all methods:
Unless it diverges
x
k
f

x
k+1
x
k+2
f

x
k
x
k+1
x
k+2
Engineering Optimization Concepts and Applications
Summary single variable methods
Bracketing +
Dichotomous sectioning
Fibonacci sectioning
Golden ratio sectioning
Quadratic interpolation
Cubic interpolation
Bisection method
Secant method
Newton method
And many, many more!
0
th
order
1
st
order
2
nd
order
In practice:
additional tricks
needed to deal
with:
Multimodality
Strong
fluctuations
Round-off
errors
Divergence