Engineering Optimization

Concepts and Applications

Fred van Keulen

Matthijs Langelaar
CLA H21.1
[email protected]

Engineering Optimization – Concepts and Applications

 Contents
● Unconstrained Optimization:
Methods for Multiple Variables
– 1st order methods: CG

– 2nd order methods

– Quasi-Newton Methods

● Constrained Optimization: Optimality Criteria

Engineering Optimization – Concepts and Applications

 Multivariate Unconstrained
Optimization Algorithms
● Zeroth order (direct search):

– Random methods: random jumping / walk / S.A.

– Cyclic coordinate search

– Powell’s Conjugate Directions method

– Nelder and Mead Simplex method

– Biologically inspired algorithms (GA, swarms, …)

● Conclusions:

– Nelder-Mead usually best

– Inefficient for N>10

Engineering Optimization – Concepts and Applications

 Fletcher-Reeves conjugate
gradient method
● Based on building set of conjugate directions,
combined with line searches
T
● Conjugate directions: d i Ad j  0 | i  j
● Conjugate directions: guaranteed convergence in N
steps for quadratic problems
(recall Powell: N cycles of N line searches)
1 T T
Quadratic function: f (x)  x Ax  b x  c
2
Engineering Optimization – Concepts and Applications
 CG practical
1. Start with abritrary x1

2. Set first search direction: d1  f1

3. Line search to find next point: x i 1  x i   i d i
2
f i 1
4. Next search direction: di 1  f i 1  2
di
f i
5. Repeat 3

6. Restart every (n+1) steps, using step 2

Engineering Optimization – Concepts and Applications

 CG properties

● Theoretically converges in N steps

or less for quadratic functions
● In practice:

– Non-quadratic functions

– Finite line search accuracy Slower convergence;

– Round-off errors > N steps

● After N steps / bad convergence: restart procedure

d N 1  f N 1 etc.

Engineering Optimization – Concepts and Applications

 Application to mechanics (FE)
1 T
● Structural mechanics: f U   U KU  F T U
2
Quadratic function!
● Equilibrium: f  KU  F  0
KU  Fint 
 f  Fext  Fint  R
F  Fext 
● CG: d1  R1  R U1 
T
dk R k
Line search: U k 1  U k   k d k k  T
2
d k Kd k
f i 1 R k 1T R k 1
di 1  fi 1  2
di  R k 1 
f i RkT Rk
● Simple operations on element level. Attractive for large N!
Engineering Optimization – Concepts and Applications
 Multivariate Unconstrained
Optimization Algorithms (2)
● First order methods (descent methods):

– Steepest descent method (with line search)

– Fletcher-Reeves Conjugate Gradient method

– Quasi-Newton methods y2

● Conclusions (for now):

– Scaling important for Steepest descent (zig-zag) y1

– For quadratic problem, CG converges in N steps

Engineering Optimization – Concepts and Applications

 Unconstrained optimization
algorithms
● Single-variable methods

● Multiple variable methods

– 0th order

– 1st order

– 2nd order

Engineering Optimization – Concepts and Applications

 Newton’s method
● Concept:

– Construct local quadratic approximation

– Minimize approximation

– Repeat

● Local approximation: 2nd order Taylor series:

1 T
f (x  x)  f (x)  f  x  x Hx  f (x)
T

2
● First order sufficiency condition applied to approximation:

f (x)  0  f  Hx  0  x   H 1f

Engineering Optimization – Concepts and Applications

 Newton’s method (2)
● Step: s   H 1f
Evaluated at x

● Update: x k 1  x k  s k sk
xk x k 1
● Note:

– Finds minimum of quadratic functions in 1 step!

– Step includes solving (dense) linear system of equations

– If H not positive definite, divergence occurs

Engineering Optimization – Concepts and Applications

 Similarity to linear FE
● Linear mechanics problem: min  (u)
u

f   ( f ) (quadratic in u)

K   2 ( H )

● Newton: s   H 1f  u  K 1f

(Evaluated at u = 0)

● Nonlinear mechanics similar, with multiple steps.

Engineering Optimization – Concepts and Applications

 Newton’s method (3)
● To avoid divergence: line search.
Error
1
Search direction: s   H f 0
10

● Update: x k 1  x k   k s k -5
10

 k sk
xk x k 1
-10
10

● Newton’s method has quadratic

convergence (best!) close to
-15
10

optimum: 2
x k 1  x*  c x k  x* Iteration
1 2 3 4

Engineering Optimization – Concepts and Applications

 Levenberg-Marquardt method
● Problems in Newton method:

– Bad convergence / divergence when far from optimum

– What to do when H singular / not positive definite?

~
● Remedy: use modified H: H H I
with  such that H positive definite
● Levenberg-Marquardt method: start with large k, and
decrease gradually:
 large: s k   H   k I 1
f  f Steepest descent
s k  H   k I  f   H 1f
1
 small: Newton

Engineering Optimization – Concepts and Applications

 Trust region methods
● Different approach to make Newton’s method more
robust: local approximation locally valid:
 1 T
min f (x)  f (x)  f  x  x Hx x2
T

x 2
s.t. x   Defines trust region
x1
● Trust region adjusted based on approximation quality:
f ( x )  f ( x  x ) 1
r rk  0.25  k 1  x k
f (x)  f (x) 4
Actual reduction
rk  0.75  k 1  2 x k
Predicted reduction else  k 1   k
Engineering Optimization – Concepts and Applications
 Trust region methods (2)
● Performance: robust, can also deal with neg. def. H

● Not sensitive to variable scaling

● Similar concept in computational mechanics:

arc-length methods
● Trust region subproblem is quadratic constrained
optimization problem:
– Rather expensive

– Approximate solution methods popular:

dogleg methods, Steihaug method, …
Engineering Optimization – Concepts and Applications
 Newton summary

● Second order method: Newton’s method

● Conclusions:

– Most efficient: solves quadratic problem in 1 step

– Not robust!

● Robustness improvements:

– Levenberg-Marquardt (blend Steepest decent / Newton)

– Trust region approaches

Engineering Optimization – Concepts and Applications

 Quasi-Newton methods
● Drawbacks remain for Newton method:

– Need to evaluate H (often impractical)

– Storage of H, solving system of equations

● Alternatives: quasi-Newton methods:

– Based on approximating H (or H-1)

– Use only first-order derivative information

Newton step: Quasi-Newton step:
~ 1
x k 1  x k   k H f 1
x k 1  x k   k H f
~ ~ 1
Update H (or H )
Engineering Optimization – Concepts and Applications
 Quasi-Newton fundamentals
● First order Taylor approximation of gradient:

f k 1  f k  Hx k 1  x k 
~
 f k 1  f k  H k 1 x k 1  x k 

● Operating on H-1 avoids solving a linear system each

~
step. Define: H
1
B  x k 1  x k  B k 1 f k 1  f k 

● Update equations: B k 1  B k  B k

B k  auuT Rank one update
B k  auuT  bvvT Rank two update
Engineering Optimization – Concepts and Applications
 Update functions: rank 1

● Notation:   f k 1  f k , x  x k 1  x k

● Broyden’s update:
1
 x  B   x  B  
T
B k 
 x  B  
T k k
k 

● Drawback: updates not guaranteed to remain positive

definite (possible divergence)

Engineering Optimization – Concepts and Applications

 Update functions: rank 2
● Davidson-Fletcher-Powell (DFP):

  
T

B  B
k 
T
xx k
B k  T 
x  T B k 
● Broyden, Fletcher, Goldfarb, Shannon (BFGS):

 T B k   xxT xT B k  B k xT

B k  1  T  T  T
 x    x   x 

● Both DFP and BFGS are conjugate gradient methods

● Rank 2 quasi-Newton methods: best general-purpose

unconstrained optimization (hill climbing) methods
Engineering Optimization – Concepts and Applications
 BFGS example
● Minimization of quadratic function:
2 2
f ( x1 , x2 )  x1  x2  2 x1  2 x1 x2  x2

1 0
Starting point: x1  (0,0) B1   
 0 1 
 1  4 x1  2 x2  1
f      f1   
 1  2 x1  2 x2   1
 1 Line search:
 s1  B1f1     1
1 1  1  x 2  x1  1s1   
1
Engineering Optimization – Concepts and Applications
 BFGS example (2)
 1  4 x1  2 x2   1
f     f 2   
 1  2 x1  2 x2   1
1  1 1 1  1 1
BFGS: B1    B2  
2  1 3  2  1 5 
0  Line search:
 s 2   B 2 f 2   
2  1  1 
1   x 3  x 2   2s 2   
4 3 2
0
f 3    Solution found.
0
Engineering Optimization – Concepts and Applications
 BFGS example (3)
2 2
Check: f ( x1 , x2 )  x1  x2  2 x1  2 x1 x2  x2

 1  4 x1  2 x2   4 2 1  1  1
f    H 
1
BH   
  1  2 x1  2 x 2  2 2  2   1 2 
1 0 0 
BFGS: B 2  
2 0 3
 B 3  B 2  B 2

1  1  1 1 0 0  1  1  1
     
2  1 5  2 0  3 2  1 2 
 

Engineering Optimization – Concepts and Applications

 Comparison on Banana function
● Nelder-Mead simplex: 210 Newton
DFPdescent
Nelder-Mead
Levenberg-Marquardt
Steepest
BFGS simplex
● Steepest descent: >300
(no convergence)

● BFGS: 53

● DFP: 260

● Newton: 18

● Levenberg-Marquardt: 29

Engineering Optimization – Concepts and Applications

 Summary multi-variable
algorithms for unconstrained
optimization
● Theoretically nothing beats Newton, but:
– Expensive computations and storage (Hessian)

– Not robust

● Quasi-Newton (BFGS) best if gradients available

● Zeroth-order methods robust and simple, but inefficient for n > 10

● CG inferior to quasi-Newton methods, but calculations simple:

more efficient for n > 1000
● Exact line search important!

● Variable scaling can have large impact

Engineering Optimization – Concepts and Applications

 Contents
● Unconstrained Optimization:
Methods for Multiple Variables
– 1st order methods: CG

– 2nd order methods

– Quasi-Newton Methods

● Constrained Optimization: Optimality Criteria

Engineering Optimization – Concepts and Applications

 Summary optimality conditions
● Conditions for local minimum of unconstrained problem:

– First Order Necessity Condition: f  0

– Second Order Sufficiency Condition: H positive
definite

● For convex f in convex feasible domain:

condition for global minimum:
– Sufficiency Condition: f  0

Engineering Optimization – Concepts and Applications

 Boundary optima
● Today’s topic:
optimality conditions for constrained problems
g2 f
f Interior
x2 optimum

min f ( x) g1
x

s. t. g ( x)  0
h ( x)  0
Boundary
optima
x1
Engineering Optimization – Concepts and Applications
 Feasible perturbations / directions
● Consider feasible space X

● Feasible perturbation: x | x  x  X , x  X

● Feasible direction s: g2
x2
line in direction s remains y X g1
in X for some finite length: x

s | y  s  X
y  X ,   0,  ,   0
x1

Engineering Optimization – Concepts and Applications

 Boundary optimum
● Necessary condition for boundary optimum: f cannot
decrease further in any feasible direction

f T s  0 s f
g2
x2
(no feasible direction exists g1
s
for which f decreases)

● Approach for numerical

algorithms: move along x1
feasible directions until  condition holds

Engineering Optimization – Concepts and Applications

 Equality constrained problem
● First, only equality constraints considered:

– Simplest case!

– Active inequality constraints can be treated as equality

constraints
– Active inequality constraints can be identified by e.g.
Monotonicity Analysis

min f ( x) x  n
x

s. t. hi (x)  0 i  1 m

Engineering Optimization – Concepts and Applications

 Equality constrained problem (2)
● Each (functionally independent) equality constraint
reduces the dimension of the problem:
min f ( x) x  n
x
Problem dimension
s. t. hi (x)  0 i  1 m  nm
● Solutions can only exist in the feasible subspace X
of dimension n – m (hypersurface)

● Examples: n = 3, m = 2: X = line (1-D)

n = 3, m = 1: X = surface (2-D)

Engineering Optimization – Concepts and Applications

 Description of constraint surface
● Why?

– Local characterization of constraint surface leads to

optimality conditions
– Basis for numerical algorithms

● Assumptions:

– Constraints differentiable, functionally independent

– All points are regular points:

Constraint gradients hi all linearly independent

Engineering Optimization – Concepts and Applications

 Some definitions …
● Normal hyperplane: spanned by all gradients, normal to
constraint surface
z   n :

N (x)  z  x  1h1 (x)   2h2 (x)     mhm (x),
  
 i
● Tangent hyperplane: orthogonal to the normal plane:

y   n :
T ( x)  
h(x) T y  x   0

Engineering Optimization – Concepts and Applications

 Normal / tangent hyperplane (2)
● Example: x3
h2 : x2  c2

Normal
hyperplane

x h2
x2
h1
h1 : x1  c1

x1 Tangent
hyperplane (line)
Engineering Optimization – Concepts and Applications
 Optimality conditions
● Simplest approach:

– Eliminate variables using equality constraints

– Result: unconstrained problem of dimension n – m

– Apply unconstrained optimality conditions

● But often not possible / practical:

– Elimination fails when variables cannot be explicitly

solved from equality constraints
– No closed form of objective function (e.g. simulations)

Engineering Optimization – Concepts and Applications

 Local approximation
● Approach: build local approximation for constrained
case, using (very small) feasible perturbations:
min f ( x) x  n
x

s. t. hi (x)  0 i  1 m
n
f
f (x)  f  x 
T
xi
i 1 xi
 n h j
h j (x)  h j  x 
T
xi  0 j  1 m
i 1 xi
m + 1 equations, n + 1 unknowns (f, xi), where n > m
 n - m = p degrees of freedom
Engineering Optimization – Concepts and Applications
 Local approximation (2)
● Divide design variables in two subsets:

– p decision/control variables d s 
x 
– m state/solution variables s d 

p
f
n m
f f
f (x)   xi  si   d i
i 1 xi i 1 si i 1 d i
n h j m h j p
h j
h j (x)   xi  0   si   d i
i 1 xi i 1 si i 1 d i
dependent independent
Engineering Optimization – Concepts and Applications
 Dependent/independent variables
● Example: 2
h : x1  3 x2  x3  0
x3

x2

x1

2
x3  x1 2
x1  x3  3x2 x2  x3  x1  3x2
3
Engineering Optimization – Concepts and Applications
 Local approximation (3)
● Eliminating solution variable perturbation (dependent):
p
m
f f f f
f   si   d i  s  d
i 1 si i 1 d i s d

  m  
    h p
h j  h h
h j    si   d i   s  d  0
j

    i 1 si i 1 d i  s d
    
1
h h h h
s   d  s   d
s d s d

Engineering Optimization – Concepts and Applications

 Reduced gradient
● Variation of f expressed in decision variables:

f f  f f h 1 h 
f  s  d f    d
s d  d s s d 
1
h h Reduced / constrained
s   d
s d gradient

● f locally expressed in decision variables only:


f (s(d), d)   (d) f    d
d

Engineering Optimization – Concepts and Applications

 Optimality condition
●  unconstrained function of decision variables d 
unconstrained optimality condition can be used:
T

   (d)    0
d
 Optimality condition for equality-constrained problem:
1
f f h h
  0T
d s s d

Reduced gradient zero.

Engineering Optimization – Concepts and Applications

 Example L
x1

min f  mgx2 , h  L(1  cos x1 )  x2  0

x1 , x2 x2
 x1  d  m
Take x       min f  mgs
 x2   s  d ,s

s.t . h  L(1  cos d )  s  0

f f
 mg , 0
s d
h h
 1,  L sin d 
s d
1
f f h h
 0  mg  1 L sin d  mgL sin d
1
  
d s s d
Equilibrium:   0  mgL sin d  0  d  0,  ,2 ,
Engineering Optimization – Concepts and Applications
 Lagrange approach
● Alternative way to formulate optimality
conditions: formulate Lagrangian:
L(x,  )  f (x)  λ T h(x) x  n , λ  m
● Consider stationary point of L: Lagrange multipliers

  x L   x f  λ T  x h  T
L    0
 λ L   ( λ λ )h 
T

1
 f T h T f h
 s  λ  0 T
 λ  
s s s
 1
 f  λ T h  0T   f  f h h
  0T
 d d d s s d
Engineering Optimization – Concepts and Applications
 Example
min f  mgx2 , h  L(1  cos x1 )  x2  0
x1 , x2
x1
L(x, λ )  f (x)  λ T h( x) L

 mgx2    L(1  cos x1 )  x2  x2

m
 T  L sin x1 
 L L L   
L  0     mg       mg
 x1 x2    
 L (1  cos x1 )  x2

 mgL sin x1  0

Equilibrium: x1  0,  , 2 ,
Engineering Optimization – Concepts and Applications
 Comparison
f (Often not
● Elimination: n – m variables,  0T
x possible)

● Reduced gradient: 1
f f h h
  0T
m decision variables, d s s d
n - m solution variables (total n)
● Lagrange approach: h0
1
n design variables, T f h
λ 
m multipliers (total m + n) s s
f T h
λ  0T
x x
Engineering Optimization – Concepts and Applications
 Application in mechanics

 
● Lagrangian method widely used to enforce kinematic
constraints, e.g. multibody dynamics:
– Variational formulation of
equations of motion (planar case):
3n  d  T  T 
    
 i  qi
i 1  dt  q
 Qi qi  0
                                                       


3 equations of Generalized
motion per body Total kinetic forces
energy Generalized
displacement

Engineering Optimization – Concepts and Applications

 Multibody dynamics (2)
● Bodies connected by m joints  2m constraint equations
(1 DOF per joint):  j (q)  0 j  1 2m
● Constrained equations of motion (based on Lagrangian):
3n  d  T  T 2m  j 
    
 i  qi
i 1  dt  q
 Qi    j
j 1 qi 
qi  0

Same units as force.

● System simulation: solve  Interpretation: force required
to satisfy constraint.
for qi (3n) and j (2m)
simultaneously

Engineering Optimization – Concepts and Applications

 Geometrical interpretation
● For single equality constraint: simple geometrical
interpretation of Lagrange optimality condition:

f h x2
f
  0T
x x h
Meaning: h
f h
//
x x f
x1
Gradients parallel 
tangents parallel  h tangent to isolines
Engineering Optimization – Concepts and Applications