OPTIMIZATION

TECHNIQUES
[AcSIR-01-ES-AD-009]
Presented by:
Dr. R. S. Bisht, Pr. Scientist,
[email protected]
CSIR-CBRI, Roorkee, India
Contents

• Classical Optimization techniques (single and multivariable

optimization with and without constraints)

• Stochastic Optimization techniques (multivariable optimization with

and without constraints) using PSO

• Hands-on examples will also be done in MATLAB for both Classical

and Stochastic Optimization methods
• A function f(x) has a local minimum at 𝑥* if
𝑓(𝑥*)<𝑓(𝑥*+ℎ) where h is a small negative or
positive disturbance around 𝑥*.

• Similarly, a function f(𝑥) has a local maximum at 𝑥* if

𝑓(𝑥*)>𝑓(𝑥*+ℎ) where h is a small negative or
positive disturbance around 𝑥*.

• Necessary Condition: Given a piecewise smooth

function f(𝑥*) that is defined in the interval, 𝑎≤𝑥≤𝑏. If
the function has an extremum 𝑥* within this period,
then,
𝑑𝑓(𝑥*)/𝑑𝑥=𝑓′(𝑥*)=0
Sufficient Condition:
Given a piecewise smooth function f(𝑥) that is defined in
the interval 𝑎≤𝑥≤𝑏.
• If 𝑓′(𝑥*)=𝑓′′(𝑥*)=⋯=𝑓𝑛−1(𝑥*)=0 and 𝑓𝑛(𝑥*)≠0, then
𝑓(𝑥*) is,
1. Minimum if 𝑓 𝑛 (𝑥*)>0 and n is even

2. Maximum if 𝑓 𝑛 (𝑥*)<0 and n is even

3. Neither minimum or maximum if n is odd

(inflection/saddle point)
  Proof: This can be easily shown using Taylor’s theorem:

Brook Taylor
(1685-1731),
Wikipedia

Note: The theory cannot make distinction between local and global extrema.
Multivariable Optimization with no Constraints

Necessary Condition: Given a piecewise smooth function

f(𝑥) that is defined in the interval 𝑎≤𝑥≤𝑏.

If the function has an extremum 𝑥* within this period, then,

𝜕𝑓(𝑥∗) 𝜕𝑓(𝑥∗) 𝜕𝑓(𝑥∗)
𝜕𝑥 1 = 𝜕𝑥2
=⋯= =
𝜕𝑥 𝑛 0

In we think in terms of 𝑥 as one-dimensional array of

𝜕𝑓(𝑥∗)
variables, =∇𝑓(𝑥*)=0
𝜕𝑥
Sufficient Condition:
Given a piecewise smooth function f(𝑥) that is defined in
the interval 𝑎≤𝑥≤𝑏, a sufficient condition for a point 𝑥* to
be an extremum is that the matrix of second partial
derivatives (Hessian matrix, J) of f(𝑥) when evaluated at 𝑥*
is,
1. Minimum if 𝐽(𝑥*) is positive definite

2. Maximum if 𝐽(𝑥*) is negative definite

3. Inflection/saddle point if 𝐽(𝑥*) is indefinite*

Note: The theory cannot make distinction between local and global extrema.
Definition:
Minima: A matrix is positive definite if all eigenvalues are positive.
This means that |J − 𝜆 𝐼| = 0

Maxima: A matrix is negative definite if all eigenvalues are negative.

At saddle point: both positive and negative eigenvalues

Note: Eigenvalues can be calculated using eig function in MATLAB.

Proof: This can be shown using Taylor’s theorem,

It may be more useful to express the above equation in a

matrix form
 where J is the Hessian matrix of the function.

Note: The Hessian matrix is always square and symmetric

Hessian Matrix MATLAB

syms x y z
f = x*y + 2*z*x; Otto Hesse
(1811-1874),
hessian(f,[x,y,z])
Wikipedia
Optimal values
Convexity and Concavity of Functions of
Single and Multi Variables
 convex
concave

 The sign of the eigenvalue says something about the

convexity of the function along that direction (positive
implies convex, negative concave, zero inflexion point).
Convex Function (Function of
one variable)
➢ A real-valued function f defined on an interval (or on any convex subset
C of some vector space) is called convex, if for any two points x and y in
its domain C and any t in [0,1], we have

f (ta + (1− t)b)  tf (a) + (1− t) f (b))

➢ In other words, a function is convex if and only if its epigraph (the set of
points lying on or above the graph) is a convex set. A function is also said
to be strictly convex if
f (ta + (1 − t)b)  tf (a) + (1− t) f (b)
for any t in (0,1) and a line connecting any two points on the function lies
completely above the function.
Testing for convexity of a single
variable function

➢ A function is convex if its slope is non decreasing or

 2 f / x 2  0. It is strictly convex if its slope is continually
increasing or  2 f / x 2  0 throughout the function.
Concave function (function of one
variable)

⚫ A differentiable function f is concave on an interval if its derivative

function f ′ is decreasing on that interval: a concave function has a
decreasing slope.
⚫ A function f(x) is said to be concave on an interval if, for all a and b
in that interval,
t [0,1], f (ta + (1− t)b)  tf (a) + (1− t) f (b)

⚫ Additionally, f(x) is strictly concave if

t [0,1], f (ta + (1− t)b)  tf (a) + (1− t) f (b)

Testing for concavity of a single
variable function

⚫ A function is convex if its slope is non increasing or

 2 f / x 2  0. It is strictly concave if its slope is continually
decreasing or  2 f / x 2  0 throughout the function.
Example
Consider the example in lecture notes 1 for a function of two
variables.
Locate the stationary points of f (x) = 12x 5 − 45x 4 + 40x 3 + 5

and find out if the function is convex, concave or neither at the points
of optima based on the testing rules discussed above.
Functions of two variables

⚫ To determine convexity or concavity of a function of

multiple variables, the eigen values of its Hessian matrix
is examined and the following rules apply.
– If all eigen values of the Hessian are positive the function is
strictly convex.
– If all eigen values of the Hessian are negative the function is
strictly concave.
– If some eigen values are positive and some are negative, or if
some are zero, the function is neither strictly concave nor strictly
convex.
Example
 Consider the example in lecture notes 1 for a function of two variables
Locate the stationary points of f(X) and find out if the function is conve
concave or neither at the points of optima based on the rules discussed
lecture.
 f (X) = 2x 3 / 3 − 2x x− 5x + 2x 2 + 4x + 5
 1 1 2 1 2 2
 Example
(contd..)

23
 Example: A three-
dimensional surface
plot

 [X,Y] = meshgrid(0:0.5:20);
 Z = sin(X) + cos(Y);
 surf(X,Y,Z)

[X,Y] = meshgrid(1:0.5:10,1:20);
Z = sin(X) + cos(Y);
surfc(X,Y,Z)
clear all
[x,y] = meshgrid(-2:0.05:2);
f = x.*exp(-x.^2-y.^2)+(x.^2+y.^2)/20;

surfc(x,y,f)
 Newton’s Method for single variable

 Consider a equation to be solved:

 f(x)=0;
 Step 1: give an initial point x0
 Step 2: xn+1=xn-f(xn)/f’(xn)
 Step 3: is f(xn) small enough? If not go back to step 2
 Step 4: stop

Isaac Newton (1642-1726), Wikipedia

 Newton’s Method (example single variable)
MATLAB code for i=1:N
x=x-f1(x)/f2(x);
j=i+1;
clc; clear all; close all; xx(j)=x;
% syms x ii=i;
f=@(x) 2*sin(x)-0.1*x^2; Err=abs(xx(j)-xx(j-1));
% fd=2*sin(x)-0.1*x^2' if Err<err
% f1=diff(fd,1) break;
% f2=diff(fd,2) end
f1=@(x) 2*cos(x) - 0.2*x end
f2=@(x)- 2*sin(x) - 0.2 FF=f(x);
N=100; % Number of Iterations disp(['the optimal value at : '
x=2.5; num2str(x) ', is:' num2str(FF), ',No
iterations : ' num2str(ii)] )
xx(1)=x; % x history x=0:0.1: 10
err=0.01; % Result accuracy F=2.*sin(x)-0.1*x.^2;
plot(x,F)
hold on
plot(1.4276, 1.7757, 'o')
Newton’s Method for multiple variables
Newton’s Method for multiple variables
(Problems)
Multi-variate Newton’s Method:

1.
0.5
Newtons method for 𝑓 𝑥 = 𝑥13 + 2 𝑥1 − 𝑥2 2
− 3𝑥1 , start from 𝑥 (0) =
0.5
1
Ans: 𝑥 (𝑜𝑝𝑡) =
1

2.
1
Newtons method for 𝑓 𝑥 = 𝑥1 𝑥2 + 4𝑥2 − 3𝑥12 − 𝑥22 , start from 𝑥 (0) =
−1
0.36
Ans: analytical solutions: 𝑥 (𝑜𝑝𝑡) =
2.18