Scribd
Upload
3 views

OPTIMIZATION _Lecture1_

The document presents various optimization techniques, including classical and stochastic methods, highlighting their applications in solving optimization problems. It discusses the characteristics and classifications of these techniques, such as heuristics and metaheuristics, and provides examples using MATLAB for practical understanding. The content is aimed at educating readers on the mathematical foundations and practical implementations of optimization strategies.

Uploaded by

gi7282
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
3 views

OPTIMIZATION _Lecture1_

The document presents various optimization techniques, including classical and stochastic methods, highlighting their applications in solving optimization problems. It discusses the characteristics and classifications of these techniques, such as heuristics and metaheuristics, and provides examples using MATLAB for practical understanding. The content is aimed at educating readers on the mathematical foundations and practical implementations of optimization strategies.

Uploaded by

gi7282
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 30

Optimization

Techniques
[AcSIR-01-ES-AD-009]

Presented by:
Dr. R. S. Bisht, Pr. Scientist,
Lecture-1 [email protected]
CSIR-CBRI, Roorkee, India
Contents

• Classical Optimization techniques (single and multivariable


optimization with and without constraints)

• Stochastic Optimization techniques (multivariable optimization with


and without constraints) using PSO

• Hands-on examples will also be done in MATLAB for both Classical


and Stochastic Optimization methods
Optimization

A branch of mathematics devoted to solving


optimization problems

Optimization problems are mathematical functions


where we want to minimize or maximize the function
value

generally referred to as finding the best solution to an


optimization problem
Types of optimization techniques

Optimization
techniques

Classical (Gradient
Stochastic method, Hessian method,
Newton’s method)

Meta-
Heuristics heuristics
(PSO)
Classification

Classical
Toutouh, Jamal. "Natural computing for vehicular networks." PhD thesis (2016).

Ahsan, R., Shahzad, W. and Baig, A.R., 2011, December. Trends and challenges in dynamic optimization problems. In 2011 11th International
Conference on Hybrid Intelligent Systems (HIS) (pp. 283-288). IEEE.
Kumar, A. and Bawa, S., 2020. A comparative review of meta-heuristic approaches to optimize the SLA violation
costs for dynamic execution of cloud services. Soft Computing, 24(6), pp.3909-3922.
Classical optimization

 Analytical process based on differential calculus


 The classical optimization techniques are very useful to obtain the
optimal solution of problems involving continuous and
differentiable functions.
 Such types of techniques are analytical in nature to obtain maximum
and minimum points for unconstrained and constrained continuous
objective functions.
Stochastic optimization

Real-life optimization problems contain rich and soft


constraints, as well as non-smooth objective functions,
which makes many of them NP-hard optimization or or
global optimization problems.
In addition, they frequently are of large size, including
hundreds or thousands of variables (e.g., customers,
products, assets, facilities, etc.). In this scenario, exact
optimization methods are of limited effectiveness
Juan, A.A., Keenan, P., Martí, R., McGarraghy, S., Panadero, J., Carroll, P. and Oliva, D., 2021. A review of the role of
heuristics in stochastic optimisation: From metaheuristics to learnheuristics. Annals of Operations Research, pp.1-31.
Heuristic
 A heuristic is designed to provide better
computational performance as compared to
conventional optimization techniques, at the
expense of lower accuracy.
 Heuristic algorithms are intended to find out the
good solution to an optimization problem by ‘trial –
and-error’ in a reasonable amount of time.
 Heuristics are problem-dependent techniques
 they usually get trapped in a local optimum and thus
fail, in general, to obtain the global optimum solution.
Metaheuristic

 It is a repetitive process used to solve a problem


that even the toughest algorithm, with greatest
optimization powers, fail to be effective or efficient
 the ‘metaheuristics’ refers to all modern higher level
nature inspired algorithms including particle swarm
optimization and ant colony optimization algorithm
 Meta-heuristics, on the other hand, are problem-
independent techniques
Classical Optimization
• A function f(x) has a local minimum at 𝑥* if
𝑓(𝑥*)<𝑓(𝑥*+ℎ) where h is a small negative or
positive disturbance around 𝑥*.

• Similarly, a function f(𝑥) has a local maximum at 𝑥* if


𝑓(𝑥*)>𝑓(𝑥*+ℎ) where h is a small negative or
positive disturbance around 𝑥*.

• Necessary Condition: Given a piecewise smooth


function f(𝑥*) that is defined in the interval, 𝑎≤𝑥≤𝑏. If
the function has an extremum 𝑥* within this period,
then,
𝑑𝑓(𝑥*)/𝑑𝑥=𝑓′(𝑥*)=0
Sufficient Condition:
Given a piecewise smooth function f(𝑥) that is defined in
the interval 𝑎≤𝑥≤𝑏.
• If 𝑓′(𝑥*)=𝑓′′(𝑥*)=⋯=𝑓𝑛−1(𝑥*)=0 and 𝑓𝑛(𝑥*)≠0, then
𝑓(𝑥*) is,
1. Minimum if 𝑓 𝑛 (𝑥*)>0 and n is even

2. Maximum if 𝑓 𝑛 (𝑥*)<0 and n is even

3. Neither minimum or maximum if n is odd


(inflection/saddle point)
 Proof: This can be easily shown using Taylor’s theorem:

Brook Taylor
(1685-1731),
Wikipedia

Note: The theory cannot make distinction between local and global extrema.
Example-1

 Analyze the extrema of 𝑓(𝑥) = 5𝑥6 − 36𝑥5 + 165 /2 𝑥4 − 60𝑥3 + 36 in the interval in the
interval −1 ≤ 𝑥 ≤ 4
MATLAB

Note: 2D plot can be plotted using plot function in MATLAB.


Multivariable Optimization with no Constraints

Necessary Condition: Given a piecewise smooth function


f(𝑥) that is defined in the interval 𝑎≤𝑥≤𝑏.

If the function has an extremum 𝑥* within this period, then,


𝜕𝑓(𝑥∗) 𝜕𝑓(𝑥∗) 𝜕𝑓(𝑥∗)
𝜕𝑥 1 = 𝜕𝑥2
=⋯= =
𝜕𝑥 𝑛 0

In we think in terms of 𝑥 as one-dimensional array of


𝜕𝑓(𝑥∗)
variables, =∇𝑓(𝑥*)=0
𝜕𝑥
Sufficient Condition:
Given a piecewise smooth function f(𝑥) that is defined in
the interval 𝑎≤𝑥≤𝑏, a sufficient condition for a point 𝑥* to
be an extremum is that the matrix of second partial
derivatives (Hessian matrix, J) of f(𝑥) when evaluated at 𝑥*
is,
1. Minimum if 𝐽(𝑥*) is positive definite

2. Maximum if 𝐽(𝑥*) is negative definite

3. Inflection/saddle point if 𝐽(𝑥*) is indefinite*

Note: The theory cannot make distinction between local and global extrema.
Proof: This can be shown using Taylor’s theorem,

It may be more useful to express the above equation in a


matrix form
where J is the Hessian matrix of the function.

Note: The Hessian matrix is always square and symmetric

Otto Hesse
(1811-1874),
Wikipedia
Definition:
Minima: A matrix is positive definite if all eigenvalues are positive.
This means that |J − 𝜆 𝐼| = 0

Maxima: A matrix is negative definite if all eigenvalues are negative.

At saddle point: both positive and negative eigenvalues

Note: Eigenvalues can be calculated using eig function in MATLAB.


Example-1

 A saddle point is maximum in one variable and minimum in another. An example


𝑓(𝑥) = 𝑥12 − 𝑥22

Note: A three-dimensional surface plot can be calculated using


surf function in MATLAB.
Example: A three-dimensional surface plot

 [X,Y] = meshgrid(0:0.5:20);
 Z = sin(X) + cos(Y);
 surf(X,Y,Z)
Problem-1

 Analyze the extrema of 𝑓(𝑥1,𝑥2)=2−𝑥12 −𝑥1𝑥2− 𝑥22 in the interval in the


interval −2≤𝑥1≤2 and −2≤𝑥2≤2
 Visualize the optimal points using MATLAB

Using MATLAB in Calculus


Department of Mathematics, CSI

https://www.math.csi.cuny.edu/Computing/matlab/Tutorial/
Problem-2

 Analyze the extrema of 𝑓(𝑥1,𝑥2)=(𝑥1−1)2 (𝑥2+1)−𝑥2 in the interval in the


interval −1≤𝑥1≤3 and −2≤𝑥2≤2
 Visualize the optimal points using MATLAB
Books

Engineering optimization theory &


practices by S S Rao
Optimization in practice with MATLAB for
engineering students and professionals by
Achille Messac

You might also like