MULTI VARIABLE OPTIMIZATION

Min f(x1, x2, x3,----- xn)

UNIDIRECTIONAL SEARCH Min Point r

s
- CONSIDER A DIRECTION S
r r
x(α ) = x + α s
- REDUCE TO
Min f (α )
- SOLVE AS
A SINGLE VARIABLE PROBLEM
Uni directional search (example)

Min f(x1, x2) = (x1-10)2 + (x2-10)2

S = (2, 5) (search direction)
X = (2, 1) (Initial guess)
DIRECT SERACH METHODS

- SEARCH THROUGH MANY DIRECTIONS

- FOR N VARIABLES 2N DIRECTIONS
-Obtained by altering each of the n values and taking all
combinations
 EVOLUTIONARY OPTIMIZATION METHOD

- COMPARE ALL 2N+1 POINTS & CHOOSE THE BEST.

- CONTINUE TILL THERE IS AN IMPROVE MENT
- ELSE DECREASE INCREAMENT

STEP 1: x0 = INITIAL POINT

∆i = STEP REDUCTION PARAMETER FOR EACH VARIABLE

∈ = TERMINATION PARA METER
STEP 2 : IF ∆ <∈ STOP
ELSE CREATE 2N POINTS
xi = xi ± ∆i / 2
STEP 3 : x = Min (2 N + 1)
STEP 4 : IF x = x 0 − ∆ i = ∆ i / 2 ; GOTO 2
ELSE x0 = x & GOTO 2
 Hooke Jeeves pattern search
• Pattern Search ---
– Create a set of search directions iteratively
– Should be linearly independent
• A combination of exploratory and pattern moves
– Exploratory – find the best point in the vicinity of the
current point
– Pattern – Jump in the direction of change, if better then
continue, else reduce size of exploratory move and
continue
 Exploratory move
• Current solution is xc; set i = 1; x = xc
• S1: f = f(x), f+ = f(xi + ∆i), f- = f(xi - ∆i)
• S2: fmin = min (f, f+, f-); set x corresponding
to fmin
• S3: If i = N, go to 4; else i = i + 1, go to 1
• S4: If x ≠ xc, success, else failure
 Pattern Move
• S1: Choose x(0), ∆I, for I = 1, 2, …N, ε, and
set k = 0
• S2: Perform exploratory move with xk as
base point;
– If success, xk+1 = x, go to 4 else goto 3
• S3: If |∆| < ε, terminate
– Else set ∆i = ∆i / α --- ∀i, go to 2
 Pattern Move (contd)
• S4: k = k+1; xpk+1 = xk + (xk – xk-1)
• S5: Perform another exploratory move with
xpk+1 as the base point; Result = xk+1
• S6: If f(xk+1) < f(xk), goto S4
– Else goto S3
 Example :
• Consider the Himmelblau function:

f ( x1 , x2 ) = ( x12 + x2 − 11) 2 + ( x1 + x22 − 7 ) 2

• Solution
Step 1 Selection of initial conditions
1. Initial Point : x ( 0 ) = (0,0) T
2. Increment vector : ∆ = (0.5,0.5) T
3. Reduction factor : α =2
4. Termination parameter : ε = 10
−3

5. Iteration counter : k=0

• Step 2
Perform an Iteration of the exploratory move
with base point as x = x ( 0 )

Thus we set x = x ( 0 ) = (0,0) T and i = 1

The exploratory move will be performed with the

following steps
Steps for the Exploratory move
Step 1 : Explore the vicinity of the variable x1
Calculate the function values at three points
+
( x ( 0) + ∆1 x ( 0 ) )T = (0.5,0.5)T f = f ((0.5,0.5) T ) = 157.81

x (0)
= (0,0) T f = f ((0,0) T ) = 170

−
(x (0)
− ∆1 x ) = (−0.5,0.5)
(0) T T
f = f ((−0.5,0.5) T ) = 171.81

Step 2 : Take the Minimum of above function and

corresponding point
Step 3 : As i ≠ 1 : all variables are not explored
Increment counter i=2 and explore second variable.
First iteration completed
Step1: At this point the base point is x = (0.5,0) T explore
the variable x2 and calculate the function values.
+
f = f ((0.5,0.5) T ) = 144.12
f = f ((0.5,0)T ) = 157.81
−
f = f ((0.5,−0.5) T ) = 165.62

Step 2 : fmin = 144.12 and point , x=(0.5,0.5)

Step 3 : As i=2 move to the step 4 of the exploratory move

Step 4 : ( of the Exploratory move )
Since x ≠ x c the move is success and we set
x = (0.5,0.5) T
 • As the move is success, set x (1) = x = (0.5,0.5) T move
to step 4

STEP 4 : We set k=1 and perform Pattern move

x ( 2 ) p = ( x (1) + ( x (1) − x ( 0 ) ) T ) = 2(0.5,0.5) T − (0,0) T = (1,1) T
Step 5 : Perform another exploratory move as before
(2)
and with x p as the base point.
The new point is x = (1.5,1.5) T
Set the new point x ( 2) = x = (1.5,1.5) T
Step 6 : f ( x ( 2)
) = 63 . 12 is smaller than f ( x (1)
) = 144.12
Proceed to next step to perform another pattern move
STEP 4 : Set k=2 and create a new point
x ( 3) p = (2 x ( 2 ) − x (1) ) = (2.5,2.5) T
Note: as x ( 2 )is better than x (1) , a jump along the direction
( x ( 2 ) − x (1) ) is made, this will take search closer to
true minimum

STEP 5 : Perform another exploratory move to find any

better point around the new point.
Performing the move on both variables we have
New point
x (3) = (3.0,2.0) T
This point is the true minimum point
 In the example the minimum of the Hookes-Jeeves
algorithm happen in two iterations : this may not be
the case always

Even though the minimum point is reached there is

no way of finding whether the optimum is reached
or not
The algorithm proceeds until the norm if the
increment vector is small.
STEP 6 : function value at new point
f ( x 3 ) = 0 < f ( x 2 ) = 63.12
Thus move on to step 4
STEP 4 : The iteration counter k = 3 and the new point
x ( 4 ) p = (2 x (3) − x ( 2 ) ) = (4.5,2.5) T
STEP 5 : With the new point as base the search is
success and x=(4.0,2.0) and thus we set
x ( 4 ) = (4.0,2.0) T
STEP 6 : The function value is 50, which is larger that
the earlier i.e. 0 . Thus we move to step 3
Step 3 : Since ∆ = 0.5 ⊄ ε we reduce the increment vector
∆ = (−0.5,0.5) T / 2 = (0.25,0.25) T
and proceed to Step 2 to perform the iterations
Step 2 : Perform an exploratory move with the
following as current point x (3) = (3.0,2.0) T
The exploratory move on both the variables is failure
and we obtain x (3) = (3.0,2.0) T
thus we proceed to Step 3
Step 3 : Since ∆ is not small reduce the increment
vector and move to Step 2.
The new increment vector is ∆ = (0.125,0.125) T
The algorithm now continues with step 2 and step 3
until ∆ is smaller than the termination factor.
The final solution is x * = (3.0,2.0) T with the function
value 0
POWELL’S CONJUGATE DIRECTION METHOD

For a quadratic function IN 2 VARIABLES

- TAKE 2 POINTS x1 & x2 AND
- A DIRECTION ‘d’

IF y1 IS A SOLUTION OF MIN f ( x1 + λd ) &

y2 IS A SOLUTION OF MIN f ( x 2 + λd )

THEN (y2- y1) IS CONJUGATE TO d

OPTIMUM LIES ALONG (y2- y1)

y2

x2 y1

x1
OR

FROM
x1 GET y1 ALONG (1,0)
y1 GET x2 ALONG (0,1)
x2 GET y2 ALONG (1,0)

TAKE (y2 - y1 )
Alternate to the above method
- One point ( x1 ) and both coordinate
directions ((1,0) T and (0,1) T )
- Can be used to create a pair of conjugate
directions (d and ( y 2 − y 1 ))
• Point ( y 1 ) obtained by unidirectional search
along (1,0) T from the point ( x1 ) .
• Point ( x 2 ) obtained by unidirectional search
along (0,1) T from the point ( y 1 ) .
• Point ( y 2 ) obtained by unidirectional search
along (1,0) T from the point ( x 2 ) .

The figure shown also follows the Parallel

Subspace Property.
This requires Three Unidirectional searches.
 Example :
• Consider the Himmelblau function:

f ( x1 , x2 ) = ( x12 + x2 − 11) 2 + ( x1 + x22 − 7 ) 2

• Solution
Step1 : Begin with a point x (0)
= ( 0, 4) T

Initial direction s (1) = (1,0) T and s ( 2) = (0,1) T

Step 2: Find minimum along the first search direction.
Any point along that direction can be written as
x ( p ) = x ( 0 ) + αs (1)
Thus the point xp can be written as xp=(α,4)T

Now the two variable function can be expressed in

terms of one variable
F (α ) = (α 2 − 7) 2 + (α + 9) 2
We are looking for the point which the function value
is minimum.

== Following the procedure of numerical differentiation.

Using the bounding phase method, the minimum is
bracketed in the interval (1,4), and using the golden
search method the minimum α*=2.083 with three
decimals places of accuracy. Thus x1 = (2.083,4.00)T
 Similarly find the minimum point along the second
search direction.
A general point on the line is
x(α ) = ( x 1 + αs 2 ) = (2.083, (α + 4)) T
Using similar approach as said earlier we have
α*=-1.592 x2 = (2.083,2.408)T
From the above point perform a unidirectional search
along the first search direction and obtain the
minimum point
x3 = (2.881,2.408)T
Step 3 : According to the parallel subspace property,
we find the new conjugate direction
s ( 2 ) = (1,0) T
Step 4 : the magnitude of search vector d is not
small. Thus the new conjugate search direction are

s (1) = (0.798,−1.592) T / (0.798,−1.592) T

= (0.448,−0.894) T

This completes one iteration of Powell’s conjugate

direction method.
Step 2 : A single variable minimization along the search
direction s(1) from the point x(3)=(2.881,2.408)T results
in the new point x = (3.063,2.045)
One more unidirectional search along the s2 from the
point x4 results in the point x5. Another minimization
along s1 results in x6

Step 3 : the new conjugate direction is

d = ( x ( 6 ) − x ( 4 ) ) = (0.055,−0.039) T
The unit vector along this direction is ( 0.816,-0.578)
Step 4 : The new pair of conjugate search direction
are
s (1) = (0.448,−0.894) T s ( 2 ) = (0.055,−0.039) T
The search direction d ( before normalizing ) may
be considered to be small and therefore the
algorithm may be terminated
 EXTENDED PARALLEL SUBSPACE
PROPERTY.
1
Let us assume that the point y is found after
unidirectional searches along each of m ( < N )
1
conjugate directions from a chosen point x and
similarly, the point y 2 is found after
unidirectional searches along each of m
2
conjugate directions from another point x . The
vector ( y 2 − y 1 ) is the conjugate to all m search
directions.
• ALGORITHM
Step 1 : Choose starting point
And a set of N linearly independent direction
Step 2 : Minimize along N unidirectional search directions
using the previous minimum point to begin the next
search.
Step 3 : Form a new conjugate direction “d ” using the
extended parallel subspace property.
Step 4 : If || d || is small or search directions are linearly
dependent,
TERMINATE,
Else replace starting point for N directions, set s(1)=d /||d||
And go to Step 2
FOR N VARIABLES

STEP 1: TAKE x0 & N LINEARLY INDEPENDENT DIRECTIONS

s1, s2, s3, s4,------- sN s i = ei

STEP 2: - MINIMIZE ALONG N UNI-DIRECTIONAL SEARCH

DIRECTIONS,USING PREVIOUS BEST EVERY TIME
- PERFORM ANOTHER SEARCH ALONG s1

STEP 3: FROM THE CONJUGATE DIRECTION “d”

STEP 4: IF d IS SMALL TERMINATE

ELSE s j = s j −1 ∀j &
s =d/d
1

GOTO 2
 Powell’s method with N variables

• Start from x1
• Get y1 by doing search along s1
• Find y2 by doing search along s2, s3, s4 .. sn,
s1
• (y2 – y1) is conjugate to s1
• Replace sn by (y2-y1) and start same
procedure starting from s2
 GRADIENT BASED METHODS
• The methods exploit the derivative information
of the function and are faster.
• Cannot be applied to problems where the
objective function is discrete or discontinuous.
• Efficient when the derivative information is easily
available
• Some algorithms require first order derivatives
while some require first and second order
derivatives of the objective function.
• The derivatives can be obtained by numerical
computations
• Methods in Gradient Search.
– Decent Direction
– Cauchy’s (steepest decent) method
– Newton’s Method
– Marquardt’s Method
– Conjugate gradient method
– Variable-metric method
• By definition , the first derivative ∇f ( x (t ) ) at any
point ( x (t ) ) represents the direction of the
maximum increase of the function value.
• If we are interested in finding a point with the
minimum function value, ideally we should be
searching along the opposite to the first
derivative direction, that is , we should search
along − ∇f ( x (t ) )
• Any search made in this direction will have
smaller function value.
 • DECENT DIRECTION
A search direction (d (t ) ) is a decent direction at a
point ( x (t ) ) if the condition ∇f ( x ( t ) ).d ( t ) ≤ 0 is
(t )
satisfied in the vicinity of the point ( x )

It can also be proved by comparing function values

at two points along any decent direction.

The magnitude of the vector ∇f ( x (t ) ).d (t ) for a

decent direction (d (t ) ) specifies how decent the
search direction is.
The above statement can be explained with the help
of an example on the next side
• If
(t )
.d 2 = −∇f ( x ) is used,
(t )
the quantity ∇ f ( x (t )
).d (t )

is maximally negative

Thus the search direction − ∇f ( x (t ) ) is called the

steepest decent direction

Note : ∇ = ⎡⎢ ∂ ∂ ∂
−−−−−
∂ ⎤
⎥
⎣ ∂x1 ∂x 2 ∂x 3 ∂x n ⎦
 Example
• Consider the Himmelblau function:

f ( x1 , x2 ) = ( x12 + x2 − 11) 2 + ( x1 + x22 − 7 ) 2

We would like to determine whether the direction

(d ) = (1,0)
(t ) T at the point
( x (t )
) = (1,1) T
is a decent
direction or not.
Refer to the figure below
• It is clear from the figure that if we move locally
along (d (t ) ) from the point ( x (t ) ) will reduce the
function value
• Investigate this aspect by calculating the
derivative ∇f ( x (t ) ) at the point.
• Derivative as calculated numerically is
∇f ( x (t ) ) = (−46,−38) T
• Taking the dot product we obtain ∇f ( x (t ) ) and ( x (t ) )
⎛1⎞
∇f ( x ) d (t ) = (−46,−38) ⎜ ⎟ = −46
(t ) T T

⎝0⎠
• The above is a negative quantity, thus the search
direction is a Decent direction
• The amount of non-negativity suggests the extent
of decent in the direction
• If the search direction d (t ) = −∇f ( x (t ) ) T = (−46,−38) T
is used, the magnitude of the above dot product is
⎛ 46 ⎞
(−46,−38) ⎜ ⎟ = −3,560
T

⎝ 38 ⎠
• The direction (46,38) or (0.771,0.637) is more
descent than
• The above direction is the steepest direction at the
point x(t)
• In nonlinear function the steepest decent
direction at any point may not exactly pass
through the true minimum.

• The steepest decent direction is a direction which

is a local best direction

• It is not guaranteed that moving along the

steepest decent direction will always take the
search closer to the true minimum
CAUCHY’S ( STEEPEST DECENT) METHOD

• Search direction used is the negative of the gradient at

any particular point x(t) :

s k = −∇f ( x (k ) )

• As the direction gives the maximum decent in function

values, it is also known as Steepest Decent Methods

• The algorithm guarantees improvement in the function

value at every iteration
• Algorithm in brief ;
• At every iteration ;
– To find the minimum point along direction
– Compute derivative at current point
– Perform unidirectional search in the negative
to this derivative direction
• The minimum point becomes the current point
and the search continues from this point ;

• Algorithm continues until a point having a small

enough gradient vector is found.
Algorithm in Steps
STEP 1: CHOOSE M (MAX. NO. OF ITERATIONS)
∈ 1 ,∈ 2 , x 0
,k = 0
STEP 2: CALCULATE

∇ f (x k
)

STEP 3: IF ∇ f (x k
) ≤∈ 1
TERMINATE

IF k ≥ M TERMINATE

STEP 4: UNIDIRECTIONAL SEARCH USING ∈ 2

 MIN. f (x ) = f (x −α∇f (x ))
k+1 k k

k+1
STEP 5: IF
x −x k

k
≤∈1
x
ELSE k =k +1 GOTO STEP 2

- METHOD WORKS WELL WHEN xk IS FAR FROM x* (OPTIMUM)

- IF POINT IS CLOSE TO x* THEN CHANGE IN GRADIENT VECTOR IS
VERY SMALL.
- OTHER METHODS USE VARIATION
-SECOND DERIVATIVES(NEWTON’S)
-COMBIMATION
-CONJUGATE GRADIENT METHOD
 Example
• Consider the Himmelblau function:

f ( x1 , x2 ) = ( x12 + x2 − 11) 2 + ( x1 + x22 − 7 ) 2

Step 1: Choose large value of M for proper convergence

M ≈100 is normally chosen.
Initial conditions :
M = 100
x(0) = (0,0)T
ε1= ε2=10-3
k=0
• Step 2 : The derivative at the initial point x0 is
calculated and is found to be (- 14, - 22)T

• Step 3 : As the magnitude of the derivative is not

small and k = 0 < M = 100. do not terminate
proceed to step 4
• Step 4 : perform a line search form x(0) in the
direction − ∇f ( x ( 0) ) such that the function value
is minimum.
Along that direction, any point can be expressed
by fixing a value for the parameter α 0 in the
equation.
x = x − α ∇f ( x
0 0 (0)
(
) = 14α ,22α
0
)
0 T

Using Golden section search in the interval (0,1)

α 0*= 0.127 minimum point along the direction is
x1 = (1.788,2.810)T
• Step 5 : As x1 and x0 are quite different, we do not
terminate, but go back to step 2, this completes
one iteration.
• Step 2: The derivative vector at this point is
• ( - 30.707, 18.803 )T
• Step 3 : The magnitude of the derivative vector is
not smaller than ε1, thus, we continue with step 4.
• Step 4 : Another unidirectional search along
• ( 30.707,-18.803) from the point
x1=(1.788,2.810) using the golden section search
finds the new point x2=(23.008,1.99) with a
function value equal to 0.018

• Continue the process until the termination

criteria is reached
 Penalty function approach
• Transformation method
- convert to a sequence of unconstrained
problems.
• Give penalty to (violated) constraints.
• Add to objective function.
• Solve.
• Use result as starting point for next iteration.
• Alter penalties ands repeat.
• Minimise ( x12 + x2 − 11) 2 + ( x1 + x22 − 7) 2
Subjected to ( x1 − 5) 2 + x22 − 26 ≥ 0
Penalty = 0.1 < ( x1 − 5) 2 + x22 − 26 ≥ 0 > 2

Feasible
region Infeasible region
Minimum
point

x2

x1
 Process
1. Choose ε1 , ε 2 , R, Ω.
2. Form modified objective function
P ( x k , R k ) = f ( x k ) + Ω( R k , g ( x k ), h( x k ))

3. Start with xk . Find x k +1 so as to minimize P.

(use ε1 )
4. If P( x k +1 , R k ) − P( x k , R k −1 ) < ε 2
Terminate.
5. Else R k +1
= cR, k=k+1; go to step 2.
• At any stage minimize
P(x,R) = f(x)+Ω(R,g(x),h(x))
R = set of penalty parameters
Ω= penalty function
 Types of penalty function
• Parabolic penalty
Ω = R{h(x)}2
- for equality constraints
- only for infeasible points
• Interior penalty functions
- penalize feasible points
• Exterior penalty functions
- penalize infeasible points
• Mixed penalty functions
- combination of both
• Infinite barrier penalty
Ω = R ∑ g j (x)
- inequality constraints.
- R is very large.
- Exterior.
• Log penalty
Ω=-R ln[g(x)]
-inequality constraints
- for feasible points
- interior.
-initially large R.
- larger penalty close to border.
• Inverse penalty
⎡ 1 ⎤
Ω = R⎢ ⎥
⎣ g ( x) ⎦
- interior.
- larger penalty close to border.
- initially large R
• Bracket order penalty
R < g ( x) > 2

- <A>=A if A<0.
- exterior.
- initially small R.
 Direct search
• Variable elimination
- for equality constraints
- Express one variable as a function of others
and
- Eliminate one variable
- Remove all equality constraints
 Complex search
- generate a set of points at random
- If a point is infeasible reflect beyond
centroid of remaining points
- Take worst point and push towards centroid
of feasible points.
 Complex Search Algo
• S1: Assume a bound in x (xL, xU), a
reflection parameter α, and ε & δ
• S2: Generate a set of P (= 2N) initial points
– For each point
• Sample N times to determine xi(P), in the given
bound
• if x(P) is infeasible, calculate x (centroid) of the
current set of points and set xP = xP+1/2(x -xP) until
xP is feasible.
• If xP is feasible, continue till u get P points
 Complex Search Algo (contd)
• S3: Reflection step
– select xR such that f(xR) = max (f(xP)) = Fmax
– calculate x, centroid of remaining points
– xm = x + α (x - xR)
– If xm is feasible and f(xm) > Fmax, retrtact half
the distance to x , and continue till f(xm) < Fmax
– If xm is feasible and f(xm) < Fmax, Go to S5
– If xm is infeasible, Goto S4
 Complex Search Algo (contd)
• S4: Check for feasibility of the solution
– For all i, reset violated variable bounds
• if xim < xiL, xim = xiL
• if xim > xiU, xim = xiU
– If the resulting xim is infeasible, retract half
the distance to the centroid, repeat till xm is
feasible
 Complex Search Algo (contd)
• S5: Replace xR by xm, check for
termination
– fmean = mean of f(xP),
– xmean = mean (xP)

∑p
( f ( x p
) − f mean ) 2
≤ε

∑x
2
p
− xmean ≤δ
p
Complex Search Algo (contd)
Characteristics of complex search
• For complex feasible region
• If the optimum is well inside the search space,
the algo is efficient
• Not so good if the search space is narrow, or
the optimum is close to the constraint
boundary