Optimization Problems

Content
Introduction
Definitions
Localand Global Optima
Convex Sets and Functions
Convex Programming Problems
Optimization Problems

Introduction


General Nonlinear Programming Problems

minimize f ( x) objective function

subject to gi ( x) 0 i 1, ,m
h j ( x) 0 j 1, ,p constraints

x Rn
Local Minima vs. Global Minima

minimize f ( x) objective function

subject to gi ( x) 0 i 1, ,m
h j ( x) 0 j 1, ,p constraints

x Rn

local minimum

global minimum
 Local optimality Global optimality

Convex Programming Problems

minimize f ( x) objective function

subject to gi ( x) 0 i 1, ,m
h j ( x) 0 j 1, ,p constraints

x Rn

f (x) convex

gi (x) concave
hj (x) linear
 Local optimality Global optimality

Linear Programming Problems

minimize f ( x) objective function

subject to gi ( x) 0 i 1, ,m
h j ( x) 0 j 1, ,p constraints

x Rn

f (x) linear
a special case of
gi (x) linear convex programming problems

hj (x) linear
 Local optimality Global optimality

Linear Programming Problems

minimize f ( x) objective function

subject to gi ( x) 0 i 1, ,m
h j ( x) 0 j 1, ,p constraints

x Rn

f (x) linear

gi (x) linear
hj (x) linear
Integer Programming Problems

minimize f ( x) objective function

subject to gi ( x) 0 i 1, ,m
h j ( x) 0 j 1, ,p constraints

xZn

f (x) linear

gi (x) linear
hj (x) linear
The Hierarchy of Optimization Problems

Nonlinear Convex Linear

Programs Programs Programs
(Polynomial) Integer
Flow Programs
and (NP-Hard)
Matching
Optimization Problems
General Nonlinear Programming Problems

Convex Programming Problems

Linear Programming Problems

Integer Linear Programming Problems

Optimization Techniques
General Nonlinear Programming Problems
Continuous
Optimization
Continuous
Convex Programming Problems
Variables
Linear Programming Problems
Combinatorial
Discrete Optimization
Integer Linear Programming Problems
Variables
Optimization Problems

Definitions

Optimization Problems

minimize f ( x)

subject to gi ( x) 0 i 1, ,m
h j ( x) 0 j 1, ,p
xR n
Optimization Problems

minimize
Minimize
f ( x) cost c: FR1

subject to gi ( x) 0 i 1, ,m
Define the set of feasible points
h j ( x) 0 j 1, ,p
F
xR n
Definition:
Instance of an Optimization Problem

(F, c) F: the domain of feasible points

c: F R1 cost function

Goal:
To find f F such that
c( f ) c(g) for all gF.
A global optimum
Definition:
Optimization Problem

A set of instances of an optimization

problem, e.g.
Traveling Salesman Problem (TSP)
Minimal Spanning Tree (MST)
Shortest Path (SP)
Linear Programming (LP)
Traveling Salesman Problem (TSP)
Traveling Salesman Problem (TSP)

Instance of the TSP

Given n cities and an n n distance matrix [dij],
the problem is to find a Hamiltonian cycle with
minimal total length.

F all cyclic permutations on n objects

n
c d j ( j ) 1 2 3 4 5 6 7 8
e.g.,
j 1
2 5 3 6 1 8 4 7
Minimal Spanning Tree (MST)
Minimal Spanning Tree (MST)

Instance of the MST

Given an integer n > 0 and an n n symmetric distance
matrix [dij], the problem is to find a spanning tree on n
vertices that has minimum total length of its edge.

F all spanning trees (V , E) with V {1, 2, , n}

c : (V , E )
( i , j )E
dij
Linear Programming (LP)
minimize c1 x1 c2 x2 cn xn

Subject to a11 x1 a12 x2 a1n xn b1

a21 x1 a22 x2 a2 n xn b2

am1 x1 am 2 x2 amn xn bm

x1 , x2 , , xn 0
 minimize c1 x1 c2 x2 cn xn

Subject to a11 x1 a12 x2 a1n xn b1

a21 x1 a22 x2 a2 n xn b2

am1 x1 am 2 x2 amn xn bm

Linear Programming (LP) x1 , x2 , , xn 0

minimize cx

Subject to Ax b
x0

a11 a12 a1n b1 c1 x1

a21 a22 a2 n b2 c2 x2
A b c x


am1 am 2 amn bm cn xn
 minimize c1 x1 c2 x2 cn xn

Subject to a11 x1 a12 x2 a1n xn b1

a21 x1 a22 x2 a2 n xn b2

am1 x1 am 2 x2 amn xn bm

Linear Programming (LP) x1 , x2 , , xn 0

minimize cx

Subject to Ax b
x0

F x x R , Ax b, x 0
n

c : x cx
 minimize c1 x1 c2 x2 cn xn

Subject to a11 x1 a12 x2 a1n xn b1

a21 x1 a22 x2 a2 n xn b2

Example: am1 x1 am 2 x2 amn xn bm

x1 , x2 , , xn 0

Linear Programming (LP)

minimize 4 x1 2 x2 3x3

Subject to x1 x2 x3 2
x1 , x2 , x3 0

c 4 2 3

A 1 1 1 b 2
 minimize c1 x1 c2 x2 cn xn

Subject to a11 x1 a12 x2 a1n xn b1

a21 x1 a22 x2 a2 n xn b2

Example: am1 x1 am 2 x2 amn xn bm

x1 , x2 , , xn 0

Linear Programming (LP)

minimize 4 x1 2 x2 3x3

Subject to x1 x2 x3 2
x1 , x2 , x3 0 x3
v
c(v3) = 6 3
The optimal point is at
c(v2) = 4
one of the vertices.
x2
c(v1) = 8
x1 v2
v1 The optimum
 Example:
Minimal Spanning Tree (3 Nodes)

minimize 4 x1 2 x2 3x3

Subject to x1 x2 x3 2 Integer Programming

x1 , x2 , x3 {0,1} x3

x3{0, 1} x1{0, 1}
c2=2 c1=4

x2
c3=3
x2{0, 1} x1
 Some integer programs can be
transformed into linear programs.
Example:
Minimal Spanning Tree (3 Nodes)

minimize 4 x1 2 x2 3x3

Subject to x1 x2 x3 2 Linear Programming

x1 , x2 , x3 1
x3
x1 , x2 , x3 0
x3{0, 1} x1{0, 1}
c2=2 c1=4

x2
c3=3
x2{0, 1} x1
Optimization Problems

Local and Global

Optima

 For combinatorial optimization,
the choice of N is critical.

Neighborhoods
Given an optimization problem with instance

(F, c),
a neighborhood is a mapping

N :F 2 F

defined for each instance.

TSP (2-Change)

f F gN2(f )

N 2 ( f ) g g F and g can be obtained as above

TSP (k-Change)

g F and g can be obtained

Nk ( f ) g
by changing k edges of f .
MST

1. Adding an edge to form a cycle.

f F 2. Deleting any edge on the cycle.
gN(f )

N ( f ) g g F and g can be obtained as above

LP
minimize cx

Subject to Ax b
x0

N ( x) y Ay b, y 0, and y x
Local Optima
(F, c) an instance of an optimization problem

Given
N neighborhood

f F is called locally optimum with respect to N (or simply

locally optimum whenever N is understood by context) if

c(f ) c(g) for all gN(f ).

Local Optima
F = [0, 1] R1

N ( f ) x x F , y 0, and x f
c

small

C Global
Local minimum
minimum A Local
minimum
B

0 1 F
 Improve( f ) s s N ( f ) and c( s) c( f )

Decent Algorithm
f = initial feasible solution
While Improve(f ) do
f = any element in Improve(f )
return f
Decent algorithm is usually stuck at a local
minimum unless the neighborhood N is exact.
Exactness of Neighborhood

Neighborhood N is said to be exact if it makes

Local minimum Global Minimum

 N is exact if 1.
Exactness of Neighborhood
F = [0, 1] R1

N ( f ) x x F , y 0, and x f
c

C Global
Local minimum
minimum A Local
minimum
B

0 1 F
TSP f F gN2(f )

N2: not exact

Nn: exact
 N is exact
MST

1. Adding an edge to form a cycle.

f F 2. Deleting any edge on the cycle.
gN(f )

N ( f ) g g F and g can be obtained as above

Optimization Problems

Convex Sets and

Functions

Convex Combination
x, y Rn
01 z = x +(1)y
A convex combination of x, y.

A strict convex combination of x, y

if 0, 1.
 z = x +(1)y
Convex Sets 01

S Rn is convex if it contains all convex

combinations of pairs x, y S.

convex nonconvex
 z = x +(1)y
Convex Sets 01

S Rn is convex if it contains all convex

combinations of pairs x, y S.

n=1
S is convex iff S is an interval.
Convex Sets

Fact: The intersection of any number of convex sets is convex.

 Every linear function is convex.

Convex Functions
S Rn a convex set

c:S R a convex function if

c(x +(1)y) c(x) + (1)c(y), 0 1
c

c(x) + (1)c(y) c(y)

c(x) c(x +(1)y)

x x +(1)y y
 Lemma
S a convex set
St x c ( x ) t , x S
c(x) a convex function on S
t a real number is convex.

Pf) Let x, y St x +(1)y S

c(x +(1)y) c(x) + (1)c(y)
t + (1)t
=t
x +(1)y St
Level Contours
c=5
c=4
c=3
c=2
c=1
 Every linear function is
concave as well as convex.

Concave Functions
S Rn a convex set

c:S R a concave function if

c is a convex
Optimization Problems

Convex Programming
Problems

Theorem

(F, c) an instance of optimization problem

a convex function
a convex set

Define N ( x) y y F and x y
N ( x) is exact for every > 0.
 (F, c) an instance of optimization problem

a convex function
a convex set
Theorem
Define N ( x) y y F and x y
N ( x) is exact for every > 0.
Pf)
Let x be a local minimum w.r.t. N for any fixed > 0.
Let yF be any other feasible point.

F x

N ( x )
 (F, c) an instance of optimization problem

a convex function
a convex set
Theorem
Define N ( x) y y F and x y
N ( x) is exact for every > 0.
Pf)
Let x be a local minimum w.r.t. N for any fixed > 0.
Let yF be any other feasible point.
<<1 such that z x (1 ) y and z N ( x). c( z ) c( x)
Since c is convex, we have
c( z ) c( x (1 ) y ) y
c( x) (1 )c( y ) z
c( z ) c( x) F x
Therefore, c( y )
1
c( x) c( x)
c( x)
1 N ( x )
 Important property:
Local minimum Global Minimum

Convex Programming Problems

(F, c) an instance of optimization problem

Convex function

Defined by gi ( x) 0, i 1, ,m

gi : R R
n
Concave functions
 Important property:
Local minimum Global Minimum

Convexity of Feasible Set

gi ( x) : concave
gi ( x) : convex
(F, c) an instance of optimization
gi ( x) problem
0 : convex
gi ( x) 0 : convex
Fi : convex
m
Convex function F Fi : convex
i 1

Defined by gi ( x) 0, i 1, ,m

gi : R R
n
Concave functions
 Important property:
Local minimum Global Minimum

Convex Programming Problems

(F, c) an instance of optimization problem

Convex function

Convex
Defined by gi ( x) 0, i 1, ,m

g :R R
i
n
Concave functions
Theorem

In a convex programming problem, every point

locally optimal with respect to the Euclidean

distance neighborhood N is also global optimal.