Convex Optimization

Stephen Boyd Lieven Vandenberghe

Revised slides by Stephen Boyd, Lieven Vandenberghe, and Parth Nobel

1. Introduction
 Outline

Mathematical optimization

Convex optimization

Convex Optimization Boyd and Vandenberghe 1.1

 Optimization problem

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
gi (x) = 0, i = 1, . . . , p

I x ∈ Rn is (vector) variable to be chosen (n scalar variables x1 , . . . , xn )

I f0 is the objective function, to be minimized
I f1 , . . . , fm are the inequality constraint functions
I g1 , . . . , gp are the equality constraint functions

I variations: maximize objective, multiple objectives, …

Convex Optimization Boyd and Vandenberghe 1.2

 Finding good (or best) actions

I x represents some action, e.g.,

– trades in a portfolio
– airplane control surface deflections
– schedule or assignment
– resource allocation
I constraints limit actions or impose conditions on outcome
I the smaller the objective f0 (x), the better
– total cost (or negative profit)
– deviation from desired or target outcome
– risk
– fuel use

Convex Optimization Boyd and Vandenberghe 1.3

 Finding good models

I x represents the parameters in a model

I constraints impose requirements on model parameters (e.g., nonnegativity)
I objective f0 (x) is sum of two terms:
– a prediction error (or loss) on some observed data
– a (regularization) term that penalizes model complexity

Convex Optimization Boyd and Vandenberghe 1.4

 Worst-case analysis (pessimization)

I variables are actions or parameters out of our control

(and possibly under the control of an adversary)
I constraints limit the possible values of the parameters
I minimizing −f0 (x) finds worst possible parameter values

I if the worst possible value of f0 (x) is tolerable, you’re OK

I it’s good to know what the worst possible scenario can be

Convex Optimization Boyd and Vandenberghe 1.5

 Optimization-based models

I model an entity as taking actions that solve an optimization problem

– an individual makes choices that maximize expected utility
– an organism acts to maximize its reproductive success
– reaction rates in a cell maximize growth
– currents in a circuit minimize total power

I (except the last) these are very crude models

I and yet, they often work very well

Convex Optimization Boyd and Vandenberghe 1.6

 Basic use model for mathematical optimization

I instead of saying how to choose (action, model) x

I you articulate what you want (by stating the problem)
I then let an algorithm decide on (action, model) x

Convex Optimization Boyd and Vandenberghe 1.7

 Can you solve it?

I generally, no
I but you can try to solve it approximately, and it often doesn’t matter

I the exception: convex optimization

– includes linear programming (LP), quadratic programming (QP), many others
– we can solve these problems reliably and efficiently
– come up in many applications across many fields

Convex Optimization Boyd and Vandenberghe 1.8

 Nonlinear optimization

traditional techniques for general nonconvex problems involve compromises

local optimization methods (nonlinear programming)

I find a point that minimizes f0 among feasible points near it
I can handle large problems, e.g., neural network training
I require initial guess, and often, algorithm parameter tuning
I provide no information about how suboptimal the point found is

global optimization methods

I find the (global) solution
I worst-case complexity grows exponentially with problem size
I often based on solving convex subproblems

Convex Optimization Boyd and Vandenberghe 1.9

 Outline

Mathematical optimization

Convex optimization

Convex Optimization Boyd and Vandenberghe 1.10

 Convex optimization

convex optimization problem:

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
Ax = b

I variable x ∈ Rn
I equality constraints are linear
I f0 , . . . , fm are convex: for 𝜃 ∈ [0, 1],

fi (𝜃x + (1 − 𝜃)y) ≤ 𝜃fi (x) + (1 − 𝜃)fi (y)

i.e., fi have nonnegative (upward) curvature

Convex Optimization Boyd and Vandenberghe 1.11

 When is an optimization problem hard to solve?

I classical view:
– linear (zero curvature) is easy
– nonlinear (nonzero curvature) is hard

I the classical view is wrong

I the correct view:

– convex (nonnegative curvature) is easy
– nonconvex (negative curvature) is hard

Convex Optimization Boyd and Vandenberghe 1.12

 Solving convex optimization problems

I many different algorithms (that run on many platforms)

– interior-point methods for up to 10000s of variables
– first-order methods for larger problems
– do not require initial point, babysitting, or tuning
I can develop and deploy quickly using modeling languages such as CVXPY
I solvers are reliable, so can be embedded
I code generation yields real-time solvers that execute in milliseconds
(e.g., on Falcon 9 and Heavy for landing)

Convex Optimization Boyd and Vandenberghe 1.13

 Modeling languages for convex optimization

I domain specific languages (DSLs) for convex optimization

– describe problem in high level language, close to the math
– can automatically transform problem to standard form, then solve

I enables rapid prototyping

I it’s now much easier to develop an optimization-based application
I ideal for teaching and research (can do a lot with short scripts)

I gets close to the basic idea: say what you want, not how to get it

Convex Optimization Boyd and Vandenberghe 1.14

 CVXPY example: non-negative least squares

math: CVXPY code:

minimize kAx − bk 22 import cvxpy as cp
subject to x0
A, b = ...

I variable is x x = cp.Variable(n)
obj = cp.norm2(A @ x - b)**2
I A, b given
constr = [x >= 0]
I x  0 means x1 ≥ 0, . . . , xn ≥ 0 prob = cp.Problem(cp.Minimize(obj), constr)
prob.solve()

Convex Optimization Boyd and Vandenberghe 1.15

 Brief history of convex optimization

I theory (convex analysis): 1900–1970

I algorithms
– 1947: simplex algorithm for linear programming (Dantzig)
– 1960s: early interior-point methods (Fiacco & McCormick, Dikin, …)
– 1970s: ellipsoid method and other subgradient methods
– 1980s & 90s: interior-point methods (Karmarkar, Nesterov & Nemirovski)
– since 2000s: many methods for large-scale convex optimization

I applications
– before 1990: mostly in operations research, a few in engineering
– since 1990: many applications in engineering (control, signal processing, communications,
circuit design, …)
– since 2000s: machine learning and statistics, finance

Convex Optimization Boyd and Vandenberghe 1.16

 Summary

convex optimization problems

I are optimization problems of a special form
I arise in many applications
I can be solved effectively
I are easy to specify using DSLs

Convex Optimization Boyd and Vandenberghe 1.17

2. Convex sets
 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.1

 Affine set

line through x1 , x2 : all points of form x = 𝜃x1 + (1 − 𝜃)x2 , with 𝜃 ∈ R

𝜃 = 1.2 x1
𝜃=1
𝜃 = 0.6
x2
𝜃=0
𝜃 = −0.2

affine set: contains the line through any two distinct points in the set

example: solution set of linear equations {x | Ax = b}

(conversely, every affine set can be expressed as solution set of system of linear equations)

Convex Optimization Boyd and Vandenberghe 2.2

 Convex set

line segment between x1 and x2 : all points of form x = 𝜃x1 + (1 − 𝜃)x2 , with 0 ≤ 𝜃 ≤ 1

convex set: contains line segment between any two points in the set

x1 , x2 ∈ C, 0≤𝜃≤1 =⇒ 𝜃x1 + (1 − 𝜃)x2 ∈ C

examples (one convex, two nonconvex sets)

Convex Optimization Boyd and Vandenberghe 2.3

 Convex combination and convex hull

convex combination of x1 ,…, xk : any point x of the form

x = 𝜃 1 x1 + 𝜃 2 x2 + · · · + 𝜃 k xk

with 𝜃 1 + · · · + 𝜃 k = 1, 𝜃 i ≥ 0

convex hull conv S: set of all convex combinations of points in S

Convex Optimization Boyd and Vandenberghe 2.4

 Convex cone
conic (nonnegative) combination of x1 and x2 : any point of the form

x = 𝜃 1 x1 + 𝜃 2 x2

with 𝜃 1 ≥ 0, 𝜃 2 ≥ 0

x1

x2
0

convex cone: set that contains all conic combinations of points in the set

Convex Optimization Boyd and Vandenberghe 2.5

 Hyperplanes and halfspaces
a

hyperplane: set of the form {x | aT x = b}, with a ≠ 0 x0

x
aT x = b

x0 aT x ≥ b
halfspace: set of the form {x | aT x ≤ b}, with a ≠ 0
aT x ≤ b

I a is the normal vector

I hyperplanes are affine and convex; halfspaces are convex

Convex Optimization Boyd and Vandenberghe 2.6

 Euclidean balls and ellipsoids

(Euclidean) ball with center xc and radius r:

B(xc , r) = {x | kx − xc k 2 ≤ r} = {xc + ru | kuk 2 ≤ 1}

ellipsoid: set of the form

{x | (x − xc ) T P −1 (x − xc ) ≤ 1}
with P ∈ Sn++ (i.e., P symmetric positive definite)

xc

another representation: {xc + Au | kuk 2 ≤ 1} with A square and nonsingular

Convex Optimization Boyd and Vandenberghe 2.7

 Norm balls and norm cones
I norm: a function k · k that satisfies
– kxk ≥ 0; kxk = 0 if and only if x = 0
– ktxk = |t| kxk for t ∈ R
– kx + yk ≤ kxk + kyk
I notation: k · k is general (unspecified) norm; k · k symb is particular norm
I norm ball with center xc and radius r: {x | kx − xc k ≤ r}
I norm cone: {(x, t) | kxk ≤ t}
I norm balls and cones are convex

Euclidean norm cone

0.5

t
n+1
{(x, t) | kxk 2 ≤ t} ⊂ R
0
1
is called second-order cone 0
0
1

x2 −1 −1
x1
Convex Optimization Boyd and Vandenberghe 2.8
 Polyhedra

I polyhedron is solution set of finitely many linear inequalities and equalities

{x | Ax  b, Cx = d}

(A ∈ Rm×n , C ∈ Rp×n ,  is componentwise inequality)

I intersection of finite number of halfspaces and hyperplanes
I example with no equality constraints; aTi are rows of A
a1 a2

P
a5
a3

a4

Convex Optimization Boyd and Vandenberghe 2.9

 Positive semidefinite cone
notation:
I Sn is set of symmetric n × n matrices
I Sn+ = {X ∈ Sn | X  0}: positive semidefinite (symmetric) n × n matrices

X ∈ Sn+ ⇐⇒ zT Xz ≥ 0 for all z

I Sn+ is a convex cone, the positive semidefinite cone

I Sn++ = {X ∈ Sn | X  0}: positive definite (symmetric) n × n matrices

x y
 
0.5
example: ∈ S2+

z
y z
0
1
1
0 0.5
y −1 0 x
Convex Optimization Boyd and Vandenberghe 2.10
 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.11

 Showing a set is convex

methods for establishing convexity of a set C

1. apply definition: show x1 , x2 ∈ C, 0 ≤ 𝜃 ≤ 1 =⇒ 𝜃x1 + (1 − 𝜃)x2 ∈ C

– recommended only for very simple sets

2. use convex functions (next lecture)

3. show that C is obtained from simple convex sets (hyperplanes, halfspaces, norm balls, …)
by operations that preserve convexity
– intersection
– affine mapping
– perspective mapping
– linear-fractional mapping

you’ll mostly use methods 2 and 3

Convex Optimization Boyd and Vandenberghe 2.12

 Intersection
I the intersection of (any number of) convex sets is convex
I example:
– S = {x ∈ Rm | |p(t)| ≤ 1 for |t| ≤ 𝜋/3}, with p(t) = x1 cos t + · · · + xm cos mt
– write S = |t | ≤ 𝜋/3 {x | |p(t)| ≤ 1}, i.e., an intersection of (convex) slabs
Ñ

I picture for m = 2:
2

1
1
0
p(t)

x2
0
−1

−1

−2
0 2 𝜋/3
𝜋/3
t
𝜋 −2 −1
x01 1 2

Convex Optimization Boyd and Vandenberghe 2.13

 Affine mappings

I suppose f : Rn → Rm is affine, i.e., f (x) = Ax + b with A ∈ Rm×n , b ∈ Rm

I the image of a convex set under f is convex

S ⊆ Rn convex =⇒ f (S) = {f (x) | x ∈ S} convex

I the inverse image f −1 (C) of a convex set under f is convex

C ⊆ Rm convex =⇒ f −1 (C) = {x ∈ Rn | f (x) ∈ C} convex

Convex Optimization Boyd and Vandenberghe 2.14

 Examples

I scaling, translation: aS + b = {ax + b | x ∈ S}, a, b ∈ R

I projection onto some coordinates: {x | (x, y) ∈ S}
I if S ⊆ Rn is convex and c ∈ Rn , cT S = {cT x | x ∈ S} is an interval
I solution set of linear matrix inequality {x | x1 A1 + · · · + xm Am  B} with Ai , B ∈ Sp
I hyperbolic cone {x | xT Px ≤ (cT x) 2 , cT x ≥ 0} with P ∈ Sn+

Convex Optimization Boyd and Vandenberghe 2.15

 Perspective and linear-fractional function

I perspective function P : Rn+1 → Rn :

P(x, t) = x/t, dom P = {(x, t) | t > 0}

I images and inverse images of convex sets under perspective are convex

I linear-fractional function f : Rn → Rm :

Ax + b
f (x) = , dom f = {x | cT x + d > 0}
cT x + d
I images and inverse images of convex sets under linear-fractional functions are convex

Convex Optimization Boyd and Vandenberghe 2.16

 Linear-fractional function example

1
f (x) = x
x1 + x2 + 1

1 1
x2

x2
0 C 0 f (C)

−1 −1
−1 0 1 −1 0 1
x1 x1

Convex Optimization Boyd and Vandenberghe 2.17

 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.18

 Proper cones

a convex cone K ⊆ Rn is a proper cone if

I K is closed (contains its boundary)
I K is solid (has nonempty interior)
I K is pointed (contains no line)

examples
I nonnegative orthant K = Rn+ = {x ∈ Rn | xi ≥ 0, i = 1, . . . , n}
I positive semidefinite cone K = Sn+
I nonnegative polynomials on [0, 1]:

K = {x ∈ Rn | x1 + x2 t + x3 t 2 + · · · + xn t n−1 ≥ 0 for t ∈ [0, 1]}

Convex Optimization Boyd and Vandenberghe 2.19

 Generalized inequality

I (nonstrict and strict) generalized inequality defined by a proper cone K:

x K y ⇐⇒ y − x ∈ K, x ≺K y ⇐⇒ y − x ∈ int K
I examples
– componentwise inequality (K = Rn+ ): x Rn y ⇐⇒ xi ≤ yi , i = 1, . . . , n
+
– matrix inequality (K = Sn+ ): X Sn Y ⇐⇒ Y − X positive semidefinite
+
these two types are so common that we drop the subscript in K

I many properties of K are similar to ≤ on R, e.g.,

x K y, u K v =⇒ x + u K y + v

Convex Optimization Boyd and Vandenberghe 2.20

 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.21

 Separating hyperplane theorem

I if C and D are nonempty disjoint (i.e., C ∩ D = ∅) convex sets, there exist a ≠ 0, b s.t.

aT x ≤ b for x ∈ C, aT x ≥ b for x ∈ D

aT x ≥ b aT x ≤ b

D
C

I the hyperplane {x | aT x = b} separates C and D

I strict separation requires additional assumptions (e.g., C is closed, D is a singleton)

Convex Optimization Boyd and Vandenberghe 2.22

 Supporting hyperplane theorem

I suppose x0 is a boundary point of set C ⊂ Rn

I supporting hyperplane to C at x0 has form {x | aT x = aT x0 }, where a ≠ 0 and
aT x ≤ aT x0 for all x ∈ C

a
x0
C

I supporting hyperplane theorem: if C is convex, then there exists a supporting

hyperplane at every boundary point of C

Convex Optimization Boyd and Vandenberghe 2.23

3. Convex functions
 Outline

Convex functions

Operations that preserve convexity

Constructive convex analysis

Perspective and conjugate

Quasiconvexity

Convex Optimization Boyd and Vandenberghe 3.1

 Definition

I f : Rn → R is convex if dom f is a convex set and for all x, y ∈ dom f , 0 ≤ 𝜃 ≤ 1,

f (𝜃x + (1 − 𝜃)y) ≤ 𝜃f (x) + (1 − 𝜃)f (y)

(y, f (y))
(x, f (x))

I f is concave if −f is convex
I f is strictly convex if dom f is convex and for x, y ∈ dom f , x ≠ y, 0 < 𝜃 < 1,

f (𝜃x + (1 − 𝜃)y) < 𝜃f (x) + (1 − 𝜃)f (y)

Convex Optimization Boyd and Vandenberghe 3.2

 Examples on R
convex functions:
I affine: ax + b on R, for any a, b ∈ R
I exponential: eax , for any a ∈ R
I powers: x 𝛼 on R++ , for 𝛼 ≥ 1 or 𝛼 ≤ 0
I powers of absolute value: |x| p on R, for p ≥ 1
I positive part (relu): max{0, x}

concave functions:
I affine: ax + b on R, for any a, b ∈ R
I powers: x 𝛼 on R++ , for 0 ≤ 𝛼 ≤ 1
I logarithm: log x on R++
I entropy: −x log x on R++
I negative part: min{0, x}

Convex Optimization Boyd and Vandenberghe 3.3

 Examples on Rn

convex functions:
I affine functions: f (x) = aT x + b
I any norm, e.g., the ℓp norms
– kxk p = (|x1 | p + · · · + |xn | p ) 1/p for p ≥ 1
– kxk ∞ = max{|x1 |, . . . , |xn |}
I sum of squares: kxk 22 = x12 + · · · + xn2
I max function: max(x) = max{x1 , x2 , . . . , xn }
I softmax or log-sum-exp function: log(exp x1 + · · · + exp xn )

Convex Optimization Boyd and Vandenberghe 3.4

 Examples on Rm×n

I X ∈ Rm×n (m × n matrices) is the variable

I general affine function has form
m Õ
n
f (X) = tr(AT X) + b =
Õ
Aij Xij + b
i=1 j=1

for some A ∈ Rm×n , b ∈ R

I spectral norm (maximum singular value) is convex

f (X) = kX k 2 = 𝜎max (X) = (𝜆max (X T X)) 1/2

I log-determinant: for X ∈ Sn++ , f (X) = log det X is concave

Convex Optimization Boyd and Vandenberghe 3.5

 Extended-value extension

I suppose f is convex on Rn , with domain dom f

I its extended-value extension f̃ is function f̃ : Rn → R ∪ {∞}

f (x) x ∈ dom f

f̃ (x) =
∞ x ∉ dom f

I often simplifies notation; for example, the condition

0≤𝜃≤1 =⇒ f̃ (𝜃x + (1 − 𝜃)y) ≤ 𝜃 f̃ (x) + (1 − 𝜃) f̃ (y)

(as an inequality in R ∪ {∞}), means the same as the two conditions

– dom f is convex
– x, y ∈ dom f , 0 ≤ 𝜃 ≤ 1 =⇒ f (𝜃x + (1 − 𝜃)y) ≤ 𝜃f (x) + (1 − 𝜃)f (y)

Convex Optimization Boyd and Vandenberghe 3.6

 Restriction of a convex function to a line

I f : Rn → R is convex if and only if the function g : R → R,

g(t) = f (x + tv), dom g = {t | x + tv ∈ dom f }

is convex (in t) for any x ∈ dom f , v ∈ Rn

I can check convexity of f by checking convexity of functions of one variable

Convex Optimization Boyd and Vandenberghe 3.7

 Example

I f : Sn → R with f (X) = log det X, dom f = Sn++

I consider line in Sn given by X + tV, X ∈ Sn++ , V ∈ Sn , t ∈ R

g(t) = log det(X + tV)

log det X 1/2 I + tX −1/2 VX −1/2 X 1/2
   
=
 
= log det X + log det I + tX −1/2 VX −1/2
n
Õ
= log det X + log(1 + t𝜆i )
i=1

where 𝜆i are the eigenvalues of X −1/2 VX −1/2

I g is concave in t (for any choice of X ∈ Sn++ , V ∈ Sn ); hence f is concave

Convex Optimization Boyd and Vandenberghe 3.8

 First-order condition
I f is differentiable if dom f is open and the gradient
 
𝜕f (x) 𝜕f (x) 𝜕f (x)
∇f (x) = , ,..., ∈ Rn
𝜕x1 𝜕x2 𝜕xn
exists at each x ∈ dom f
I 1st-order condition: differentiable f with convex domain is convex if and only if

f (y) ≥ f (x) + ∇f (x) T (y − x) for all x, y ∈ dom f

I first order Taylor approximation of convex f is a global underestimator of f

f (y)
f (x) + ∇f (x) T (y − x)

(x, f (x))

Convex Optimization Boyd and Vandenberghe 3.9

 Second-order conditions

I f is twice differentiable if dom f is open and the Hessian ∇2 f (x) ∈ Sn ,

𝜕 2 f (x)
∇2 f (x)ij = , i, j = 1, . . . , n,
𝜕xi 𝜕xj

exists at each x ∈ dom f

I 2nd-order conditions: for twice differentiable f with convex domain

– f is convex if and only if ∇2 f (x)  0 for all x ∈ dom f
– if ∇2 f (x)  0 for all x ∈ dom f , then f is strictly convex

Convex Optimization Boyd and Vandenberghe 3.10

 Examples
I quadratic function: f (x) = (1/2)xT Px + qT x + r (with P ∈ Sn )
∇f (x) = Px + q, ∇2 f (x) = P
convex if P  0 (concave if P  0)
I least-squares objective: f (x) = kAx − bk 22
∇f (x) = 2AT (Ax − b), ∇2 f (x) = 2AT A
convex (for any A)

2
I quadratic-over-linear: f (x, y) = x2 /y, y > 0

f (x, y)
1
T
2 y y
 
2
∇ f (x, y) = 3 0
y −x −x 0
2 2

convex for y > 0 1 0

y 0 −2 x
Convex Optimization Boyd and Vandenberghe 3.11
 More examples

Ín
I log-sum-exp: f (x) = log k=1 exp xk is convex

1 1
∇2 f (x) = diag(z) − T 2 zzT (zk = exp xk )
1 z
T (1 z)

I to show ∇2 f (x)  0, we must verify that vT ∇2 f (x)v ≥ 0 for all v:

( k zk vk2 )( k zk ) − ( k vk zk ) 2
Í Í Í
T 2
v ∇ f (x)v = ≥0
( k zk ) 2
Í

since ( k vk zk ) 2 ≤ ( k zk vk2 )( k zk ) (from Cauchy-Schwarz inequality)

Í Í Í

În
I geometric mean: f (x) = ( k=1 xk )
1/n on Rn++ is concave (similar proof as above)

Convex Optimization Boyd and Vandenberghe 3.12

 Epigraph and sublevel set

I 𝛼-sublevel set of f : Rn → R is C 𝛼 = {x ∈ dom f | f (x) ≤ 𝛼}

I sublevel sets of convex functions are convex sets (but converse is false)
I epigraph of f : Rn → R is epi f = {(x, t) ∈ Rn+1 | x ∈ dom f , f (x) ≤ t}

epi f

I f is convex if and only if epi f is a convex set

Convex Optimization Boyd and Vandenberghe 3.13

 Jensen’s inequality

I basic inequality: if f is convex, then for x, y ∈ dom f , 0 ≤ 𝜃 ≤ 1,

f (𝜃x + (1 − 𝜃)y) ≤ 𝜃f (x) + (1 − 𝜃)f (y)

I extension: if f is convex and z is a random variable on dom f ,

f (E z) ≤ E f (z)

I basic inequality is special case with discrete distribution

prob(z = x) = 𝜃, prob(z = y) = 1 − 𝜃

Convex Optimization Boyd and Vandenberghe 3.14

 Example: log-normal random variable

I suppose X ∼ N (𝜇, 𝜎 2 )
I with f (u) = exp u, Y = f (X) is log-normal
I we have E f (X) = exp(𝜇 + 𝜎 2 /2)
I Jensen’s inequality is

f (E X) = exp 𝜇 ≤ E f (X) = exp(𝜇 + 𝜎 2 /2)

which indeed holds since exp 𝜎 2 /2 > 1

Convex Optimization Boyd and Vandenberghe 3.15

 Example: log-normal random variable

p(f (X))

E f (X)
f (E X)

p(X)
EX

Convex Optimization Boyd and Vandenberghe 3.16

 Outline

Convex functions

Operations that preserve convexity

Constructive convex analysis

Perspective and conjugate

Quasiconvexity

Convex Optimization Boyd and Vandenberghe 3.17

 Showing a function is convex

methods for establishing convexity of a function f

1. verify definition (often simplified by restricting to a line)

2. for twice differentiable functions, show ∇2 f (x)  0

– recommended only for very simple functions

3. show that f is obtained from simple convex functions by operations that preserve convexity
– nonnegative weighted sum
– composition with affine function
– pointwise maximum and supremum
– composition
– minimization
– perspective

you’ll mostly use methods 2 and 3

Convex Optimization Boyd and Vandenberghe 3.18

 Nonnegative scaling, sum, and integral

I nonnegative multiple: 𝛼f is convex if f is convex, 𝛼 ≥ 0

I sum: f1 + f2 convex if f1 , f2 convex

I infinite sum: if f1 , f2 , . . . are convex functions, infinite sum i=1 fi is convex
Í∞
∫
I integral: if f (x, 𝛼) is convex in x for each 𝛼 ∈ A, then f (x, 𝛼) d𝛼 is convex
𝛼∈ A

I there are analogous rules for concave functions

Convex Optimization Boyd and Vandenberghe 3.19

 Composition with affine function

(pre-)composition with affine function: f (Ax + b) is convex if f is convex

examples
I log barrier for linear inequalities
m
log(bi − aTi x), dom f = {x | aTi x < bi , i = 1, . . . , m}
Õ
f (x) = −
i=1

I norm approximation error: f (x) = kAx − bk (any norm)

Convex Optimization Boyd and Vandenberghe 3.20

 Pointwise maximum

if f1 , …, fm are convex, then f (x) = max{f1 (x), . . . , fm (x)} is convex

examples
I piecewise-linear function: f (x) = maxi=1,...,m (aTi x + bi )
I sum of r largest components of x ∈ Rn :

f (x) = x [1] + x [2] + · · · + x [r ]

(x [i] is ith largest component of x)

proof: f (x) = max{xi1 + xi2 + · · · + xir | 1 ≤ i1 < i2 < · · · < ir ≤ n}

Convex Optimization Boyd and Vandenberghe 3.21

 Pointwise supremum

if f (x, y) is convex in x for each y ∈ A, then g(x) = supy∈ A f (x, y) is convex

examples
I distance to farthest point in a set C: f (x) = supy∈C kx − yk
I maximum eigenvalue of symmetric matrix: for X ∈ Sn , 𝜆max (X) = sup kyk 2 =1 yT Xy is convex
I support function of a set C: SC (x) = supy∈C yT x is convex

Convex Optimization Boyd and Vandenberghe 3.22

 Partial minimization

I the function g(x) = inf y∈C f (x, y) is called the partial minimization of f (w.r.t. y)
I if f (x, y) is convex in (x, y) and C is a convex set, then partial minimization g is convex

examples
I f (x, y) = xT Ax + 2xT By + yT Cy with

A B
 
 0, C0
BT C

minimizing over y gives g(x) = inf y f (x, y) = xT (A − BC −1 BT )x

g is convex, hence Schur complement A − BC −1 BT  0
I distance to a set: dist(x, S) = inf y∈S kx − yk is convex if S is convex

Convex Optimization Boyd and Vandenberghe 3.23

 Composition with scalar functions

I composition of g : Rn → R and h : R → R is f (x) = h(g(x)) (written as f = h ◦ g)

I composition f is convex if
– g convex, h convex, h̃ nondecreasing
– or g concave, h convex, h̃ nonincreasing
(monotonicity must hold for extended-value extension h̃)
I proof (for n = 1, differentiable g, h)

f 00 (x) = h00 (g(x))g0 (x) 2 + h0 (g(x))g00 (x)

examples
I f (x) = exp g(x) is convex if g is convex
I f (x) = 1/g(x) is convex if g is concave and positive

Convex Optimization Boyd and Vandenberghe 3.24

 General composition rule

I composition of g : Rn → Rk and h : Rk → R is f (x) = h(g(x)) = h(g1 (x), g2 (x), . . . , gk (x))

I f is convex if h is convex and for each i one of the following holds
– gi convex, h̃ nondecreasing in its ith argument
– gi concave, h̃ nonincreasing in its ith argument
– gi affine

I you will use this composition rule constantly throughout this course
I you need to commit this rule to memory

Convex Optimization Boyd and Vandenberghe 3.25

 Examples

I log m i=1 exp gi (x) is convex if gi are convex

Í

I f (x) = p(x) 2 /q(x) is convex if

– p is nonnegative and convex
– q is positive and concave

I composition rule subsumes others, e.g.,

– 𝛼f is convex if f is, and 𝛼 ≥ 0
– sum of convex (concave) functions is convex (concave)
– max of convex functions is convex
– min of concave functions is concave

Convex Optimization Boyd and Vandenberghe 3.26

 Outline

Convex functions

Operations that preserve convexity

Constructive convex analysis

Perspective and conjugate

Quasiconvexity

Convex Optimization Boyd and Vandenberghe 3.27

 Constructive convexity verification

I start with function f given as expression

I build parse tree for expression
– leaves are variables or constants
– nodes are functions of child expressions
I use composition rule to tag subexpressions as convex, concave, affine, or none
I if root node is labeled convex (concave), then f is convex (concave)
I extension: tag sign of each expression, and use sign-dependent monotonicity

I this is sufficient to show f is convex (concave), but not necessary

I this method for checking convexity (concavity) is readily automated

Convex Optimization Boyd and Vandenberghe 3.28

 Example

the function
(x − y) 2
f (x, y) = , x < 1, y<1
1 − max(x, y)
is convex

constructive analysis:
I (leaves) x, y, and 1 are affine
I max(x, y) is convex; x − y is affine
I 1 − max(x, y) is concave
I function u2 /v is convex, monotone decreasing in v for v > 0
I f is composition of u2 /v with u = x − y, v = 1 − max(x, y), hence convex

Convex Optimization Boyd and Vandenberghe 3.29

 Example (from dcp.stanford.edu)

Convex Optimization Boyd and Vandenberghe 3.30

 Disciplined convex programming

in disciplined convex programming (DCP) users construct convex and concave functions as
expressions using constructive convex analysis

I expressions formed from

– variables,
– constants,
– and atomic functions from a library
I atomic functions have known convexity, monotonicity, and sign properties
I all subexpressions match general composition rule
I a valid DCP function is
– convex-by-construction
– ‘syntactically’ convex (can be checked ‘locally’)
I convexity depends only on attributes of atomic functions, not their meanings
√ √4
– e.g., could swap · and ·, or exp · and (·)+ , since their attributes match

Convex Optimization Boyd and Vandenberghe 3.31

 CVXPY example

(x − y) 2
, x < 1, y<1
1 − max(x, y)

import cvxpy as cp
x = cp.Variable()
y = cp.Variable()
expr = cp.quad_over_lin(x - y, 1 - cp.maximum(x, y))
expr.curvature # Convex
expr.sign # Positive
expr.is_dcp() # True

(atom quad_over_lin(u,v) includes domain constraint v>0)

Convex Optimization Boyd and Vandenberghe 3.32

 DCP is only sufficient

√
I consider convex function f (x) = 1 + x2
I expression f1 = cp.sqrt(1+cp.square(x)) is not DCP
I expression f2 = cp.norm2([1,x]) is DCP

I CVXPY will not recognize f1 as convex, even though it represents a convex function

Convex Optimization Boyd and Vandenberghe 3.33

 Outline

Convex functions

Operations that preserve convexity

Constructive convex analysis

Perspective and conjugate

Quasiconvexity

Convex Optimization Boyd and Vandenberghe 3.34

 Perspective

I the perspective of a function f : Rn → R is the function g : Rn × R → R,

g(x, t) = tf (x/t), dom g = {(x, t) | x/t ∈ dom f , t > 0}

I g is convex if f is convex

examples
I f (x) = xT x is convex; so g(x, t) = xT x/t is convex for t > 0
I f (x) = − log x is convex; so relative entropy g(x, t) = t log t − t log x is convex on R2++

Convex Optimization Boyd and Vandenberghe 3.35

 Conjugate function

I the conjugate of a function f is f ∗ (y) = supx∈dom f (yT x − f (x))

f (x)
xy

(0, −f ∗ (y))

I f ∗ is convex (even if f is not)

I will be useful in chapter 5

Convex Optimization Boyd and Vandenberghe 3.36

 Examples

I negative logarithm f (x) = − log x

y<0

−1 − log(−y)
f (y) = sup (xy + log x) =
∗
x>0 ∞ otherwise

I strictly convex quadratic, f (x) = (1/2)xT Qx with Q ∈ Sn++

1 T −1
f ∗ (y) = sup (yT x − (1/2)xT Qx) = y Q y
x 2

Convex Optimization Boyd and Vandenberghe 3.37

 Outline

Convex functions

Operations that preserve convexity

Constructive convex analysis

Perspective and conjugate

Quasiconvexity

Convex Optimization Boyd and Vandenberghe 3.38

 Quasiconvex functions
I f : Rn → R is quasiconvex if dom f is convex and the sublevel sets

S 𝛼 = {x ∈ dom f | f (x) ≤ 𝛼}

are convex for all 𝛼

a b c

I f is quasiconcave if −f is quasiconvex
I f is quasilinear if it is quasiconvex and quasiconcave

Convex Optimization Boyd and Vandenberghe 3.39

 Examples

|x| is quasiconvex on R
p
I
I ceil(x) = inf{z ∈ Z | z ≥ x} is quasilinear
I log x is quasilinear on R++

I f (x1 , x2 ) = x1 x2 is quasiconcave on R2++

I linear-fractional function

aT x + b
f (x) = , dom f = {x | cT x + d > 0}
cT x + d
is quasilinear

Convex Optimization Boyd and Vandenberghe 3.40

 Example: Internal rate of return

I cash flow x = (x0 , . . . , xn ); xi is payment in period i (to us if xi > 0)

I we assume x0 < 0 (i.e., an initial investment) and x0 + x1 + · · · + xn > 0
Ín
I net present value (NPV) of cash flow x, for interest rate r, is PV(x, r) = i=0 (1 + r) −i xi
I internal rate of return (IRR) is smallest interest rate for which PV(x, r) = 0:

IRR(x) = inf{r ≥ 0 | PV(x, r) = 0}

I IRR is quasiconcave: superlevel set is intersection of open halfspaces

n
(1 + r) −i xi > 0 for 0 ≤ r < R
Õ
IRR(x) ≥ R ⇐⇒
i=0

Convex Optimization Boyd and Vandenberghe 3.41

 Properties of quasiconvex functions
I modified Jensen inequality: for quasiconvex f

0≤𝜃≤1 =⇒ f (𝜃x + (1 − 𝜃)y) ≤ max{f (x), f (y)}

I first-order condition: differentiable f with convex domain is quasiconvex if and only if

f (y) ≤ f (x) =⇒ ∇f (x) T (y − x) ≤ 0

∇f (x)
x

I sum of quasiconvex functions is not necessarily quasiconvex

Convex Optimization Boyd and Vandenberghe 3.42

4. Convex optimization problems
 Outline

Optimization problems

Some standard convex problems

Transforming problems

Disciplined convex programming

Geometric programming

Quasiconvex optimization

Multicriterion optimization

Convex Optimization Boyd and Vandenberghe 4.1

 Optimization problem in standard form

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p
I x ∈ Rn is the optimization variable
I f0 : Rn → R is the objective or cost function
I fi : Rn → R, i = 1, . . . , m, are the inequality constraint functions
I hi : Rn → R are the equality constraint functions

Convex Optimization Boyd and Vandenberghe 4.2

 Feasible and optimal points

I x ∈ Rn is feasible if x ∈ dom f0 and it satisfies the constraints

I optimal value is p★ = inf{f0 (x) | fi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , p}

I p★ = ∞ if problem is infeasible
I p★ = −∞ if problem is unbounded below

I a feasible x is optimal if f0 (x) = p★

I Xopt is the set of optimal points

Convex Optimization Boyd and Vandenberghe 4.3

 Locally optimal points
x is locally optimal if there is an R > 0 such that x is optimal for

minimize (over z) f0 (z)

subject to fi (z) ≤ 0, i = 1, . . . , m, hi (z) = 0, i = 1, . . . , p
kz − xk 2 ≤ R

f0 (xlo )

p★

x★ xlo

Convex Optimization Boyd and Vandenberghe 4.4

 Examples

examples with n = 1, m = p = 0
I f0 (x) = 1/x, dom f0 = R++ : p★ = 0, no optimal point
I f0 (x) = − log x, dom f0 = R++ : p★ = −∞
I f0 (x) = x log x, dom f0 = R++ : p★ = −1/e, x = 1/e is optimal
I f0 (x) = x3 − 3x: p★ = −∞, x = 1 is locally optimal
10
6
0
5 3 0
−3
0
0
0 1 2 0 1 2 0 1/e 1 −2 0 2

f0 (x) = 1/x f0 (x) = − log x f0 (x) = x log x f0 (x) = x3 − 3x

Convex Optimization Boyd and Vandenberghe 4.5

 Implicit and explicit constraints

standard form optimization problem has implicit constraint

m p
dom fi ∩ dom hi ,
Ù Ù
x∈D=
i=0 i=1

I we call D the domain of the problem

I the constraints fi (x) ≤ 0, hi (x) = 0 are the explicit constraints
I a problem is unconstrained if it has no explicit constraints (m = p = 0)

example:
Ík
minimize f0 (x) = − i=1 log(bi − aTi x)
is an unconstrained problem with implicit constraints aTi x < bi

Convex Optimization Boyd and Vandenberghe 4.6

 Feasibility problem

find x
subject to fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p

can be considered a special case of the general problem with f0 (x) = 0:

minimize 0
subject to fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p

I p★ = 0 if constraints are feasible; any feasible x is optimal

I p★ = ∞ if constraints are infeasible

Convex Optimization Boyd and Vandenberghe 4.7

 Standard form convex optimization problem

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
aTi x = bi , i = 1, . . . , p
I objective and inequality constraints f0 , f1 , …, fm are convex
I equality constraints are affine, often written as Ax = b
I feasible and optimal sets of a convex optimization problem are convex

I problem is quasiconvex if f0 is quasiconvex, f1 , …, fm are convex, h1 , . . . , hp are affine

Convex Optimization Boyd and Vandenberghe 4.8

 Example

I standard form problem

minimize f0 (x) = x12 + x22

subject to f1 (x) = x1 /(1 + x22 ) ≤ 0
h1 (x) = (x1 + x2 ) 2 = 0

I f0 is convex; feasible set {(x1 , x2 ) | x1 = −x2 ≤ 0} is convex

I not a convex problem (by our definition) since f1 is not convex, h1 is not affine
I equivalent (but not identical) to the convex problem

minimize x12 + x22

subject to x1 ≤ 0
x1 + x2 = 0

Convex Optimization Boyd and Vandenberghe 4.9

 Local and global optima

any locally optimal point of a convex problem is (globally) optimal

proof:
I suppose x is locally optimal, but there exists a feasible y with f0 (y) < f0 (x)
I x locally optimal means there is an R > 0 such that

z feasible, kz − xk 2 ≤ R =⇒ f0 (z) ≥ f0 (x)

I consider z = 𝜃y + (1 − 𝜃)x with 𝜃 = R/(2ky − xk 2 )

I ky − xk 2 > R, so 0 < 𝜃 < 1/2
I z is a convex combination of two feasible points, hence also feasible
I kz − xk 2 = R/2 and f0 (z) ≤ 𝜃f0 (y) + (1 − 𝜃)f0 (x) < f0 (x), which contradicts our assumption
that x is locally optimal

Convex Optimization Boyd and Vandenberghe 4.10

 Optimality criterion for differentiable f0

I x is optimal for a convex problem if and only if it is feasible and

∇f0 (x) T (y − x) ≥ 0 for all feasible y

−∇f0 (x)
X x

I if nonzero, ∇f0 (x) defines a supporting hyperplane to feasible set X at x

Convex Optimization Boyd and Vandenberghe 4.11

 Examples

I unconstrained problem: x minimizes f0 (x) if and only if ∇f0 (x) = 0

I equality constrained problem: x minimizes f0 (x) subject to Ax = b if and only if there

exists a 𝜈 such that
Ax = b, ∇f0 (x) + AT 𝜈 = 0
I minimization over nonnegative orthant: x minimizes f0 (x) over Rn+ if and only if

∇f0 (x)i ≥ 0 xi = 0

x  0,
∇f0 (x)i = 0 xi > 0

Convex Optimization Boyd and Vandenberghe 4.12

 Outline

Optimization problems

Some standard convex problems

Transforming problems

Disciplined convex programming

Geometric programming

Quasiconvex optimization

Multicriterion optimization

Convex Optimization Boyd and Vandenberghe 4.13

 Linear program (LP)

minimize cT x + d
subject to Gx  h
Ax = b
I convex problem with affine objective and constraint functions
I feasible set is a polyhedron

−c

P x★

Convex Optimization Boyd and Vandenberghe 4.14

 Example: Diet problem

I choose nonnegative quantities x1 , …, xn of n foods

I one unit of food j costs cj and contains amount Aij of nutrient i
I healthy diet requires nutrient i in quantity at least bi
I to find cheapest healthy diet, solve

minimize cT x
subject to Ax  b, x0

I express in standard LP form as

minimize cT x   
−A −b
subject to x
−I 0

Convex Optimization Boyd and Vandenberghe 4.15

 Example: Piecewise-linear minimization

I minimize convex piecewise-linear function f0 (x) = maxi=1,...,m (aTi x + bi ), x ∈ Rn

I equivalent to LP
minimize t
subject to aTi x + bi ≤ t, i = 1, . . . , m
n
with variables x ∈ R , t ∈ R

I constraints describe epi f0

Convex Optimization Boyd and Vandenberghe 4.16

 Example: Chebyshev center of a polyhedron

Chebyshev center of P = {x | aTi x ≤ bi , i = 1, . . . , m} is xcheb

center of largest inscribed ball B = {xc + u | kuk 2 ≤ r}

I aTi x ≤ bi for all x ∈ B if and only if

sup{aTi (xc + u) | kuk 2 ≤ r} = aTi xc + rkai k 2 ≤ bi

I hence, xc , r can be determined by solving LP with variables xc , r

maximize r
subject to aTi xc + rkai k 2 ≤ bi , i = 1, . . . , m

Convex Optimization Boyd and Vandenberghe 4.17

 Quadratic program (QP)

minimize (1/2)xT Px + qT x + r
subject to Gx  h
Ax = b

I P ∈ Sn+ , so objective is convex quadratic

I minimize a convex quadratic function over a polyhedron

−∇f0 (x★)

x★
P

Convex Optimization Boyd and Vandenberghe 4.18

 Example: Least squares

I least squares problem: minimize kAx − bk 22

I analytical solution x★ = A† b (A† is pseudo-inverse)

I can add linear constraints, e.g.,
– x  0 (nonnegative least squares)
– x1 ≤ x2 ≤ · · · ≤ xn (isotonic regression)

Convex Optimization Boyd and Vandenberghe 4.19

 Example: Linear program with random cost

I LP with random cost c, with mean c̄ and covariance Σ

I hence, LP objective cT x is random variable with mean c̄T x and variance xT Σx
I risk-averse problem:
minimize E cT x + 𝛾 var(cT x)
subject to Gx  h, Ax = b
I 𝛾 > 0 is risk aversion parameter; controls the trade-off between expected cost and
variance (risk)
I express as QP
minimize c̄T x + 𝛾xT Σx
subject to Gx  h, Ax = b

Convex Optimization Boyd and Vandenberghe 4.20

 Quadratically constrained quadratic program (QCQP)

minimize (1/2)xT P0 x + q0T x + r0

subject to (1/2)xT Pi x + qiT x + ri ≤ 0, i = 1, . . . , m
Ax = b

I Pi ∈ Sn+ ; objective and constraints are convex quadratic

I if P1 , . . . , Pm ∈ Sn++ , feasible region is intersection of m ellipsoids and an affine set

Convex Optimization Boyd and Vandenberghe 4.21

 Second-order cone programming

minimize f T x
subject to kAi x + bi k 2 ≤ ciT x + di , i = 1, . . . , m
Fx = g
(Ai ∈ Rni ×n , F ∈ Rp×n )
I inequalities are called second-order cone (SOC) constraints:

(Ai x + bi , ciT x + di ) ∈ second-order cone in Rni +1

I for ni = 0, reduces to an LP; if ci = 0, reduces to a QCQP

I more general than QCQP and LP

Convex Optimization Boyd and Vandenberghe 4.22

 Example: Robust linear programming
suppose constraint vectors ai are uncertain in the LP

minimize cT x
subject to aTi x ≤ bi , i = 1, . . . , m,

two common approaches to handling uncertainty

I deterministic worst-case: constraints must hold for all ai ∈ Ei (uncertainty ellipsoids)

minimize cT x
subject to aTi x ≤ bi for all ai ∈ Ei , i = 1, . . . , m,

I stochastic: ai is random variable; constraints must hold with probability 𝜂

minimize cT x
subject to prob(aTi x ≤ bi ) ≥ 𝜂, i = 1, . . . , m

Convex Optimization Boyd and Vandenberghe 4.23

 Deterministic worst-case approach

I uncertainty ellipsoids are Ei = {āi + Pi u | kuk 2 ≤ 1}, (āi ∈ Rn , Pi ∈ Rn×n )

I center of Ei is āi ; semi-axes determined by singular values/vectors of Pi
I robust LP
minimize cT x
subject to aTi x ≤ bi ∀ai ∈ Ei , i = 1, . . . , m
I equivalent to SOCP

minimize cT x
subject to āTi x + kPiT xk 2 ≤ bi , i = 1, . . . , m

(follows from sup kuk 2 ≤1 ( āi + Pi u) T x = āTi x + kPiT xk 2 )

Convex Optimization Boyd and Vandenberghe 4.24

 Stochastic approach

I assume ai ∼ N ( āi , Σi )
I aTi x ∼ N ( āTi x, xT Σi x), so
bi − āTi x
!
prob(aTi x ≤ bi ) = Φ
kΣi1/2 xk 2
√ ∫u 2
where Φ(u) = (1/ 2𝜋) −∞ e−t /2 dt is N (0, 1) CDF
I prob(aTi x ≤ bi ) ≥ 𝜂 can be expressed as āTi x + Φ−1 (𝜂)kΣi1/2 xk 2 ≤ bi
I for 𝜂 ≥ 1/2, robust LP equivalent to SOCP

minimize cT x
subject to āTi x + Φ−1 (𝜂)kΣi1/2 xk 2 ≤ bi , i = 1, . . . , m

Convex Optimization Boyd and Vandenberghe 4.25

 Conic form problem

minimize cT x
subject to Fx + g K 0
Ax = b
I constraint Fx + g K 0 involves a generalized inequality with respect to a proper cone K
I linear programming is a conic form problem with K = Rm
+

I as with standard convex problem

– feasible and optimal sets are convex
– any local optimum is global

Convex Optimization Boyd and Vandenberghe 4.26

 Semidefinite program (SDP)

minimize cT x
subject to x1 F1 + x2 F2 + · · · + xn Fn + G  0
Ax = b
with Fi , G ∈ Sk
I inequality constraint is called linear matrix inequality (LMI)
I includes problems with multiple LMI constraints: for example,

x1 F̂1 + · · · + xn F̂n + Ĝ  0, x1 F̃1 + · · · + xn F̃n + G̃  0

is equivalent to single LMI

F̂1 0 F̂2 0 F̂n 0 Ĝ 0

       
x1 + x2 + · · · + xn + 0
0 F̃1 0 F̃2 0 F̃n 0 G̃

Convex Optimization Boyd and Vandenberghe 4.27

 Example: Matrix norm minimization

1/2
minimize kA(x)k 2 = 𝜆max (A(x) T A(x))
where A(x) = A0 + x1 A1 + · · · + xn An (with given Ai ∈ Rp×q )
equivalent SDP
minimize t
tI A(x)

subject to 0
A(x) T tI

I variables x ∈ Rn , t ∈ R
I constraint follows from

kAk 2 ≤ t ⇐⇒ AT A  t 2 I, t ≥ 0
tI A
 
⇐⇒ 0
AT tI

Convex Optimization Boyd and Vandenberghe 4.28

 LP and SOCP as SDP
LP and equivalent SDP

LP: minimize cT x SDP: minimize cT x

subject to Ax  b subject to diag(Ax − b)  0

(note different interpretation of generalized inequalities  in LP and SDP)

SOCP and equivalent SDP

SOCP: minimize f T x
subject to kAi x + bi k 2 ≤ ciT x + di , i = 1, . . . , m

SDP: minimize fT x

(ciT x + di )I Ai x + bi

subject to  0, i = 1, . . . , m
(Ai x + bi ) T ciT x + di

Convex Optimization Boyd and Vandenberghe 4.29

 Outline

Optimization problems

Some standard convex problems

Transforming problems

Disciplined convex programming

Geometric programming

Quasiconvex optimization

Multicriterion optimization

Convex Optimization Boyd and Vandenberghe 4.30

 Change of variables
I 𝜙 : Rn → Rn is one-to-one with 𝜙(dom 𝜙) ⊇ D
I consider (possibly non-convex) problem

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p

I change variables to z with x = 𝜙(z)

I can solve equivalent problem

minimize f̃0 (z)

subject to f̃i (z) ≤ 0, i = 1, . . . , m
h̃i (z) = 0, i = 1, . . . , p

where f̃i (z) = fi (𝜙(z)) and h̃i (z) = hi (𝜙(z))

I recover original optimal point as x★ = 𝜙(z★)
Convex Optimization Boyd and Vandenberghe 4.31
 Example

I non-convex problem
minimize x1 /x2 + x3 /x1
subject to x2 /x3 + x1 ≤ 1
with implicit constraint x  0

I change variables using x = 𝜙(z) = exp z to get

minimize exp(z1 − z2 ) + exp(z3 − z1 )

subject to exp(z2 − z3 ) + exp(z1 ) ≤ 1

which is convex

Convex Optimization Boyd and Vandenberghe 4.32

 Transformation of objective and constraint functions

suppose
I 𝜙0 is monotone increasing
I 𝜓i (u) ≤ 0 if and only if u ≤ 0, i = 1, . . . , m
I 𝜑i (u) = 0 if and only if u = 0, i = 1, . . . , p

standard form optimization problem is equivalent to

minimize 𝜙0 (f0 (x))

subject to 𝜓i (fi (x)) ≤ 0, i = 1, . . . , m
𝜑i (hi (x)) = 0, i = 1, . . . , p

example: minimizing kAx − bk is equivalent to minimizing kAx − bk 2

Convex Optimization Boyd and Vandenberghe 4.33

 Converting maximization to minimization
I suppose 𝜙0 is monotone decreasing
I the maximization problem

maximize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p

is equivalent to the minimization problem

minimize 𝜙0 (f0 (x))

subject to fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p

I examples:
– 𝜙0 (u) = −u transforms maximizing a concave function to minimizing a convex function
– 𝜙0 (u) = 1/u transforms maximizing a concave positive function to minimizing a convex
function
Convex Optimization Boyd and Vandenberghe 4.34
 Eliminating equality constraints

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
Ax = b
is equivalent to
minimize (over z) f0 (Fz + x0 )
subject to fi (Fz + x0 ) ≤ 0, i = 1, . . . , m
where F and x0 are such that Ax = b ⇐⇒ x = Fz + x0 for some z

Convex Optimization Boyd and Vandenberghe 4.35

 Introducing equality constraints

minimize f0 (A0 x + b0 )
subject to fi (Ai x + bi ) ≤ 0, i = 1, . . . , m
is equivalent to
minimize (over x, yi ) f0 (y0 )
subject to fi (yi ) ≤ 0, i = 1, . . . , m
yi = Ai x + bi , i = 0, 1, . . . , m

Convex Optimization Boyd and Vandenberghe 4.36

 Introducing slack variables for linear inequalities

minimize f0 (x)
subject to aTi x ≤ bi , i = 1, . . . , m
is equivalent to
minimize (over x, s) f0 (x)
subject to aTi x + si = bi , i = 1, . . . , m
si ≥ 0, i = 1, . . . m

Convex Optimization Boyd and Vandenberghe 4.37

 Epigraph form

standard form convex problem is equivalent to

minimize (over x, t) t
subject to f0 (x) − t ≤ 0
fi (x) ≤ 0, i = 1, . . . , m
Ax = b

Convex Optimization Boyd and Vandenberghe 4.38

 Minimizing over some variables

minimize f0 (x1 , x2 )
subject to fi (x1 ) ≤ 0, i = 1, . . . , m
is equivalent to
minimize f̃0 (x1 )
subject to fi (x1 ) ≤ 0, i = 1, . . . , m
where f̃0 (x1 ) = inf x2 f0 (x1 , x2 )

Convex Optimization Boyd and Vandenberghe 4.39

 Convex relaxation

I start with nonconvex problem: minimize h(x) subject to x ∈ C

I find convex function ĥ with ĥ(x) ≤ h(x) for all x ∈ dom h (i.e., a pointwise lower bound on
h)
I find set Ĉ ⊇ C (e.g., Ĉ = conv C) described by linear equalities and convex inequalities

Ĉ = {x | fi (x) ≤ 0, i = 1, . . . , m, fm (x) ≤ 0, Ax = b}

I convex problem
minimize ĥ(x)
subject to fi (x) ≤ 0, i = 1, . . . , m, Ax = b
is a convex relaxation of the original problem
I optimal value of relaxation is lower bound on optimal value of original problem

Convex Optimization Boyd and Vandenberghe 4.40

 Example: Boolean LP

I mixed integer linear program (MILP):

minimize cT (x, z)
subject to F (x, z)  g, A(x, z) = b, z ∈ {0, 1}q

with variables x ∈ Rn , z ∈ Rq
I zi are called Boolean variables
I this problem is in general hard to solve

I LP relaxation: replace z ∈ {0, 1}q with z ∈ [0, 1] q

I optimal value of relaxation LP is lower bound on MILP
I can use as heuristic for approximately solving MILP, e.g., relax and round

Convex Optimization Boyd and Vandenberghe 4.41

 Outline

Optimization problems

Some standard convex problems

Transforming problems

Disciplined convex programming

Geometric programming

Quasiconvex optimization

Multicriterion optimization

Convex Optimization Boyd and Vandenberghe 4.42

 Disciplined convex program

I specify objective as
– minimize {scalar convex expression}, or
– maximize {scalar concave expression}

I specify constraints as
– {convex expression} <= {concave expression} or
– {concave expression} >= {convex expression} or
– {affine expression} == {affine expression}

I curvature of expressions are DCP certified, i.e., follow composition rule

I DCP-compliant problems can be automatically transformed to standard forms, then solved

Convex Optimization Boyd and Vandenberghe 4.43

 CVXPY example

math: CVXPY code:

minimize kxk 1 import cvxpy as cp
subject to Ax = b
A, b = ...
kxk ∞ ≤ 1
x = cp.Variable(n)
I x is the variable obj = cp.norm(x, 1)
I A, b are given constr = [
A @ x == b,
cp.norm(x, 'inf') <= 1,
]
prob = cp.Problem(cp.Minimize(obj), constr)
prob.solve()

Convex Optimization Boyd and Vandenberghe 4.44

 How CVXPY works

I starts with your optimization problem P1

I finds a sequence of equivalent problems P2 , . . . , PN
I final problem PN matches a standard form (e.g., LP, QP, SOCP, or SDP)
I calls a specialized solver on PN
I retrieves solution of original problem by reversing the transformations

P1 ⇐⇒ P2 ⇐⇒ ··· ⇐⇒ PN−1 ⇐⇒ PN

your problem standard problem

Convex Optimization Boyd and Vandenberghe 4.45

 Outline

Optimization problems

Some standard convex problems

Transforming problems

Disciplined convex programming

Geometric programming

Quasiconvex optimization

Multicriterion optimization

Convex Optimization Boyd and Vandenberghe 4.46

 Geometric programming
I monomial function:
f (x) = cx1a1 x2a2 · · · xnan , dom f = Rn++
with c > 0; exponent ai can be any real number
I posynomial function: sum of monomials

K
ck x1a1k x2a2k · · · xnank , dom f = Rn++
Õ
f (x) =
k=1

I geometric program (GP)

minimize f0 (x)
subject to fi (x) ≤ 1, i = 1, . . . , m
hi (x) = 1, i = 1, . . . , p

with fi posynomial, hi monomial

Convex Optimization Boyd and Vandenberghe 4.47
 Geometric program in convex form
I change variables to yi = log xi , and take logarithm of cost, constraints
I monomial f (x) = cx1a1 · · · xnan transforms to

log f (ey1 , . . . , eyn ) = aT y + b (b = log c)

a1k a2k
· · · xnank transforms to
ÍK
I posynomial f (x) = k=1 ck x1 x2

K
!
T
log f (ey1 , . . . , eyn ) = log eak y+bk
Õ
(bk = log ck )
k=1

I geometric program transforms to convex problem

minimize log Kk=1 exp(aT0k y + b0k )

Í 

subject to log Kk=1 exp(aTik y + bik ) ≤ 0,

Í 
i = 1, . . . , m
Gy + d = 0

Convex Optimization Boyd and Vandenberghe 4.48

 Examples: Frobenius norm diagonal scaling

I we seek diagonal matrix D = diag(d), d  0, to minimize kDMD−1 k 2F

I express as
n  2 Õ n
kDMD−1 k 2F = Mij2 di2 /dj2
Õ
DMD−1 =
ij
i,j=1 i,j=1

I a posynomial in d (with exponents 0, 2, and −2)

I in convex form, with y = log d,

n
log kDMD−1 k 2F = log ­
©Õ
exp 2(yi − yj + log |Mij |) ®

ª

«i,j=1 ¬

Convex Optimization Boyd and Vandenberghe 4.49

 Outline

Optimization problems

Some standard convex problems

Transforming problems

Disciplined convex programming

Geometric programming

Quasiconvex optimization

Multicriterion optimization

Convex Optimization Boyd and Vandenberghe 4.50

 Quasiconvex optimization

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
Ax = b
with f0 : Rn → R quasiconvex, f1 , …, fm convex
can have locally optimal points that are not (globally) optimal

(x, f0 (x))

Convex Optimization Boyd and Vandenberghe 4.51

 Linear-fractional program

I linear-fractional program

minimize (cT x + d)/(eT x + f )

subject to Gx  h, Ax = b

with variable x and implicit constraint eT x + f > 0

I equivalent to the LP (with variables y, z)

minimize cT y + dz
subject to Gy  hz, Ay = bz
eT y + fz = 1, z ≥ 0

I recover x★ = y★/z★

Convex Optimization Boyd and Vandenberghe 4.52

 Von Neumann model of a growing economy

I x, x+ ∈ Rn++ : activity levels of n economic sectors, in current and next period

I (Ax)i : amount of good i produced in current period
I (Bx+ )i : amount of good i consumed in next period
I Bx+  Ax: goods consumed next period no more than produced this period
I xi+ /xi : growth rate of sector i
I allocate activity to maximize growth rate of slowest growing sector

maximize (over x, x+ ) mini=1,...,n xi+ /xi

subject to x+  0, Bx+  Ax

I a quasiconvex problem with variables x, x+

Convex Optimization Boyd and Vandenberghe 4.53

 Convex representation of sublevel sets

I if f0 is quasiconvex, there exists a family of functions 𝜙t such that:

– 𝜙t (x) is convex in x for fixed t
– t-sublevel set of f0 is 0-sublevel set of 𝜙t , i.e., f0 (x) ≤ t ⇐⇒ 𝜙t (x) ≤ 0

example:
I f0 (x) = p(x)/q(x), with p convex and nonnegative, q concave and positive
I take 𝜙t (x) = p(x) − tq(x): for t ≥ 0,
– 𝜙t convex in x
– p(x)/q(x) ≤ t if and only if 𝜙t (x) ≤ 0

Convex Optimization Boyd and Vandenberghe 4.54

 Bisection method for quasiconvex optimization
I for fixed t, consider convex feasiblity problem

𝜙t (x) ≤ 0, fi (x) ≤ 0, i = 1, . . . , m, Ax = b (1)

if feasible, we can conclude that t ≥ p★; if infeasible, t ≤ p★

I bisection method:

given l ≤ p★, u ≥ p★, tolerance 𝜖 > 0.

repeat
1. t := (l + u)/2.
2. Solve the convex feasibility problem (1).
3. if (1) is feasible, u := t; else l := t.
until u − l ≤ 𝜖.

I requires exactly dlog2 ((u − l)/𝜖)e iterations

Convex Optimization Boyd and Vandenberghe 4.55

 Outline

Optimization problems

Some standard convex problems

Transforming problems

Disciplined convex programming

Geometric programming

Quasiconvex optimization

Multicriterion optimization

Convex Optimization Boyd and Vandenberghe 4.56

 Multicriterion optimization

I multicriterion or multi-objective problem:

minimize f0 (x) = (F1 (x), . . . , Fq (x))

subject to fi (x) ≤ 0, i = 1, . . . , m, Ax = b

I objective is the vector f0 (x) ∈ Rq

I q different objectives F1 , . . . , Fq ; roughly speaking we want all Fi ’s to be small
I feasible x★ is optimal if y feasible =⇒ f0 (x★)  f0 (y)
I this means that x★ simultaneously minimizes each Fi ; the objectives are noncompeting
I not surprisingly, this doesn’t happen very often

Convex Optimization Boyd and Vandenberghe 4.57

 Pareto optimality

I feasible x dominates another feasible x̃ if f0 (x)  f0 ( x̃) and for at least one i, Fi (x) < Fi ( x̃)
I i.e., x meets x̃ on all objectives, and beats it on at least one

I feasible xpo is Pareto optimal if it is not dominated by any feasible point

I can be expressed as: y feasible, f0 (y)  f0 (xpo ) =⇒ f0 (xpo ) = f0 (y)

I there are typically many Pareto optimal points

I for q = 2, set of Pareto optimal objective values is the optimal trade-off curve
I for q = 3, set of Pareto optimal objective values is the optimal trade-off surface

Convex Optimization Boyd and Vandenberghe 4.58

 Optimal and Pareto optimal points

set of achievable objective values O = {f0 (x) | x feasible}

I feasible x is optimal if f0 (x) is the minimum value of O

I feasible x is Pareto optimal if f0 (x) is a minimal value of O

O
O
f0 (xpo )

f0 (x★)
x★ is optimal xpo is Pareto optimal

Convex Optimization Boyd and Vandenberghe 4.59

 Regularized least-squares

I minimize (kAx − bk 22 , kxk 22 ) (first objective is loss; second is regularization)

I example with A ∈ R100×10 ; heavy line shows Pareto optimal points

25

20 O

F2 (x) = kxk 22
15

10

0
0 10 20 30 40 50

F1 (x) = kAx − bk 22

Convex Optimization Boyd and Vandenberghe 4.60

 Risk return trade-off in portfolio optimization

I variable x ∈ Rn is investment portfolio, with xi fraction invested in asset i

I p̄ ∈ Rn is mean, Σ is covariance of asset returns

I portfolio return has mean p̄T x, variance xT Σx

I minimize (−p̄T x, xT Σx), subject to 1T x = 1, x  0

I Pareto optimal portfolios trace out optimal risk-return curve

Convex Optimization Boyd and Vandenberghe 4.61

 Example

15%
1

x(4) x(3) x(2)

mean return

10%

allocation x
0.5
x(1)
5%

0
0%
0% 10% 20% 0% 10% 20%

standard deviation of return standard deviation of return

Convex Optimization Boyd and Vandenberghe 4.62

 Scalarization

I scalarization combines the multiple objectives into one (scalar) objective

I a standard method for finding Pareto optimal points
I choose 𝜆  0 and solve scalar problem

minimize 𝜆T f0 (x) = 𝜆1 F1 (x) + · · · + 𝜆q Fq (x)

subject to fi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , p

I 𝜆i are relative weights on the objectives

I if x is optimal for scalar problem, then it is Pareto-optimal for multicriterion problem
I for convex problems, can find (almost) all Pareto optimal points by varying 𝜆  0

Convex Optimization Boyd and Vandenberghe 4.63

 Example

f0 (x1 )

f0 (x3 )
𝜆1
𝜆2
f0 (x2 )

Convex Optimization Boyd and Vandenberghe 4.64

 Example: Regularized least-squares
I regularized least-squares problem: minimize (kAx − bk 22 , kxk 22 )
I take 𝜆 = (1, 𝛾) with 𝛾 > 0, and minimize kAx − bk 22 + 𝛾kxk 22

20

15

kxk 22 10

5
𝛾=1
0
0 5 10 15 20
kAx − bk 22

Convex Optimization Boyd and Vandenberghe 4.65

 Example: Risk-return trade-off

I risk-return trade-off: minimize (−p̄T x, xT Σx) subject to 1T x = 1, x  0

I with 𝜆 = (1, 𝛾) we obtain scalarized problem

minimize −p̄T x + 𝛾xT Σx

subject to 1T x = 1, x  0

I objective is negative risk-adjusted return, p̄T x − 𝛾xT Σx

I 𝛾 is called the risk-aversion parameter

Convex Optimization Boyd and Vandenberghe 4.66

5. Duality
 Outline

Lagrangian and dual function

Lagrange dual problem

KKT conditions

Sensitivity analysis

Problem reformulations

Theorems of alternatives

Convex Optimization Boyd and Vandenberghe 5.1

 Lagrangian
I standard form problem (not necessarily convex)

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p

variable x ∈ Rn , domain D, optimal value p★

I Lagrangian: L : Rn × Rm × Rp → R, with dom L = D × Rm × Rp ,
m
Õ p
Õ
L(x, 𝜆, 𝜈) = f0 (x) + 𝜆i fi (x) + 𝜈i hi (x)
i=1 i=1

– weighted sum of objective and constraint functions

– 𝜆 i is Lagrange multiplier associated with fi (x) ≤ 0
– 𝜈i is Lagrange multiplier associated with hi (x) = 0

Convex Optimization Boyd and Vandenberghe 5.2

 Lagrange dual function

I Lagrange dual function: g : Rm × Rp → R,

m p
!
Õ Õ
g(𝜆, 𝜈) = inf L(x, 𝜆, 𝜈) = inf f0 (x) + 𝜆i fi (x) + 𝜈i hi (x)
x∈ D x∈ D
i=1 i=1

I g is concave, can be −∞ for some 𝜆, 𝜈

I lower bound property: if 𝜆  0, then g(𝜆, 𝜈) ≤ p★
I proof: if x̃ is feasible and 𝜆  0, then

f0 ( x̃) ≥ L( x̃, 𝜆, 𝜈) ≥ inf L(x, 𝜆, 𝜈) = g(𝜆, 𝜈)

x∈ D

minimizing over all feasible x̃ gives p★ ≥ g(𝜆, 𝜈)

Convex Optimization Boyd and Vandenberghe 5.3

 Least-norm solution of linear equations

minimize xT x
subject to Ax = b

I Lagrangian is L(x, 𝜈) = xT x + 𝜈 T (Ax − b)

I to minimize L over x, set gradient equal to zero:

∇x L(x, 𝜈) = 2x + AT 𝜈 = 0 =⇒ x = −(1/2)AT 𝜈

I plug x into L to obtain

1
g(𝜈) = L((−1/2)AT 𝜈, 𝜈) = − 𝜈 T AAT 𝜈 − bT 𝜈
4

I lower bound property: p★ ≥ −(1/4)𝜈 T AAT 𝜈 − bT 𝜈 for all 𝜈

Convex Optimization Boyd and Vandenberghe 5.4

 Standard form LP

minimize cT x
subject to Ax = b, x0
I Lagrangian is

L(x, 𝜆, 𝜈) = cT x + 𝜈 T (Ax − b) − 𝜆T x = −bT 𝜈 + (c + AT 𝜈 − 𝜆) T x

I L is affine in x, so

−bT 𝜈 AT 𝜈 − 𝜆 + c = 0

g(𝜆, 𝜈) = inf L(x, 𝜆, 𝜈) =
x −∞ otherwise

I g is linear on affine domain {(𝜆, 𝜈) | AT 𝜈 − 𝜆 + c = 0}, hence concave

I lower bound property: p★ ≥ −bT 𝜈 if AT 𝜈 + c  0

Convex Optimization Boyd and Vandenberghe 5.5

 Equality constrained norm minimization

minimize kxk
subject to Ax = b
I dual function is

bT 𝜈 kAT 𝜈k ∗ ≤ 1

T T
g(𝜈) = inf (kxk − 𝜈 Ax + b 𝜈) =
x −∞ otherwise

where kvk ∗ = sup kuk ≤1 uT v is dual norm of k · k

I lower bound property: p★ ≥ bT 𝜈 if kAT 𝜈k ∗ ≤ 1

Convex Optimization Boyd and Vandenberghe 5.6

 Two-way partitioning

minimize xT Wx
subject to xi2 = 1, i = 1, . . . , n

I a nonconvex problem; feasible set contains 2n discrete points

I interpretation: partition {1, . . . , n} in two sets encoded as xi = 1 and xi = −1
I Wij is cost of assigning i, j to the same set; −Wij is cost of assigning to different sets
I dual function is
!
−1T 𝜈 W + diag(𝜈)  0

T
𝜈i (xi − 1) = inf xT (W + diag(𝜈)) x−1T 𝜈 =
2
Õ
g(𝜈) = inf x Wx +
x x −∞ otherwise
i

I lower bound property: p★ ≥ −1T 𝜈 if W + diag(𝜈)  0

Convex Optimization Boyd and Vandenberghe 5.7

 Lagrange dual and conjugate function

minimize f0 (x)
subject to Ax  b, Cx = d
I dual function

f0 (x) + (AT 𝜆 + C T 𝜈) T x − bT 𝜆 − d T 𝜈
 
g(𝜆, 𝜈) = inf
x∈dom f0

= −f0∗ (−AT 𝜆 − C T 𝜈) − bT 𝜆 − d T 𝜈

where f ∗ (y) = supx∈dom f (yT x − f (x)) is conjugate of f0

I simplifies derivation of dual if conjugate of f0 is known
I example: entropy maximization
n n
eyi −1
Õ Õ
f0 (x) = xi log xi , f0∗ (y) =
i=1 i=1

Convex Optimization Boyd and Vandenberghe 5.8

 Outline

Lagrangian and dual function

Lagrange dual problem

KKT conditions

Sensitivity analysis

Problem reformulations

Theorems of alternatives

Convex Optimization Boyd and Vandenberghe 5.9

 The Lagrange dual problem

(Lagrange) dual problem

maximize g(𝜆, 𝜈)
subject to 𝜆  0

I finds best lower bound on p★, obtained from Lagrange dual function
I a convex optimization problem, even if original primal problem is not
I dual optimal value denoted d★
I 𝜆, 𝜈 are dual feasible if 𝜆  0, (𝜆, 𝜈) ∈ dom g
I often simplified by making implicit constraint (𝜆, 𝜈) ∈ dom g explicit

Convex Optimization Boyd and Vandenberghe 5.10

 Example: standard form LP
(see slide 5.5)
I primal standard form LP:
minimize cT x
subject to Ax = b
x0
I dual problem is
maximize g(𝜆, 𝜈)
subject to 𝜆  0
with g(𝜆, 𝜈) = −bT 𝜈 if AT 𝜈 − 𝜆 + c = 0, −∞ otherwise
I make implicit constraint explicit, and eliminate 𝜆 to obtain (transformed) dual problem

maximize −bT 𝜈
subject to AT 𝜈 + c  0

Convex Optimization Boyd and Vandenberghe 5.11

 Weak and strong duality

weak duality: d★ ≤ p★
I always holds (for convex and nonconvex problems)
I can be used to find nontrivial lower bounds for difficult problems, e.g., solving the SDP

maximize −1T 𝜈
subject to W + diag(𝜈)  0

gives a lower bound for the two-way partitioning problem on page 5.7

strong duality: d★ = p★
I does not hold in general
I (usually) holds for convex problems
I conditions that guarantee strong duality in convex problems are called constraint
qualifications

Convex Optimization Boyd and Vandenberghe 5.12

 Slater’s constraint qualification

strong duality holds for a convex problem

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
Ax = b

if it is strictly feasible, i.e., there is an x ∈ int D with fi (x) < 0, i = 1, . . . , m, Ax = b

I also guarantees that the dual optimum is attained (if p★ > −∞)
I can be sharpened: e.g.,
– can replace int D with relint D (interior relative to affine hull)
– affine inequalities do not need to hold with strict inequality
I there are many other types of constraint qualifications

Convex Optimization Boyd and Vandenberghe 5.13

 Inequality form LP

primal problem
minimize cT x
subject to Ax  b

dual function   −bT 𝜆 AT 𝜆 + c = 0

g(𝜆) = inf (c + AT 𝜆) T x − bT 𝜆 =

x −∞ otherwise

dual problem
maximize −bT 𝜆
subject to AT 𝜆 + c = 0, 𝜆0

I from the sharpened Slater’s condition: p★ = d★ if the primal problem is feasible

I in fact, p★ = d★ except when primal and dual are both infeasible

Convex Optimization Boyd and Vandenberghe 5.14

 Quadratic program
primal problem (assume P ∈ Sn++ )

minimize xT Px
subject to Ax  b

dual function
1
g(𝜆) = inf xT Px + 𝜆T (Ax − b) = − 𝜆T AP −1 AT 𝜆 − bT 𝜆
 
x 4

dual problem
maximize −(1/4)𝜆T AP −1 AT 𝜆 − bT 𝜆
subject to 𝜆  0

I from the sharpened Slater’s condition: p★ = d★ if the primal problem is feasible

I in fact, p★ = d★ always

Convex Optimization Boyd and Vandenberghe 5.15

 Geometric interpretation

I for simplicity, consider problem with one constraint f1 (x) ≤ 0

I G = {(f1 (x), f0 (x)) | x ∈ D} is set of achievable (constraint, objective) values
I interpretation of dual function: g(𝜆) = inf (u,t) ∈ G (t + 𝜆u)
t t

G G
p★ p★
𝜆u + t = g(𝜆) d★
g(𝜆)
u u

I 𝜆u + t = g(𝜆) is (non-vertical) supporting hyperplane to G

I hyperplane intersects t-axis at t = g(𝜆)

Convex Optimization Boyd and Vandenberghe 5.16

 Epigraph variation

I same with G replaced with A = {(u, t) | f1 (x) ≤ u, f0 (x) ≤ t for some x ∈ D}

t
A

𝜆u + t = g(𝜆) p★
g(𝜆)
u

I strong duality holds if there is a non-vertical supporting hyperplane to A at (0, p★)

I for convex problem, A is convex, hence has supporting hyperplane at (0, p★)
I Slater’s condition: if there exist ( ũ, t̃) ∈ A with ũ < 0, then supporting hyperplane at
(0, p★) must be non-vertical

Convex Optimization Boyd and Vandenberghe 5.17

 Outline

Lagrangian and dual function

Lagrange dual problem

KKT conditions

Sensitivity analysis

Problem reformulations

Theorems of alternatives

Convex Optimization Boyd and Vandenberghe 5.18

 Complementary slackness
I assume strong duality holds, x★ is primal optimal, (𝜆★, 𝜈★) is dual optimal
m p
!
Õ Õ
f0 (x ) = g(𝜆 , 𝜈 )
★ ★ ★
= inf f0 (x) + 𝜆★i fi (x) + i hi (x)
𝜈★
x
i=1 i=1
m
Õ p
Õ
≤ f0 (x★) + 𝜆★i fi (x★) + i hi (x )
𝜈★ ★

i=1 i=1
≤ f0 (x★)

I hence, the two inequalities hold with equality

I x★ minimizes L(x, 𝜆★, 𝜈★)
I 𝜆★i fi (x★) = 0 for i = 1, . . . , m (known as complementary slackness):

𝜆★i > 0 =⇒ fi (x★) = 0, fi (x★) < 0 =⇒ 𝜆★i = 0

Convex Optimization Boyd and Vandenberghe 5.19

 Karush-Kuhn-Tucker (KKT) conditions

the KKT conditions (for a problem with differentiable fi , hi ) are

1. primal constraints: fi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , p
2. dual constraints: 𝜆  0
3. complementary slackness: 𝜆 i fi (x) = 0, i = 1, . . . , m
4. gradient of Lagrangian with respect to x vanishes:
m
Õ p
Õ
∇f0 (x) + 𝜆i ∇fi (x) + 𝜈i ∇hi (x) = 0
i=1 i=1

if strong duality holds and x, 𝜆, 𝜈 are optimal, they satisfy the KKT conditions

Convex Optimization Boyd and Vandenberghe 5.20

 KKT conditions for convex problem

if x̃, 𝜆,
˜ 𝜈˜ satisfy KKT for a convex problem, then they are optimal:
I from complementary slackness: f0 ( x̃) = L( x̃, 𝜆,
˜ 𝜈)
˜
I from 4th condition (and convexity): g(𝜆, ˜ = L( x̃, 𝜆,
˜ 𝜈) ˜ 𝜈)
˜
hence, f0 ( x̃) = g(𝜆,
˜ 𝜈)
˜

if Slater’s condition is satisfied, then

x is optimal if and only if there exist 𝜆, 𝜈 that satisfy KKT conditions
I recall that Slater implies strong duality, and dual optimum is attained
I generalizes optimality condition ∇f0 (x) = 0 for unconstrained problem

Convex Optimization Boyd and Vandenberghe 5.21

 Outline

Lagrangian and dual function

Lagrange dual problem

KKT conditions

Sensitivity analysis

Problem reformulations

Theorems of alternatives

Convex Optimization Boyd and Vandenberghe 5.22

 Perturbation and sensitivity analysis
(unperturbed) optimization problem and its dual

minimize f0 (x) maximize g(𝜆, 𝜈)

subject to fi (x) ≤ 0, i = 1, . . . , m subject to 𝜆  0
hi (x) = 0, i = 1, . . . , p

perturbed problem and its dual

minimize f0 (x) maximize g(𝜆, 𝜈) − uT 𝜆 − vT 𝜈

subject to fi (x) ≤ ui , i = 1, . . . , m subject to 𝜆  0
hi (x) = vi , i = 1, . . . , p

I x is primal variable; u, v are parameters

I p★ (u, v) is optimal value as a function of u, v
I p★ (0, 0) is optimal value of unperturbed problem

Convex Optimization Boyd and Vandenberghe 5.23

 Global sensitivity via duality

I assume strong duality holds for unperturbed problem, with 𝜆★, 𝜈★ dual optimal
I apply weak duality to perturbed problem:

p★ (u, v) ≥ g(𝜆★, 𝜈★) − uT 𝜆★ − vT 𝜈★ = p★ (0, 0) − uT 𝜆★ − vT 𝜈★

I implications
– if 𝜆★
i large: p increases greatly if we tighten constraint i (ui < 0)
★

– if 𝜆★
i small: p does not decrease much if we loosen constraint i (ui > 0)
★

– if 𝜈★
i large and positive: p increases greatly if we take vi < 0
★

– if 𝜈★
i large and negative: p increases greatly if we take vi > 0
★

– if 𝜈★
i small and positive: p does not decrease much if we take vi > 0
★

– if 𝜈★
i small and negative: p does not decrease much if we take vi < 0
★

Convex Optimization Boyd and Vandenberghe 5.24

 Local sensitivity via duality
if (in addition) p★ (u, v) is differentiable at (0, 0), then
𝜕p★ (0, 0) 𝜕p★ (0, 0)
𝜆★i = − , 𝜈★
i =−
𝜕ui 𝜕vi
proof (for 𝜆★i ): from global sensitivity result,
𝜕p★ (0, 0) p★ (tei , 0) − p★ (0, 0) 𝜕p★ (0, 0) p★ (tei , 0) − p★ (0, 0)
= lim ≥ −𝜆★i = lim ≤ −𝜆★i
𝜕ui t&0 t 𝜕ui t%0 t
hence, equality

p★ (u) for a problem with one (inequality) constraint:

u
u=0 p★ (u)

p★ (0) − 𝜆★u
Convex Optimization Boyd and Vandenberghe 5.25
 Outline

Lagrangian and dual function

Lagrange dual problem

KKT conditions

Sensitivity analysis

Problem reformulations

Theorems of alternatives

Convex Optimization Boyd and Vandenberghe 5.26

 Duality and problem reformulations

I equivalent formulations of a problem can lead to very different duals

I reformulating primal problem can be useful when dual is difficult to derive, or uninteresting

common reformulations
I introduce new variables and equality constraints
I make explicit constraints implicit or vice-versa
I transform objective or constraint functions, e.g., replace f0 (x) by 𝜙(f0 (x)) with 𝜙 convex,
increasing

Convex Optimization Boyd and Vandenberghe 5.27

 Introducing new variables and equality constraints

I unconstrained problem: minimize f0 (Ax + b)

I dual function is constant: g = inf x L(x) = inf x f0 (Ax + b) = p★
I we have strong duality, but dual is quite useless

I introduce new variable y and equality constraints y = Ax + b

minimize f0 (y)
subject to Ax + b − y = 0

I dual of reformulated problem is

maximize bT 𝜈 − f0∗ (𝜈)

subject to AT 𝜈 = 0

I a nontrivial, useful dual (assuming the conjugate f0∗ is easy to express)

Convex Optimization Boyd and Vandenberghe 5.28

 Example: Norm approximation

I minimize kAx − bk
I reformulate as minimize kyk subject to y = Ax − b
I recall conjugate of general norm:

0 kzk ∗ ≤ 1

∗
kzk =
∞ otherwise

I dual of (reformulated) norm approximation problem:

maximize bT 𝜈
subject to AT 𝜈 = 0, k𝜈k ∗ ≤ 1

Convex Optimization Boyd and Vandenberghe 5.29

 Outline

Lagrangian and dual function

Lagrange dual problem

KKT conditions

Sensitivity analysis

Problem reformulations

Theorems of alternatives

Convex Optimization Boyd and Vandenberghe 5.30

 Theorems of alternatives

I consider two systems of inequality and equality constraints

I called weak alternatives if no more than one system is feasible
I called strong alternatives if exactly one of them is feasible
I examples: for any a ∈ R, with variable x ∈ R,
– x > a and x ≤ a − 1 are weak alternatives
– x > a and x ≤ a are strong alternatives

I a theorem of alternatives states that two inequality systems are (weak or strong)
alternatives
I can be considered the extension of duality to feasibility problems

Convex Optimization Boyd and Vandenberghe 5.31

 Feasibility problems

I consider system of (not necessarily convex) inequalities and equalities

fi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , p

I express as feasibility problem

minimize 0
subject to fi (x) ≤ 0, i = 1, . . . , m,
hi (x) = 0, i = 1, . . . , p

I if system if feasible, p★ = 0; if not, p★ = ∞

Convex Optimization Boyd and Vandenberghe 5.32

 Duality for feasibility problems
Í
m Íp 
I dual function of feasibility problem is g(𝜆, 𝜈) = inf x i=1 𝜆 i fi (x) + i=1 𝜈i hi (x)
I for 𝜆  0, we have g(𝜆, 𝜈) ≤ p★
I it follows that feasibility of the inequality system

𝜆  0, g(𝜆, 𝜈) > 0

implies the original system is infeasible

I so this is a weak alternative to original system
I it is strong if fi convex, hi affine, and a constraint qualification holds
I g is positive homogeneous so we can write alternative system as

𝜆  0, g(𝜆, 𝜈) ≥ 1

Convex Optimization Boyd and Vandenberghe 5.33

 Example: Nonnegative solution of linear equations

I consider system
Ax = b, x0

−bT 𝜈 AT 𝜈 = 𝜆

I dual function is g(𝜆, 𝜈) =
−∞ otherwise

I can express strong alternative of Ax = b, x  0 as

AT 𝜈  0, bT 𝜈 ≤ −1

(we can replace bT 𝜈 ≤ −1 with bT 𝜈 = −1)

Convex Optimization Boyd and Vandenberghe 5.34

 Farkas’ lemma

I Farkas’ lemma:
Ax  0, cT x < 0 and AT y + c = 0, y0
are strong alternatives

I proof: use (strong) duality for (feasible) LP

minimize cT x
subject to Ax  0

Convex Optimization Boyd and Vandenberghe 5.35

 Investment arbitrage

I we invest xj in each of n assets 1, . . . , n with prices p1 , . . . , pn

I our initial cost is pT x
I at the end of the investment period there are only m possible outcomes i = 1, . . . , m
I Vij is the payoff or final value of asset j in outcome i
I first investment is risk-free (cash): p1 = 1 and Vi1 = 1 for all i

I arbitrage means there is x with pT x < 0, Vx  0

I arbitrage means we receive money up front, and our investment cannot lose
I standard assumption in economics: the prices are such that there is no arbitrage

Convex Optimization Boyd and Vandenberghe 5.36

 Absence of arbitrage

I by Farkas’ lemma, there is no arbitrage ⇐⇒ there exists y ∈ Rm + with V y = p

T

I since first column of V is 1, we have 1T y = 1

I y is interpreted as a risk-neutral probability on the outcomes 1, . . . , m
I V T y are the expected values of the payoffs under the risk-neutral probability
I interpretation of V T y = p:
asset prices equal their expected payoff under the risk-neutral probability

I arbitrage theorem: there is no arbitrage ⇔ there exists a risk-neutral probability

distribution under which each asset price is its expected payoff

Convex Optimization Boyd and Vandenberghe 5.37

 Example
 1.0 0.5 0.0 
 1.0   1.0 
1.0 0.8 0.0
 
V =  p =  0.9  , p̃ =  0.8 
     
1.0 1.0 1.0
,
 0.3   0.7 

1.0 1.3 4.0
 
     
 
I with prices p, there is an arbitrage

 6.2 
 2.35 
0.04
 
T T
x =  −7.7  , p x = −0.2, 1 x = 0, Vx = 
   
0.00

 1.5 

2.19
 
   
 
I with prices p̃, there is no arbitrage, with risk-neutral probability
 0.36 
 1.0 
 0.27 
 
T
y=  V y =  0.8 
 
0.26
 
 0.7 

 0.11 
 
 
 
Convex Optimization Boyd and Vandenberghe 5.38
6. Approximation and fitting
 Outline

Norm and penalty approximation

Regularized approximation

Robust approximation

Convex Optimization Boyd and Vandenberghe 6.1

 Norm approximation

I minimize kAx − bk, with A ∈ Rm×n , m ≥ n, k · k is any norm

I approximation: Ax★ is the best approximation of b by a linear combination of columns of

A
I geometric: Ax★ is point in R (A) closest to b (in norm k · k)
I estimation: linear measurement model y = Ax + v
– measurement y, v is measurement error, x is to be estimated
– implausibility of v is kvk
– given y = b, most plausible x is x★
I optimal design: x are design variables (input), Ax is result (output)
– x★ is design that best approximates desired result b (in norm k · k)

Convex Optimization Boyd and Vandenberghe 6.2

 Examples

I Euclidean approximation (k · k 2 )
– solution x★ = A† b

I Chebyshev or minimax approximation (k · k ∞ )

– can be solved via LP
minimize t
subject to −t1  Ax − b  t1

I sum of absolute residuals approximation (k · k 1 )

– can be solved via LP
minimize 1T y
subject to −y  Ax − b  y

Convex Optimization Boyd and Vandenberghe 6.3

 Penalty function approximation

minimize 𝜙(r1 ) + · · · + 𝜙(rm )

subject to r = Ax − b
(A ∈ Rm×n , 𝜙 : R → R is a convex penalty function)
examples 2

I quadratic: 𝜙(u) = u2 log barrier

quadratic
I deadzone-linear with width a:
1.5

𝜙(u)
𝜙(u) = max{0, |u| − a} 1
deadzone-linear

I log-barrier with limit a: 0.5

−a2 log(1 − (u/a) 2 ) |u| < a


0
𝜙(u) =
otherwise
−1.5 −1 −0.5 0 0.5 1 1.5
∞ u

Convex Optimization Boyd and Vandenberghe 6.4

 Example: histograms of residuals
A ∈ R100×30 ; shape of penalty function affects distribution of residuals
40

absolute value 𝜙(u) = |u|

0
−2 −1 0 1 2
10

square 𝜙(u) = u2
0
−2 −1 0 1 2
20

deadzone 𝜙(u) = max{0, |u| − 0.5}

0
−2 −1 0 1 2
10

log-barrier 𝜙(u) = − log(1 − u2 )

0
−2 −1 0 1 2
r

Convex Optimization Boyd and Vandenberghe 6.5

 Huber penalty function

1.5

𝜙hub (u)
u2 |u| ≤ M

1
𝜙hub (u) =
M (2|u| − M) |u| > M
0.5

0
−1.5 −1 −0.5 0 0.5 1 1.5
u

I linear growth for large u makes approximation less sensitive to outliers

I called a robust penalty

Convex Optimization Boyd and Vandenberghe 6.6

 Example

20

10

f (t)
0

−10

−20
−10 −5 0 5 10
t

I 42 points (circles) ti , yi , with two outliers

I affine function f (t) = 𝛼 + 𝛽t fit using quadratic (dashed) and Huber (solid) penalty

Convex Optimization Boyd and Vandenberghe 6.7

 Least-norm problems

I least-norm problem:
minimize kxk
subject to Ax = b,
with A ∈ Rm×n , m ≤ n, k · k is any norm

I geometric: x★ is smallest point in solution set {x | Ax = b}

I estimation:
– b = Ax are (perfect) measurements of x
– kxk is implausibility of x
– x★ is most plausible estimate consistent with measurements
I design: x are design variables (inputs); b are required results (outputs)
– x★ is smallest (‘most efficient’) design that satisfies requirements

Convex Optimization Boyd and Vandenberghe 6.8

 Examples

I least Euclidean norm (k · k 2 )

– solution x = A† b (assuming b ∈ R (A))

I least sum of absolute values (k · k 1 )

– can be solved via LP
minimize 1T y
subject to −y  x  y, Ax = b
– tends to yield sparse x★

Convex Optimization Boyd and Vandenberghe 6.9

 Outline

Norm and penalty approximation

Regularized approximation

Robust approximation

Convex Optimization Boyd and Vandenberghe 6.10

 Regularized approximation

I a bi-objective problem:

minimize (w.r.t. R2+ ) (kAx − bk, kxk)

I A ∈ Rm×n , norms on Rm and Rn can be different

I interpretation: find good approximation Ax ≈ b with small x

I estimation: linear measurement model y = Ax + v, with prior knowledge that kxk is small
I optimal design: small x is cheaper or more efficient, or the linear model y = Ax is only
valid for small x
I robust approximation: good approximation Ax ≈ b with small x is less sensitive to errors
in A than good approximation with large x

Convex Optimization Boyd and Vandenberghe 6.11

 Scalarized problem

I minimize kAx − bk + 𝛾kxk

I solution for 𝛾 > 0 traces out optimal trade-off curve
I other common method: minimize kAx − bk 2 + 𝛿kxk 2 with 𝛿 > 0
I with k · k 2 , called Tikhonov regularization or ridge regression

minimize kAx − bk 22 + 𝛿kxk 22

I can be solved as a least-squares problem

2
√A b
   
minimize x−
𝛿I 0 2

with solution x★ = (AT A + 𝛿I) −1 AT b

Convex Optimization Boyd and Vandenberghe 6.12

 Optimal input design

I linear dynamical system (or convolution system) with impulse response h:

t
Õ
y(t) = h(𝜏)u(t − 𝜏), t = 0, 1, . . . , N
𝜏=0

I input design problem: multicriterion problem with 3 objectives

ÍN
– tracking error with desired output ydes : Jtrack = t=0 (y(t) − ydes (t)) 2
– input variation: Jder = N (u(t + 1) − u(t)) 2
Í −1
t=0
ÍN
– input magnitude: Jmag = t=0 u(t) 2
track desired output using a small and slowly varying input signal
I regularized least-squares formulation: minimize Jtrack + 𝛿Jder + 𝜂Jmag
– for fixed 𝛿, 𝜂, a least-squares problem in u(0), …, u(N)

Convex Optimization Boyd and Vandenberghe 6.13

 Example
I minimize Jtrack + 𝛿Jder + 𝜂Jmag
I (top) 𝛿 = 0, small 𝜂; (middle) 𝛿 = 0, larger 𝜂; (bottom) large 𝛿
5 1
0 0.5

y(t)
u(t)

0
−5 −0.5
−10 −1
0 50 100 150 200 0 50 100 150 200
t t
4 1
2 0.5

y(t)
u(t)

0 0
−2 −0.5
−4 −1
0 50 100 150 200 0 50 100 150 200
t t
4 1
2 0.5

y(t)
u(t)

0 0
−2 −0.5
−4 −1
0 50 100 150 200 0 50 100 150 200
t t
Convex Optimization Boyd and Vandenberghe 6.14
 Signal reconstruction

I bi-objective problem:

minimize (w.r.t. R2+ ) (k x̂ − xcor k 2 , 𝜙( x̂))

– x ∈ Rn is unknown signal
– xcor = x + v is (known) corrupted version of x, with additive noise v
– variable x̂ (reconstructed signal) is estimate of x
– 𝜙 : Rn → R is regularization function or smoothing objective
I examples:
– quadratic smoothing, 𝜙quad ( x̂) = n−1 ( x̂i+1 − x̂i ) 2
Í
i=1
Ín−1
– total variation smoothing, 𝜙tv ( x̂) = i=1 | x̂i+1 − x̂i |

Convex Optimization Boyd and Vandenberghe 6.15

 Quadratic smoothing example
0.5

0.5

x̂
0
x

0 −0.5
0 1000 2000 3000 4000
0.5
−0.5
0 1000 2000 3000 4000

x̂
0

−0.5
0.5
0 1000 2000 3000 4000
0.5
xcor

x̂
0

−0.5 −0.5
0 1000 2000 3000 4000 0 1000 2000 3000 4000
i i

three solutions on trade-off curve

original signal x and noisy signal xcor
k x̂ − xcor k 2 versus 𝜙quad ( x̂)

Convex Optimization Boyd and Vandenberghe 6.16

 Reconstructing a signal with sharp transitions
2

x̂i
0

2 −2
0 500 1000 1500 2000
1 2
x

x̂i
−1 0

−2
0 500 1000 1500 2000 −2
0 500 1000 1500 2000
2 2
1
xcor

x̂i
0 0
−1
−2
−2
0 500 1000 1500 2000 0 500 1000 1500 2000
i i
three solutions on trade-off curve
original signal x and noisy signal xcor
k x̂ − xcor k 2 versus 𝜙quad ( x̂)

I quadratic smoothing smooths out noise and sharp transitions in signal

Convex Optimization Boyd and Vandenberghe 6.17

 Total variation reconstruction
2

x̂
0

2 −2
0 500 1000 1500 2000
1 2
x

x̂
−1 0

−2
0 500 1000 1500 2000 −2
0 500 1000 1500 2000
2 2
1
xcor

x̂
0 0
−1
−2
−2
0 500 1000 1500 2000 0 500 1000 1500 2000
i i
three solutions on trade-off curve
original signal x and noisy signal xcor
k x̂ − xcor k 2 versus 𝜙tv ( x̂)

I total variation smoothing preserves sharp transitions in signal

Convex Optimization Boyd and Vandenberghe 6.18

 Outline

Norm and penalty approximation

Regularized approximation

Robust approximation

Convex Optimization Boyd and Vandenberghe 6.19

 Robust approximation

I minimize kAx − bk with uncertain A

I two approaches:
– stochastic: assume A is random, minimize E kAx − bk
– worst-case: set A of possible values of A, minimize supA∈ A kAx − bk

I tractable only in special cases (certain norms k · k, distributions, sets A)

Convex Optimization Boyd and Vandenberghe 6.20

 Example

12

A(u) = A0 + uA1 , u ∈ [−1, 1]

10
xnom
I xnom minimizes kA0 x − bk 22 8

I xstoch minimizes E kA(u)x − bk 22

r(u)
6
xstoch
with u uniform on [−1, 1] xwc
I xwc minimizes sup−1≤u≤1 kA(u)x − bk 22 4

2
plot shows r(u) = kA(u)x − bk 2 versus u
0
−2 −1 0 1 2
u

Convex Optimization Boyd and Vandenberghe 6.21

 Stochastic robust least-squares

I A = Ā + U, U random, E U = 0, E U T U = P
I stochastic least-squares problem: minimize E k( Ā + U)x − bk 22
I explicit expression for objective:

E kAx − bk 22 = E k Āx − b + Uxk 22

= k Āx − bk 22 + E xT U T Ux
= k Āx − bk 22 + xT Px

I hence, robust least-squares problem is equivalent to: minimize k Āx − bk 22 + kP1/2 xk 22

I for P = 𝛿I, get Tikhonov regularized problem: minimize k Āx − bk 22 + 𝛿kxk 22

Convex Optimization Boyd and Vandenberghe 6.22

 Worst-case robust least-squares
I A = {Ā + u1 A1 + · · · + up Ap | kuk 2 ≤ 1} (an ellipsoid in Rm×n )
I worst-case robust least-squares problem is
minimize supA∈ A kAx − bk 22 = sup kuk 2 ≤1 kP(x)u + q(x)k 22
where P(x) = A1 x A2 x · · · Ap x , q(x) = Āx − b
 
I from book appendix B, strong duality holds between the following problems
maximize kPu + qk 22 minimize t+𝜆
subject to kuk 22 ≤ 1  I P q 
subject to  PT 0 0
 
𝜆I
 qT 0 t


 
I hence, robust least-squares problem is equivalent to SDP
minimize t+𝜆
 I P(x) q(x) 
subject to  P(x) T 0 0
 
𝜆I
 q(x) T 0 t


 
Convex Optimization Boyd and Vandenberghe 6.23
 Example
I r(u) = k(A0 + u1 A1 + u2 A2 )x − bk 2 , u uniform on unit disk
I three choices of x:
– xls minimizes kA0 x − bk 2
– xtik minimizes kA0 x − bk 22 + 𝛿kxk 22 (Tikhonov solution)
– xrls minimizes supA∈ A kAx − bk 22 + kxk 22
0.25

0.2 xrls
frequency
0.15

0.1
xtik
0.05 xls

0
0 1 2 3 4 5
r(u)

Convex Optimization Boyd and Vandenberghe 6.24

7. Statistical estimation
 Outline

Maximum likelihood estimation

Hypothesis testing

Experiment design

Convex Optimization Boyd and Vandenberghe 7.1

 Maximum likelihood estimation

I parametric distribution estimation: choose from a family of densities px (y), indexed by

a parameter x (often denoted 𝜃)
I we take px (y) = 0 for invalid values of x
I px (y), as a function of x, is called likelihood function
I l(x) = log px (y), as a function of x, is called log-likelihood function

I maximum likelihood estimation (MLE): choose x to maximize px (y) (or l(x))

I a convex optimization problem if log px (y) is concave in x for fixed y
I not the same as log px (y) concave in y for fixed x, i.e., px (y) is a family of log-concave
densities

Convex Optimization Boyd and Vandenberghe 7.2

 Linear measurements with IID noise

linear measurement model

yi = aTi x + vi , i = 1, . . . , m

I x ∈ Rn is vector of unknown parameters

I vi is IID measurement noise, with density p(z)
I yi is measurement: y ∈ Rm has density px (y) = m T
i=1 p(yi − ai x)
Î

maximum likelihood estimate: any solution x of

maximize l(x) = m T
i=1 log p(yi − ai x)
Í

(y is observed value)

Convex Optimization Boyd and Vandenberghe 7.3

 Examples
2 /(2𝜎 2 )
I Gaussian noise N (0, 𝜎 2 ): p(z) = (2𝜋𝜎 2 ) −1/2 e−z ,
m
m 1 Õ T
l(x) = − log(2𝜋𝜎 2 ) − (a x − yi ) 2
2 2𝜎 2 i=1 i

ML estimate is least-squares solution

I Laplacian noise: p(z) = (1/(2a))e− |z|/a ,
m
1Õ T
l(x) = −m log(2a) − |a x − yi |
a i=1 i

ML estimate is ℓ1 -norm solution

I uniform noise on [−a, a]:
|aTi x − yi | ≤ a, i = 1, . . . , m

−m log(2a)
l(x) =
−∞ otherwise
ML estimate is any x with |aTi x − yi | ≤ a
Convex Optimization Boyd and Vandenberghe 7.4
 Logistic regression
I random variable y ∈ {0, 1} with distribution

exp(aT u + b)
p = prob(y = 1) =
1 + exp(aT u + b)
I a, b are parameters; u ∈ Rn are (observable) explanatory variables
I estimation problem: estimate a, b from m observations (ui , yi )
I log-likelihood function (for y1 = · · · = yk = 1, yk+1 = · · · = ym = 0):

k m
exp(aT ui + b) Ö
!
Ö 1
l(a, b) = log
i=1
1 + exp(a ui + b) i=k+1 1 + exp(aT ui + b)
T

k m
(aT ui + b) − log(1 + exp(aT ui + b))
Õ Õ
=
i=1 i=1

concave in a, b
Convex Optimization Boyd and Vandenberghe 7.5
 Example

0.8

prob(y = 1)
0.6

0.4

0.2

0
0 2 4 6 8 10
u

I n = 1, m = 50 measurements; circles show points (ui , yi )

I solid curve is ML estimate of p = exp(au + b)/(1 + exp(au + b))

Convex Optimization Boyd and Vandenberghe 7.6

 Gaussian covariance estimation

I fit Gaussian distribution N (0, Σ) to observed data y1 , . . . , yN

I log-likelihood is

N
1Õ
−2𝜋n − log det Σ − yT Σ −1 y

l(Σ) =
2 k=1
N  
= −2𝜋n − log det Σ − tr Σ −1 Y
2
ÍN
with Y = (1/N) k=1 yk yk ,
T the empirical covariance
I l is not concave in Σ (the log det Σ term has the wrong sign)
I with no constraints or regularization, MLE is empirical covariance Σml = Y

Convex Optimization Boyd and Vandenberghe 7.7

 Change of variables

I change variables to S = Σ −1
I recover original parameter via Σ = S −1
I S is the natural parameter in an exponential family description of a Gaussian

I in terms of S, log-likelihood is

N
l(S) = (−2𝜋n + log det S − tr SY)
2
which is concave
I (a similar trick can be used to handle nonzero mean)

Convex Optimization Boyd and Vandenberghe 7.8

 Fitting a sparse inverse covariance

I S is the precision matrix of the Gaussian

I Sij = 0 means that yi and yj are independent, conditioned on yk , k ≠ i, j
I sparse S means
– many pairs of components are conditionally independent, given the others
– y is described by a sparse (Gaussian) Bayes network

I to fit data with S sparse, minimize convex function

− log det S + tr SY + 𝜆
Õ
|Sij |
i≠j

over S ∈ Sn , with hyper-parameter 𝜆 ≥ 0

Convex Optimization Boyd and Vandenberghe 7.9

 Example

I example with n = 4, N = 10 samples generated from a sparse Strue

 1 0 0.5 0 
0 1 0 0.1 

Strue

= 
0.5 0 1 0.3 
0 0.1 0.3 1 



I empirical and sparse estimate values of Σ −1 (with 𝜆 = 0.2)

 3 0.8 3.3 1.2   0.9 0 0.6 0 

0.8 1.2 1.2 0.9 0 0.7 0 0.1
   
Y −1 Ŝ = 
   
= 
3.2 1.2 4.6 2.1 0.6 0 1.1 0.2
, .
 
1.2 0.9 2.1 2.7 0 0.1 0.2 1.2
   
   
   
2 2
I estimation errors: Strue − Y −1 F
= 49.8, Strue − Ŝ F
= 0.2

Convex Optimization Boyd and Vandenberghe 7.10

 Outline

Maximum likelihood estimation

Hypothesis testing

Experiment design

Convex Optimization Boyd and Vandenberghe 7.11

 (Binary) hypothesis testing

detection (hypothesis testing) problem

given observation of a random variable X ∈ {1, . . . , n}, choose between:
I hypothesis 1: X was generated by distribution p = (p1 , . . . , pn )
I hypothesis 2: X was generated by distribution q = (q1 , . . . , qn )

randomized detector
I a nonnegative matrix T ∈ R2×n , with 1T T = 1T
I if we observe X = k, we choose hypothesis 1 with probability t1k , hypothesis 2 with
probability t2k
I if all elements of T are 0 or 1, it is called a deterministic detector

Convex Optimization Boyd and Vandenberghe 7.12

 Detection probability matrix

1 − Pfp Pfn
 
D= Tp Tq
 
=
Pfp 1 − Pfn

I Pfp is probability of selecting hypothesis 2 if X is generated by distribution 1 (false positive)

I Pfn is probability of selecting hypothesis 1 if X is generated by distribution 2 (false
negative)

I multi-objective formulation of detector design

minimize (w.r.t. R2+ ) (Pfp , Pfn ) = ((Tp)2 , (Tq)1 )

subject to t1k + t2k = 1, k = 1, . . . , n
tik ≥ 0, i = 1, 2, k = 1, . . . , n

variable T ∈ R2×n

Convex Optimization Boyd and Vandenberghe 7.13

 Scalarization

I scalarize with weight 𝜆 > 0 to obtain

minimize (Tp)2 + 𝜆(Tq)1

subject to t1k + t2k = 1, tik ≥ 0, i = 1, 2, k = 1, . . . , n

I an LP with a simple analytical solution

(1, 0) pk ≥ 𝜆qk

(t1k , t2k ) =
(0, 1) pk < 𝜆qk

I a deterministic detector, given by a likelihood ratio test

I if pk = 𝜆qk for some k, any value 0 ≤ t1k ≤ 1, t1k = 1 − t2k is optimal (i.e., Pareto-optimal
detectors include non-deterministic detectors)

Convex Optimization Boyd and Vandenberghe 7.14

 Minimax detector

I minimize maximum of false positive and false negative probabilities

minimize max{Pfp , Pfn } = max{(Tp)2 , (Tq)1 }

subject to t1k + t2k = 1, tik ≥ 0, i = 1, 2, k = 1, . . . , n

I an LP; solution is usually not deterministic

Convex Optimization Boyd and Vandenberghe 7.15

 Example

0.8

0.6
0.70 0.10

Pfn
 
0.20 0.10
 
p q = 
   
0.4
0.05 0.70


0.05 0.10 1
 
 
0.2
  2 4
3
0
0 0.2 0.4 0.6 0.8 1
Pfp

solutions 1, 2, 3 (and endpoints) are deterministic; 4 is minimax detector

Convex Optimization Boyd and Vandenberghe 7.16

 Outline

Maximum likelihood estimation

Hypothesis testing

Experiment design

Convex Optimization Boyd and Vandenberghe 7.17

 Experiment design
I m linear measurements yi = aTi x + wi , i = 1, . . . , m of unknown x ∈ Rn
I measurement errors wi are IID N (0, 1)
I ML (least-squares) estimate is

m m
! −1
ai aTi
Õ Õ
x̂ = yi ai
i=1 i=1

I error e = x̂ − x has zero mean and covariance

m
! −1
E = E eeT = ai aTi
Õ

i=1

I confidence ellipsoids are given by {x | (x − x̂) T E −1 (x − x̂) ≤ 𝛽}

I experiment design: choose ai ∈ {v1 , . . . , vp } (set of possible test vectors) to make E
‘small’
Convex Optimization Boyd and Vandenberghe 7.18
 Vector optimization formulation

I formulate as vector optimization problem

 −1
p
minimize (w.r.t. Sn+ ) E =
Í
m v v T
k=1 k k k
subject to mk ≥ 0, m1 + · · · + mp = m
mk ∈ Z

I variables are mk , the number of vectors ai equal to vk

I difficult in general, due to integer constraint
I common scalarizations: minimize log det E, tr E, 𝜆max (E), …

Convex Optimization Boyd and Vandenberghe 7.19

 Relaxed experiment design

I assume m  p, use 𝜆k = mk /m as (continuous) real variable

 −1
p
minimize (w.r.t. Sn+ ) E = (1/m) k=1 𝜆k vk vkT
Í

subject to 𝜆  0, 1T 𝜆 = 1

I a convex relaxation, since we ignore constraint that m𝜆k ∈ Z

I optimal value is lower bound on optimal value of (integer) experiment design problem
I simple rounding of 𝜆k m gives heuristic for experiment design problem

Convex Optimization Boyd and Vandenberghe 7.20

 D-optimal design

I scalarize via log determinant

 −1
p
Í
minimize log det 𝜆 v vT
k=1 k k k
subject to 𝜆  0, 1T 𝜆 = 1

I interpretation: minimizes volume of confidence ellipsoids

Convex Optimization Boyd and Vandenberghe 7.21

 Dual of D-optimal experiment design problem

dual problem
maximize log det W + n log n
subject to vkT Wvk ≤ 1, k = 1, . . . , p
interpretation: {x | xT Wx ≤ 1} is minimum volume ellipsoid centered at origin, that includes all
test vectors vk

complementary slackness: for 𝜆, W primal and dual optimal

𝜆k (1 − vkT Wvk ) = 0, k = 1, . . . , p

optimal experiment uses vectors vk on boundary of ellipsoid defined by W

Convex Optimization Boyd and Vandenberghe 7.22

 Example

(p = 20)
𝜆 1 = 0.5

𝜆 2 = 0.5

design uses two vectors, on boundary of ellipse defined by optimal W

Convex Optimization Boyd and Vandenberghe 7.23

 Derivation of dual
first reformulate primal problem with new variable X:

minimize ÍpX
log det −1

subject to X = k=1 𝜆k vk vkT , 𝜆  0, 1T 𝜆 = 1

p
!!
+ tr Z X − 𝜆k vk vkT − zT 𝜆 + 𝜈(1T 𝜆 − 1)
Õ
L(X, 𝜆, Z, z, 𝜈) = log det X −1

k=1

I minimize over X by setting gradient to zero: −X −1 + Z = 0

I minimum over 𝜆k is −∞ unless −vkT Zvk − zk + 𝜈 = 0
dual problem
maximize n + log det Z − 𝜈
subject to vkT Zvk ≤ 𝜈, k = 1, . . . , p
change variable W = Z/𝜈, and optimize over 𝜈 to get dual of slide 7.21

Convex Optimization Boyd and Vandenberghe 7.24

8. Geometric problems
 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.1

 Minimum volume ellipsoid around a set
I Löwner-John ellipsoid of a set C: minimum volume ellipsoid E with C ⊆ E
I parametrize E as E = {v | kAv + bk 2 ≤ 1}; can assume A ∈ Sn++
I vol E is proportional to det A−1 ; to find Löwner-John ellipsoid, solve problem

minimize (over A, b) log det A−1

subject to supv∈C kAv + bk 2 ≤ 1

convex, but evaluating the constraint can be hard (for general C)

I finite set C = {x1 , . . . , xm }:

minimize (over A, b) log det A−1

subject to kAxi + bk 2 ≤ 1, i = 1, . . . , m

also gives Löwner-John ellipsoid for polyhedron conv{x1 , . . . , xm }

Convex Optimization Boyd and Vandenberghe 8.2

 Maximum volume inscribed ellipsoid
I maximum volume ellipsoid E with E ⊆ C, C ⊆ Rn convex
I parametrize E as E = {Bu + d | kuk 2 ≤ 1}; can assume B ∈ Sn++
I vol E is proportional to det B; can find E by solving

maximize log det B

subject to sup kuk 2 ≤1 IC (Bu + d) ≤ 0

(where IC (x) = 0 for x ∈ C and IC (x) = ∞ for x ∉ C)

convex, but evaluating the constraint can be hard (for general C)
I polyhedron {x | aTi x ≤ bi , i = 1, . . . , m}:

maximize log det B

subject to kBai k 2 + aTi d ≤ bi , i = 1, . . . , m

(constraint follows from sup kuk 2 ≤1 aTi (Bu + d) = kBai k 2 + aTi d)

Convex Optimization Boyd and Vandenberghe 8.3
 Efficiency of ellipsoidal approximations
I C ⊆ Rn convex, bounded, with nonempty interior
I Löwner-John ellipsoid, shrunk by a factor n (around its center), lies inside C
I maximum volume inscribed ellipsoid, expanded by a factor n (around its center) covers C

I example (for polyhedra in R2 )

√
I factor n can be improved to n if C is symmetric

Convex Optimization Boyd and Vandenberghe 8.4

 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.5

 Centering
I many possible definitions of ‘center’ of a convex set C

I Chebyshev center: center of largest inscribed ball

– for polyhedron, can be found via linear programming

I center of maximum volume inscribed ellipsoid

– invariant under affine coordinate transformations

xcheb xmve

Convex Optimization Boyd and Vandenberghe 8.6

 Analytic center of a set of inequalities

I the analytic center of set of convex inequalities and linear equations

fi (x) ≤ 0, i = 1, . . . , m, Fx = g

is defined as solution of
minimize − m
Í
i=1 log(−fi (x))
subject to Fx = g
I objective is called the log-barrier for the inequalities

I (we’ll see later) analytic center more easily computed than MVE or Chebyshev center
I two sets of inequalities can describe the same set, but have different analytic centers

Convex Optimization Boyd and Vandenberghe 8.7

 Analytic center of linear inequalities
I aTi x ≤ bi , i = 1, . . . , m
I xac minimizes 𝜙(x) = − m T
i=1 log(bi − ai x)
Í
I dashed lines are level curves of 𝜙

xac

Convex Optimization Boyd and Vandenberghe 8.8

 Inner and outer ellipsoids from analytic center

I we have
Einner ⊆ {x | aTi x ≤ bi , i = 1, . . . , m} ⊆ Eouter
where

Einner = {x | (x − xac ) T ∇2 𝜙(xac )(x − xac ) ≤ 1}

Eouter {x | (x − xac ) T ∇2 𝜙(xac )(x − xac ) ≤ m(m − 1)}
=

I ellipsoid expansion/shrinkage factor is m(m − 1)

p

(cf. n for Löwner-John or max volume inscribed ellpsoids)

Convex Optimization Boyd and Vandenberghe 8.9

 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.10

 Linear discrimination
I separate two sets of points {x1 , . . . , xN }, {y1 , . . . , yM } by a hyperplane
I i.e., find a ∈ Rn , b ∈ R with

aT xi + b > 0, i = 1, . . . , N, aT yi + b < 0, i = 1, . . . , M
I homogeneous in a, b, hence equivalent to

aT xi + b ≥ 1, i = 1, . . . , N, aT yi + b ≤ −1, i = 1, . . . , M
a set of linear inequalities in a, b, i.e., an LP feasibility problem

Convex Optimization Boyd and Vandenberghe 8.11

 Robust linear discrimination

(Euclidean) distance between hyperplanes

H1 = {z | aT z + b = 1}
H2 = {z | aT z + b = −1}

is dist(H1 , H2 ) = 2/kak 2

to separate two sets of points by maximum margin,

minimize (1/2)kak 22
subject to aT xi + b ≥ 1, i = 1, . . . , N (2)
aT yi + b ≤ −1, i = 1, . . . , M

a QP in a, b

Convex Optimization Boyd and Vandenberghe 8.12

 Approximate linear separation of non-separable sets

minimize 1T u + 1T v
subject to aT xi + b ≥ 1 − ui , i = 1, . . . , N, aT yi + b ≤ −1 + vi , i = 1, . . . , M
u  0, v  0

I an LP in a, b, u, v
I at optimum, ui = max{0, 1 − aT xi − b}, vi = max{0, 1 + aT yi + b}
I equivalent to minimizing the sum of violations of the original inequalities

Convex Optimization Boyd and Vandenberghe 8.13

 Support vector classifier

minimize kak 2 + 𝛾(1T u + 1T v)

subject to aT xi + b ≥ 1 − ui , i = 1, . . . , N
aT yi + b ≤ −1 + vi , i = 1, . . . , M
u  0, v  0
produces point on trade-off curve between inverse of margin 2/kak 2 and classification error,
measured by total slack 1T u + 1T v

example on previous slide, with 𝛾 = 0.1:

Convex Optimization Boyd and Vandenberghe 8.14

 Nonlinear discrimination

I separate two sets of points by a nonlinear function f : find f : Rn → R with

f (xi ) > 0, i = 1, . . . , N, f (yi ) < 0, i = 1, . . . , M

I choose a linearly parametrized family of functions f (z) = 𝜃 T F (z)
– 𝜃 ∈ Rk is parameter
– F = (F1 , . . . , Fk ) : Rn → Rk are basis functions

I solve a set of linear inequalities in 𝜃:

𝜃 T F (xi ) ≥ 1, i = 1, . . . , N, 𝜃 T F (yi ) ≤ −1, i = 1, . . . , M

Convex Optimization Boyd and Vandenberghe 8.15

 Examples

I quadratic discrimination: f (z) = zT Pz + qT z + r, 𝜃 = (P, q, r)

I solve LP feasibility problem with variables P ∈ Sn , q ∈ Rn , r ∈ R

xiT Pxi + qT xi + r ≥ 1, yiT Pyi + qT yi + r ≤ −1

I can add additional constraints (e.g., P  −I to separate by an ellipsoid)

I polynomial discrimination: F (z) are all monomials up to a given degree d

I e.g., for n = 2, d = 3

F (z) = (1, z1 , z2 , z12 , z1 z2 , z22 , z13 , z12 z2 , z1 z22 , z23 )

Convex Optimization Boyd and Vandenberghe 8.16

 Example

separation by ellipsoid separation by 4th degree polynomial

Convex Optimization Boyd and Vandenberghe 8.17

 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.18

 Placement and facility location

I N points with coordinates xi ∈ R2 (or R3 )

I some positions xi are given; the other xi ’s are variables

I for each pair of points, a cost function fij (xi , xj )

I placement problem: minimize i≠j fij (xi , xj )

Í

I interpretations
– points are locations of plants or warehouses; fij is transportation cost between facilities i and
j
– points are locations of cells in an integrated circuit; fij represents wirelength

Convex Optimization Boyd and Vandenberghe 8.19

 Example
I minimize (i,j) ∈ E h(kxi − xj k 2 ), with 6 free points, 27 edges
Í

I optimal placements for h(z) = z, h(z) = z2 , h(z) = z4

1 1 1

0 0 0

−1 −1 −1
−1 0 1 −1 0 1 −1 0 1
I histograms of edge lengths kxi − xj k 2 , (i, ) ∈ E
4 4 6
5
3 3 4
2 2 3
2
1 1
1
00 0.5 1 1.5 2 00 0.5 1 1.5 00 0.5 1 1.5
Convex Optimization Boyd and Vandenberghe 8.20
B. Numerical linear algebra background
 Outline

Flop counts and BLAS

Solving systems of linear equations

Block elimination

Convex Optimization Boyd and Vandenberghe B.1

 Flop count

I flop (floating-point operation): one addition, subtraction, multiplication, or division of two

floating-point numbers
I to estimate complexity of an algorithm
– express number of flops as a (polynomial) function of the problem dimensions
– simplify by keeping only the leading terms
I not an accurate predictor of computation time on modern computers, but useful as a
rough estimate of complexity

Convex Optimization Boyd and Vandenberghe B.2

 Basic linear algebra subroutines (BLAS)

vector-vector operations (x, y ∈ Rn ) (BLAS level 1)

I inner product xT y: 2n − 1 flops (≈ 2n, O(n))
I sum x + y, scalar multiplication 𝛼x: n flops

matrix-vector product y = Ax with A ∈ Rm×n (BLAS level 2)

I m(2n − 1) flops (≈ 2mn)
I 2N if A is sparse with N nonzero elements
I 2p(n + m) if A is given as A = UV T , U ∈ Rm×p , V ∈ Rn×p

matrix-matrix product C = AB with A ∈ Rm×n , B ∈ Rn×p (BLAS level 3)

I mp(2n − 1) flops (≈ 2mnp)
I less if A and/or B are sparse
I (1/2)m(m + 1)(2n − 1) ≈ m2 n if m = p and C symmetric

Convex Optimization Boyd and Vandenberghe B.3

 BLAS on modern computers

I there are good implementations of BLAS and variants (e.g., for sparse matrices)
I CPU single thread speeds typically 1–10 Gflops/s (109 flops/sec)
I CPU multi threaded speeds typically 10–100 Gflops/s
I GPU speeds typically 100 Gflops/s–1 Tflops/s (1012 flops/sec)

Convex Optimization Boyd and Vandenberghe B.4

 Outline

Flop counts and BLAS

Solving systems of linear equations

Block elimination

Convex Optimization Boyd and Vandenberghe B.5

 Complexity of solving linear equations

I A ∈ Rn×n is invertible, b ∈ Rn

I solution of Ax = b is x = A−1 b

I solving Ax = b, i.e., computing x = A−1 b

– almost never done by computing A −1 , then multiplying by b
– for general methods, O(n3 )
– (much) less if A is structured (banded, sparse, Toeplitz, …)
– e.g., for A with half-bandwidth k (Aij = 0 for |i − j| > k, O(k 2 n)

I it’s super useful to recognize matrix structure that can be exploited in solving Ax = b

Convex Optimization Boyd and Vandenberghe B.6

 Linear equations that are easy to solve

I diagonal matrices: n flops; x = A−1 b = (b1 /a11 , . . . , bn /ann )

I lower triangular: n2 flops via forward substitution

x1 := b1 /a11
x2 := (b2 − a21 x1 )/a22
x3 := (b3 − a31 x1 − a32 x2 )/a33
..
.
xn := (bn − an1 x1 − an2 x2 − · · · − an,n−1 xn−1 )/ann

I upper triangular: n2 flops via backward substitution

Convex Optimization Boyd and Vandenberghe B.7

 Linear equations that are easy to solve
I orthogonal matrices (A−1 = AT ):
– 2n2 flops to compute x = AT b for general A
– less with structure, e.g., if A = I − 2uuT with kuk 2 = 1, we can compute
x = AT b = b − 2(uT b)u in 4n flops

I permutation matrices: for 𝜋 = (𝜋1 , 𝜋2 , . . . , 𝜋n ) a permutation of (1, 2, . . . , n)

1 j = 𝜋i

aij =
0 otherwise

– interpretation: Ax = (x 𝜋1 , . . . , x 𝜋n )
– satisfies A −1 = AT , hence cost of solving Ax = b is 0 flops
– example:
 0 1 0   0 0 1 
T
A=  0 0 1 , A = A =  1
−1 0 0
   
  
 1 0 0   0 1 0


   

Convex Optimization Boyd and Vandenberghe B.8

 Factor-solve method for solving Ax = b

I factor A as a product of simple matrices (usually 2–5):

A = A1 A2 · · · Ak

I e.g., Ai diagonal, upper or lower triangular, orthogonal, permutation, …

I compute x = A−1 b = Ak−1 · · · A2−1 A1−1 b by solving k ‘easy’ systems of equations

A1 x1 = b, A2 x2 = x1 , ... Ak x = xk−1

I cost of factorization step usually dominates cost of solve step

Convex Optimization Boyd and Vandenberghe B.9

 Solving equations with multiple righthand sides

I we wish to solve
Ax1 = b1 , Ax2 = b2 , ... Axm = bm

I cost: one factorization plus m solves

I called factorization caching

I when factorization cost dominates solve cost, we can solve a modest number of equations
at the same cost as one (!!)

Convex Optimization Boyd and Vandenberghe B.10

 LU factorization

I every nonsingular matrix A can be factored as A = PLU with P a permutation, L lower

triangular, U upper triangular

I factorization cost: (2/3)n3 flops

Solving linear equations by LU factorization.

given a set of linear equations Ax = b, with A nonsingular.
1. LU factorization. Factor A as A = PLU ((2/3)n3 flops).
2. Permutation. Solve Pz1 = b (0 flops).
3. Forward substitution. Solve Lz2 = z1 (n2 flops).
4. Backward substitution. Solve Ux = z2 (n2 flops).

I total cost: (2/3)n3 + 2n2 ≈ (2/3)n3 for large n

Convex Optimization Boyd and Vandenberghe B.11

 Sparse LU factorization

I for A sparse and invertible, factor as A = P1 LUP2

I adding permutation matrix P2 offers possibility of sparser L, U

I hence, less storage and cheaper factor and solve steps

I P1 and P2 chosen (heuristically) to yield sparse L, U

I choice of P1 and P2 depends on sparsity pattern and values of A

I cost is usually much less than (2/3)n3 ; exact value depends in a complicated way on n,
number of zeros in A, sparsity pattern

I often practical to solve very large sparse systems of equations

Convex Optimization Boyd and Vandenberghe B.12

 Cholesky factorization

I every positive definite A can be factored as A = LLT

I L is lower triangular with positive diagonal entries

I Cholesjy factorization cost: (1/3)n3 flops

Solving linear equations by Cholesky factorization.

given a set of linear equations Ax = b, with A ∈ Sn++ .
1. Cholesky factorization. Factor A as A = LLT ((1/3)n3 flops).
2. Forward substitution. Solve Lz1 = b (n2 flops).
3. Backward substitution. Solve LT x = z1 (n2 flops).

I total cost: (1/3)n3 + 2n2 ≈ (1/3)n3 for large n

Convex Optimization Boyd and Vandenberghe B.13

 Sparse Cholesky factorization

I for sparse positive define A, factor as A = PLLT PT

I adding permutation matrix P offers possibility of sparser L

I same as
– permuting rows and columns of A to get à = PT AP
– then finding Cholesky factorization of Ã

I P chosen (heuristically) to yield sparse L

I choice of P only depends on sparsity pattern of A (unlike sparse LU)

I cost is usually much less than (1/3)n3 ; exact value depends in a complicated way on n,
number of zeros in A, sparsity pattern

Convex Optimization Boyd and Vandenberghe B.14

 Example

I sparse A with upper arrow sparsity pattern

 ∗ ∗ ∗ ∗ ∗   ∗ 
∗ ∗ ∗ ∗
   
A =  L = 
   
∗ ∗  ∗ ∗ ∗ 
 
 ∗ ∗   ∗ ∗ ∗ ∗ 

 ∗ ∗ 


 ∗ ∗ ∗ ∗ ∗ 

L is full, with O(n2 ) nonzeros; solve cost is O(n2 )

I reverse order of entries (i.e., permute) to get lower arrow sparsity pattern
 ∗ ∗   ∗ 
∗ ∗  ∗
   
à =  L = 
  
 ∗ ∗  ∗ 
 
 ∗ ∗   ∗ 
 ∗ ∗ ∗ ∗ ∗   ∗ ∗ ∗ ∗ ∗ 
   
L is sparse with O(n) nonzeros; cost of solve is O(n)

Convex Optimization Boyd and Vandenberghe B.15

 LDLT factorization

I every nonsingular symmetric matrix A can be factored as

A = PLDLT PT

with P a permutation matrix, L lower triangular, D block diagonal with 1 × 1 or 2 × 2

diagonal blocks

I factorization cost: (1/3)n3

I cost of solving linear equations with symmetric A by LDLT factorization:

(1/3)n3 + 2n2 ≈ (1/3)n3 for large n

I for sparse A, can choose P to yield sparse L; cost  (1/3)n3

Convex Optimization Boyd and Vandenberghe B.16

 Outline

Flop counts and BLAS

Solving systems of linear equations

Block elimination

Convex Optimization Boyd and Vandenberghe B.17

 Equations with structured sub-blocks

I express Ax = b in blocks as

A11 A12 x1 b1
    
=
A21 A22 x2 b2

with x1 ∈ Rn1 , x2 ∈ Rn2 ; blocks Aij ∈ Rni ×nj

I assuming A11 is nonsingular, can eliminate x1 as

x1 = A11
−1
(b1 − A12 x2 )

I to compute x2 , solve
(A22 − A21 A11
−1
A12 )x2 = b2 − A21 A11
−1
b1
I S = A22 − A21 A11 12 is the Schur complement
−1 A

Convex Optimization Boyd and Vandenberghe B.18

 Block elimination method

Solving linear equations by block elimination.

given a nonsingular set of linear equations with A11 nonsingular.
1. Form A11
−1 A and A −1 b .
12 11 1
2. Form S = A22 − A21 A11−1 A and b̃ = b − A A −1 b .
12 2 21 11 1
3. Determine x2 by solving Sx2 = b̃.
4. Determine x1 by solving A11 x1 = b1 − A12 x2 .

dominant terms in flop count

I step 1: f + n2 s (f is cost of factoring A11 ; s is cost of solve step)
I step 2: 2n22 n1 (cost dominated by product of A21 and A11 −1 A )
12
I step 3: (2/3)n32
total: f + n2 s + 2n22 n1 + (2/3)n32

Convex Optimization Boyd and Vandenberghe B.19

 Examples

I for general A11 , f = (2/3)n31 , s = 2n21

#flops = (2/3)n31 + 2n21 n2 + 2n22 n1 + (2/3)n32 = (2/3)(n1 + n2 ) 3

so, no gain over standard method

I block elimination is useful for structured A11 (f  n31 )

I for example, A11 diagonal (f = 0, s = n1 ): #flops ≈ 2n22 n1 + (2/3)n32

Convex Optimization Boyd and Vandenberghe B.20

 Structured plus low rank matrices
I we wish to solve (A + BC)x = b, A ∈ Rn×n , B ∈ Rn×p , C ∈ Rp×n
I assume A has structure (i.e., Ax = b easy to solve)
I first uneliminate to write as block equations with new variable y

A B x b
    
=
C −I y 0

I now apply block elimination: solve

(I + CA−1 B)y = CA−1 b,

then solve Ax = b − By
I this proves the matrix inversion lemma: if A and A + BC are nonsingular,

(A + BC) −1 = A−1 − A−1 B(I + CA−1 B) −1 CA−1

Convex Optimization Boyd and Vandenberghe B.21

 Example: Solving diagonal plus low rank equations

I with A diagonal, p  n, A + BC is called diagonal plus low rank

I for covariance matrices, called a factor model

I method 1: form D = A + BC, then solve Dx = b

– storage n2
– solve cost (2/3)n3 + 2pn2 (cubic in n)

I method 2: solve (I + CA−1 B)y = CA−1 b, then compute x = A−1 b − A−1 By

– storage O(np)
– solve cost 2p2 n + (2/3)p3 (linear in n)

Convex Optimization Boyd and Vandenberghe B.22

9. Unconstrained minimization
 Outline

Terminology and assumptions

Gradient descent method

Steepest descent method

Newton’s method

Self-concordant functions

Implementation

Convex Optimization Boyd and Vandenberghe 9.1

 Unconstrained minimization

I unconstrained minimization problem

minimize f (x)
I we assume
– f convex, twice continuously differentiable (hence dom f open)
– optimal value p★ = inf x f (x) is attained at x★ (not necessarily unique)

I optimality condition is ∇f (x) = 0

I minimizing f is the same as solving ∇f (x) = 0

I a set of n equations with n unknowns

Convex Optimization Boyd and Vandenberghe 9.2

 Quadratic functions

I convex quadratic: f (x) = (1/2)xT Px + qT x + r, P  0

I we can solve exactly via linear equations

∇f (x) = Px + q = 0

I much more on this special case later

Convex Optimization Boyd and Vandenberghe 9.3

 Iterative methods

I for most non-quadratic functions, we use iterative methods

I these produce a sequence of points x (k) ∈ dom f , k = 0, 1, . . .

I x (0) is the initial point or starting point
I x (k) is the kth iterate
I we hope that the method converges, i.e.,

f (x (k) ) → p★, ∇f (x (k) ) → 0

Convex Optimization Boyd and Vandenberghe 9.4

 Initial point and sublevel set

I algorithms in this chapter require a starting point x (0) such that

– x (0) ∈ dom f
– sublevel set S = {x | f (x) ≤ f (x (0) )} is closed
I 2nd condition is hard to verify, except when all sublevel sets are closed
– equivalent to condition that epi f is closed
– true if dom f = Rn
– true if f (x) → ∞ as x → bd dom f

I examples of differentiable functions with closed sublevel sets:

m m
!
T
log(bi − aTi x)
Õ Õ
f (x) = log exp(ai x + bi ) , f (x) = −
i=1 i=1

Convex Optimization Boyd and Vandenberghe 9.5

 Strong convexity and implications
I f is strongly convex on S if there exists an m > 0 such that

∇2 f (x)  mI for all x ∈ S

I same as f (x) − (m/2)kxk 22 is convex

I if f is strongly convex, for x, y ∈ S,
m
f (y) ≥ f (x) + ∇f (x) T (y − x) + kx − yk 22
2
I hence, S is bounded
I we conclude p★ > −∞, and for x ∈ S,

1
f (x) − p★ ≤ k∇f (x)k 22
2m
I useful as stopping criterion (if you know m, which usually you do not)
Convex Optimization Boyd and Vandenberghe 9.6
 Outline

Terminology and assumptions

Gradient descent method

Steepest descent method

Newton’s method

Self-concordant functions

Implementation

Convex Optimization Boyd and Vandenberghe 9.7

 Descent methods

I descent methods generate iterates as

x (k+1) = x (k) + t (k) Δx (k)

with f (x (k+1) ) < f (x (k) ) (hence the name)

I other notations: x+ = x + tΔx, x := x + tΔx
I Δx (k) is the step, or search direction
I t (k) > 0 is the step size, or step length

I from convexity, f (x+ ) < f (x) implies ∇f (x) T Δx < 0

I this means Δx is a descent direction

Convex Optimization Boyd and Vandenberghe 9.8

 Generic descent method

General descent method.

given a starting point x ∈ dom f .
repeat
1. Determine a descent direction Δx.
2. Line search. Choose a step size t > 0.
3. Update. x := x + tΔx.
until stopping criterion is satisfied.

Convex Optimization Boyd and Vandenberghe 9.9

 Line search types

I exact line search: t = argmint>0 f (x + tΔx)

I backtracking line search (with parameters 𝛼 ∈ (0, 1/2), 𝛽 ∈ (0, 1))
– starting at t = 1, repeat t := 𝛽t until f (x + tΔx) < f (x) + 𝛼t∇f (x) T Δx

I graphical interpretation: reduce t (i.e., backtrack) until t ≤ t0

f (x + tΔx)

f (x) + t∇f (x) T Δx f (x) + 𝛼t∇f (x) T Δx

t
t=0 t0

Convex Optimization Boyd and Vandenberghe 9.10

 Gradient descent method
I general descent method with Δx = −∇f (x)

given a starting point x ∈ dom f .

repeat
1. Δx := −∇f (x).
2. Line search. Choose step size t via exact or backtracking line search.
3. Update. x := x + tΔx.
until stopping criterion is satisfied.

I stopping criterion usually of the form k∇f (x)k 2 ≤ 𝜖

I convergence result: for strongly convex f ,

f (x (k) ) − p★ ≤ ck (f (x (0) ) − p★)

c ∈ (0, 1) depends on m, x (0) , line search type

I very simple, but can be very slow
Convex Optimization Boyd and Vandenberghe 9.11
 Example: Quadratic function on R2

I take f (x) = (1/2)(x12 + 𝛾x22 ), with 𝛾 > 0

I with exact line search, starting at x (0) = (𝛾, 1):
k k
𝛾−1 𝛾−1
 
x1(k) = 𝛾 , x2(k) = −
𝛾+1 𝛾+1

4
x (0)
– very slow if 𝛾  1 or 𝛾  1

x2
0
– example for 𝛾 = 10 at right x (1)
– called zig-zagging −4
−10 0 10
x1

Convex Optimization Boyd and Vandenberghe 9.12

 Example: Nonquadratic function on R2

I f (x1 , x2 ) = ex1 +3x2 −0.1 + ex1 −3x2 −0.1 + e−x1 −0.1

x (0) x (0)

x (2)
x (1)

x (1)

backtracking line search exact line search

Convex Optimization Boyd and Vandenberghe 9.13

 Example: A problem in R100
I f (x) = cT x − 500 T
i=1 log(bi − ai x)
Í

I linear convergence, i.e., a straight line on a semilog plot

104

f (x (k) ) − p★ 102

100 exact line search

10 −2
backtracking line search
10 −4
0 50 100 150 200
k

Convex Optimization Boyd and Vandenberghe 9.14

 Outline

Terminology and assumptions

Gradient descent method

Steepest descent method

Newton’s method

Self-concordant functions

Implementation

Convex Optimization Boyd and Vandenberghe 9.15

 Steepest descent method

I normalized steepest descent direction (at x, for norm k · k):

Δxnsd = argmin{∇f (x) T v | kvk = 1}

I interpretation: for small v, f (x + v) ≈ f (x) + ∇f (x) T v;

I direction Δxnsd is unit-norm step with most negative directional derivative

I (unnormalized) steepest descent direction: Δxsd = k∇f (x)k ∗ Δxnsd

I satisfies ∇f (x) T Δxsd = −k∇f (x)k 2∗

I steepest descent method

– general descent method with Δx = Δxsd
– convergence properties similar to gradient descent

Convex Optimization Boyd and Vandenberghe 9.16

 Examples

I Euclidean norm: Δxsd = −∇f (x)

I quadratic norm kxk P = (xT Px) 1/2 (P ∈ Sn++ ): Δxsd = −P −1 ∇f (x)
I ℓ1 -norm: Δxsd = −(𝜕f (x)/𝜕xi )ei , where |𝜕f (x)/𝜕xi | = k∇f (x)k ∞
I unit balls, normalized steepest descent directions for quadratic norm and ℓ1 -norm:

−∇f (x)

−∇f (x)
Δxnsd Δxnsd

Convex Optimization Boyd and Vandenberghe 9.17

 Choice of norm for steepest descent

x (0)
x (0)
x (2)
x (1) x (2)
x (1)

I steepest descent with backtracking line search for two quadratic norms
I ellipses show {x | kx − x (k) k P = 1}
I interpretation of steepest descent with quadratic norm k · k P : gradient descent after
change of variables x̄ = P1/2 x
I shows choice of P has strong effect on speed of convergence

Convex Optimization Boyd and Vandenberghe 9.18

 Outline

Terminology and assumptions

Gradient descent method

Steepest descent method

Newton’s method

Self-concordant functions

Implementation

Convex Optimization Boyd and Vandenberghe 9.19

 Newton step

I Newton step is Δxnt = −∇2 f (x) −1 ∇f (x)

I interpretation: x + Δxnt minimizes second order approximation

1
f (x + v) = f (x) + ∇f (x) T v + vT ∇2 f (x)v
2
b

b
f

(x, f (x))

(x + Δxnt , f (x + Δxnt )) f

Convex Optimization Boyd and Vandenberghe 9.20

 Another intrepretation

I x + Δxnt solves linearized optimality condition

f (x + v) = ∇f (x) + ∇2 f (x)v = 0
∇f (x + v) ≈ ∇b

b
f′

f′
(x + Δxnt , f ′ (x + Δxnt ))
(x, f ′ (x))

Convex Optimization Boyd and Vandenberghe 9.21

 And one more interpretation

1/2
I Δxnt is steepest descent direction at x in local Hessian norm kuk ∇2 f (x) = uT ∇2 f (x)u

x + Δxnsd
x + Δxnt

I dashed lines are contour lines of f ; ellipse is {x + v | vT ∇2 f (x)v = 1}

I arrow shows −∇f (x)

Convex Optimization Boyd and Vandenberghe 9.22

 Newton decrement

1/2
I Newton decrement is 𝜆(x) = ∇f (x) T ∇2 f (x) −1 ∇f (x)
I a measure of the proximity of x to x★
I gives an estimate of f (x) − p★, using quadratic approximation b
f:
1
f (y) = 𝜆(x) 2
f (x) − inf b
y 2
I equal to the norm of the Newton step in the quadratic Hessian norm
 1/2
T 2

𝜆(x) = Δxnt ∇ f (x)Δxnt

I directional derivative in the Newton direction: ∇f (x) T Δxnt = −𝜆(x) 2

I affine invariant (unlike k∇f (x)k 2 )

Convex Optimization Boyd and Vandenberghe 9.23

 Newton’s method

given a starting point x ∈ dom f , tolerance 𝜖 > 0.

repeat
1. Compute the Newton step and decrement.
Δxnt := −∇2 f (x) −1 ∇f (x); 𝜆2 := ∇f (x) T ∇2 f (x) −1 ∇f (x).
2. Stopping criterion. quit if 𝜆2 /2 ≤ 𝜖.
3. Line search. Choose step size t by backtracking line search.
4. Update. x := x + tΔxnt .

I affine invariant, i.e., independent of linear changes of coordinates

I Newton iterates for f̃ (y) = f (Ty) with starting point y (0) = T −1 x (0) are y (k) = T −1 x (k)

Convex Optimization Boyd and Vandenberghe 9.24

 Classical convergence analysis
assumptions
I f strongly convex on S with constant m
I ∇2 f is Lipschitz continuous on S, with constant L > 0:

k∇2 f (x) − ∇2 f (y)k 2 ≤ Lkx − yk 2

(L measures how well f can be approximated by a quadratic function)

outline: there exist constants 𝜂 ∈ (0, m2 /L), 𝛾 > 0 such that

I if k∇f (x)k 2 ≥ 𝜂, then f (x (k+1) ) − f (x (k) ) ≤ −𝛾
I if k∇f (x)k 2 < 𝜂, then
2
L L

(k+1) (k)
k∇f (x )k 2 ≤ k∇f (x )k 2
2m2 2m2

Convex Optimization Boyd and Vandenberghe 9.25

 Classical convergence analysis

damped Newton phase (k∇f (x)k 2 ≥ 𝜂)

I most iterations require backtracking steps
I function value decreases by at least 𝛾
I if p★ > −∞, this phase ends after at most (f (x (0) ) − p★)/𝛾 iterations

quadratically convergent phase (k∇f (x)k 2 < 𝜂)

I all iterations use step size t = 1
I k∇f (x)k 2 converges to zero quadratically: if k∇f (x (k) )k 2 < 𝜂, then
 2l−k   2l−k
L L 1

l k
k∇f (x )k 2 ≤ k∇f (x )k 2 ≤ , l≥k
2m 2 2m 2 2

Convex Optimization Boyd and Vandenberghe 9.26

 Classical convergence analysis

conclusion: number of iterations until f (x) − p★ ≤ 𝜖 is bounded above by

f (x (0) ) − p★
+ log2 log2 (𝜖 0 /𝜖)
𝛾

I 𝛾, 𝜖 0 are constants that depend on m, L, x (0)

I second term is small (of the order of 6) and almost constant for practical purposes
I in practice, constants m, L (hence 𝛾, 𝜖 0 ) are usually unknown
I provides qualitative insight in convergence properties (i.e., explains two algorithm phases)

Convex Optimization Boyd and Vandenberghe 9.27

 Example: R2
(same problem as slide 9.13)

105

x (0) 100

f (x (k) ) − p★
x (1) 10 −5

10 −10

10 −150 1 2 3 4 5
k

I backtracking parameters 𝛼 = 0.1, 𝛽 = 0.7

I converges in only 5 steps
I quadratic local convergence

Convex Optimization Boyd and Vandenberghe 9.28

 Example in R100
(same problem as slide 9.14)

105 2
exact line search
100 1.5

step size t (k)

f (x (k) ) − p★

backtracking
10 −5 1
exact line search
10 −10 0.5 backtracking

10 −150 2 4 6 8 10
0
0 2 4 6 8
k k

I backtracking parameters 𝛼 = 0.01, 𝛽 = 0.5

I backtracking line search almost as fast as exact l.s. (and much simpler)
I clearly shows two phases in algorithm

Convex Optimization Boyd and Vandenberghe 9.29

 Example in R10000
(with sparse ai )
10000 100000
log(1 − xi2 ) − log(bi − aTi x)
Õ Õ
f (x) = −
i=1 i=1

105

f (x (k) ) − p★
100

10 −5

0 5 10 15 20
k

I backtracking parameters 𝛼 = 0.01, 𝛽 = 0.5.

I performance similar as for small examples
Convex Optimization Boyd and Vandenberghe 9.30
 Outline

Terminology and assumptions

Gradient descent method

Steepest descent method

Newton’s method

Self-concordant functions

Implementation

Convex Optimization Boyd and Vandenberghe 9.31

 Self-concordance

shortcomings of classical convergence analysis

I depends on unknown constants (m, L, …)
I bound is not affinely invariant, although Newton’s method is

convergence analysis via self-concordance (Nesterov and Nemirovski)

I does not depend on any unknown constants
I gives affine-invariant bound
I applies to special class of convex self-concordant functions
I developed to analyze polynomial-time interior-point methods for convex optimization

Convex Optimization Boyd and Vandenberghe 9.32

 Convergence analysis for self-concordant functions

definition
I convex f : R → R is self-concordant if |f 000 (x)| ≤ 2f 00 (x) 3/2 for all x ∈ dom f
I f : Rn → R is self-concordant if g(t) = f (x + tv) is self-concordant for all x ∈ dom f , v ∈ Rn

examples on R
I linear and quadratic functions
I negative logarithm f (x) = − log x
I negative entropy plus negative logarithm: f (x) = x log x − log x

affine invariance: if f : R → R is s.c., then f̃ (y) = f (ay + b) is s.c.:

f̃ 000 (y) = a3 f 000 (ay + b), f̃ 00 (y) = a2 f 00 (ay + b)

Convex Optimization Boyd and Vandenberghe 9.33

 Self-concordant calculus

properties
I preserved under positive scaling 𝛼 ≥ 1, and sum
I preserved under composition with affine function
I if g is convex with dom g = R++ and |g000 (x)| ≤ 3g00 (x)/x then

f (x) = log(−g(x)) − log x

is self-concordant

examples: properties can be used to show that the following are s.c.
I f (x) = − mi=1 log(bi − ai x) on {x | ai x < bi , i = 1, . . . , m}
T T
Í

I f (X) = − log det X on Sn++

I f (x) = − log(y2 − xT x) on {(x, y) | kxk 2 < y}

Convex Optimization Boyd and Vandenberghe 9.34

 Convergence analysis for self-concordant functions

summary: there exist constants 𝜂 ∈ (0, 1/4], 𝛾 > 0 such that

I if 𝜆(x) > 𝜂, then f (x (k+1) ) − f (x (k) ) ≤ −𝛾
I if 𝜆(x) ≤ 𝜂, then 2𝜆(x (k+1) ) ≤ 2𝜆(x (k) ) 2

(𝜂 and 𝛾 only depend on backtracking parameters 𝛼, 𝛽)

complexity bound: number of Newton iterations bounded by

f (x (0) ) − p★
+ log2 log2 (1/𝜖)
𝛾

for 𝛼 = 0.1, 𝛽 = 0.8, 𝜖 = 10−10 , bound evaluates to 375(f (x (0) ) − p★) + 6

Convex Optimization Boyd and Vandenberghe 9.35

 Numerical example
I 150 randomly generated instances of f (x) = − m T
i=1 log(bi − ai x), x ∈ R
n
Í

I ◦: m = 100, n = 50; : m = 1000, n = 500; ^: m = 1000, n = 50

25

20

iterations
15

10

0
0 5 10 15 20 25 30 35
f (x (0) ) − p★

I number of iterations much smaller than 375(f (x (0) ) − p★) + 6

I bound of the form c(f (x (0) ) − p★) + 6 with smaller c (empirically) valid

Convex Optimization Boyd and Vandenberghe 9.36

 Outline

Terminology and assumptions

Gradient descent method

Steepest descent method

Newton’s method

Self-concordant functions

Implementation

Convex Optimization Boyd and Vandenberghe 9.37

 Implementation

main effort in each iteration: evaluate derivatives and solve Newton system

HΔx = −g

where H = ∇2 f (x), g = ∇f (x)

via Cholesky factorization

H = LLT , Δxnt = −L −T L −1 g, 𝜆(x) = kL −1 gk 2

I cost (1/3)n3 flops for unstructured system

I cost  (1/3)n3 if H is sparse, banded, or has other structure

Convex Optimization Boyd and Vandenberghe 9.38

 Example

I f (x) = ni=1 𝜓i (xi ) + 𝜓0 (Ax + b), with A ∈ Rp×n dense, p  n

Í

I Hessian has low rank plus diagonal structure H = D + AT H0 A

I D diagonal with diagonal elements 𝜓i00 (xi ); H0 = ∇2 𝜓0 (Ax + b)

method 1: form H, solve via dense Cholesky factorization: (cost (1/3)n3 )

method 2 (block elimination): factor H0 = L0 L0T ; write Newton system as

DΔx + AT L0 w = −g, L0T AΔx − w = 0

eliminate Δx from first equation; compute w and Δx from

(I + L0T AD−1 AT L0 )w = −L0T AD−1 g, DΔx = −g − AT L0 w

cost: 2p2 n (dominated by computation of L0T AD−1 AT L0 )

Convex Optimization Boyd and Vandenberghe 9.39

10. Equality constrained minimization
 Outline

Equality constrained minimization

Newton’s method with equality constraints

Infeasible start Newton method

Implementation

Convex Optimization Boyd and Vandenberghe 10.1

 Equality constrained minimization

I equality constrained smooth minimization problem:

minimize f (x)
subject to Ax = b
I we assume
– f convex, twice continuously differentiable
– A ∈ Rp×n with rank A = p
– p★ is finite and attained

I optimality conditions: x★ is optimal if and only if there exists a 𝜈★ such that

∇f (x★) + AT 𝜈★ = 0, Ax★ = b

Convex Optimization Boyd and Vandenberghe 10.2

 Equality constrained quadratic minimization

I f (x) = (1/2)xT Px + qT x + r, P ∈ Sn+

I ∇f (x) = Px + q
I optimality conditions are a system of linear equations

P AT x
  ★   
−q
=
A 0 𝜈★ b

I coefficient matrix is called KKT matrix

I KKT matrix is nonsingular if and only if

Ax = 0, x≠0 =⇒ xT Px > 0

I equivalent condition for nonsingularity: P + AT A  0

Convex Optimization Boyd and Vandenberghe 10.3

 Eliminating equality constraints

I represent feasible set {x | Ax = b} as {Fz + x̂ | z ∈ Rn−p }

– x̂ is (any) particular solution of Ax = b
– range of F ∈ Rn× (n−p) is nullspace of A (rank F = n − p and AF = 0)

I reduced or eliminated problem: minimize f (Fz + x̂)

I an unconstrained problem with variable z ∈ Rn−p

I from solution z★, obtain x★ and 𝜈★ as

x★ = Fz★ + x̂, 𝜈★ = −(AAT ) −1 A∇f (x★)

Convex Optimization Boyd and Vandenberghe 10.4

 Example: Optimal resource allocation

I allocate resource amount xi ∈ R to agent i

I agent i cost if fi (xi )
I resource budget is b, so x1 + · · · + xn = b
I resource allocation problem is

minimize f1 (x1 ) + f2 (x2 ) + · · · + fn (xn )

subject to x1 + x2 + · · · + xn = b

I eliminate xn = b − x1 − · · · − xn−1 , i.e., choose

I
 
x̂ = ben , F= ∈ Rn× (n−1)
−1T

I reduced problem: minimize f1 (x1 ) + · · · + fn−1 (xn−1 ) + fn (b − x1 − · · · − xn−1 )

Convex Optimization Boyd and Vandenberghe 10.5

 Outline

Equality constrained minimization

Newton’s method with equality constraints

Infeasible start Newton method

Implementation

Convex Optimization Boyd and Vandenberghe 10.6

 Newton step

I Newton step Δxnt of f at feasible x is given by solution v of

∇ f (x) AT
 2
v
   
−∇f (x)
=
A 0 w 0

I Δxnt solves second order approximation (with variable v)

minimize b f (x + v) = f (x) + ∇f (x) T v + (1/2)vT ∇2 f (x)v

subject to A(x + v) = b

I Δxnt equations follow from linearizing optimality conditions

∇f (x + v) + AT w ≈ ∇f (x) + ∇2 f (x)v + AT w = 0, A(x + v) = b

Convex Optimization Boyd and Vandenberghe 10.7

 Newton decrement

I Newton decrement for equality constrained minimization is

 1/2   1/2
T 2
= −∇f (x) T Δxnt

𝜆(x) = Δxnt ∇ f (x)Δxnt

I gives an estimate of f (x) − p★ using quadratic approximation b

f:

f (y) = 𝜆(x) 2 /2
f (x) − inf b
Ay=b

I directional derivative in Newton direction:

d
f (x + tΔxnt ) = −𝜆(x) 2
dt t=0

1/2
I in general, 𝜆(x) ≠ ∇f (x) T ∇2 f (x) −1 ∇f (x)

Convex Optimization Boyd and Vandenberghe 10.8

 Newton’s method with equality constraints

given starting point x ∈ dom f with Ax = b, tolerance 𝜖 > 0.

repeat
1. Compute the Newton step and decrement Δxnt , 𝜆(x).
2. Stopping criterion. quit if 𝜆2 /2 ≤ 𝜖.
3. Line search. Choose step size t by backtracking line search.
4. Update. x := x + tΔxnt .

I a feasible descent method: x (k) feasible and f (x (k+1) ) < f (x (k) )

I affine invariant

Convex Optimization Boyd and Vandenberghe 10.9

 Newton’s method and elimination

I reduced problem: minimize f̃ (z) = f (Fz + x̂)

– variables z ∈ Rn−p
– x̂ satisfies Ax̂ = b; rank F = n − p and AF = 0

I (unconstrained) Newton’s method for f̃ , started at z (0) , generates iterates z (k)

I iterates of Newton’s method with equality constraints, started at x (0) = Fz (0) + x̂, are

x (k) = Fz (k) + x̂

I hence, don’t need separate convergence analysis

Convex Optimization Boyd and Vandenberghe 10.10

 Outline

Equality constrained minimization

Newton’s method with equality constraints

Infeasible start Newton method

Implementation

Convex Optimization Boyd and Vandenberghe 10.11

 Newton step at infeasible points

I with y = (x, 𝜈), write optimality condition as r(y) = 0, where

r(y) = (∇f (x) + AT 𝜈, Ax − b)

is primal-dual residual

I consider x ∈ dom f , Ax ≠ b, i.e., x is infeasible

I linearizing r(y) = 0 gives r(y + Δy) ≈ r(y) + Dr(y)Δy = 0:

∇ f (x) AT ∇f (x) + AT 𝜈
 2    
Δxnt
=−
A 0 Δ𝜈nt Ax − b

I (Δxnt , Δ𝜈nt ) is called infeasible or primal-dual Newton step at x

Convex Optimization Boyd and Vandenberghe 10.12

 Infeasible start Newton method

given starting point x ∈ dom f , 𝜈, tolerance 𝜖 > 0, 𝛼 ∈ (0, 1/2), 𝛽 ∈ (0, 1).
repeat
1. Compute primal and dual Newton steps Δxnt , Δ𝜈nt .
2. Backtracking line search on krk 2 .
t := 1.
while kr(x + tΔxnt , 𝜈 + tΔ𝜈nt )k 2 > (1 − 𝛼t)kr(x, 𝜈)k 2 , t := 𝛽t.
3. Update. x := x + tΔxnt , 𝜈 := 𝜈 + tΔ𝜈nt .
until Ax = b and kr(x, 𝜈)k 2 ≤ 𝜖.

I not a descent method: f (x (k+1) ) > f (x (k) ) is possible

I directional derivative of kr(y)k 2 in direction Δy = (Δxnt , Δ𝜈nt ) is

d
kr(y + tΔy) k 2 = −kr(y)k 2
dt t=0

Convex Optimization Boyd and Vandenberghe 10.13

 Outline

Equality constrained minimization

Newton’s method with equality constraints

Infeasible start Newton method

Implementation

Convex Optimization Boyd and Vandenberghe 10.14

 Solving KKT systems

I feasible and infeasible Newton methods require solving KKT system

H AT v g
    
=−
A 0 w h

I in general, can use LDLT factorization

I or elimination (if H nonsingular and easily inverted):

– solve AH −1 AT w = h − AH −1 g for w
– v = −H −1 (g + AT w)

Convex Optimization Boyd and Vandenberghe 10.15

 Example: Equality constrained analytic centering

Ín
I primal problem: minimize − i=1 log xi subject to Ax = b
Ín
I dual problem: maximize −bT 𝜈 + T
i=1 log(A 𝜈)i +n
– recover x★ as x★
i = 1/(AT 𝜈) i

I three methods to solve:

– Newton method with equality constraints
– Newton method applied to dual problem
– infeasible start Newton method
these have different requirements for initialization

I we’ll look at an example with A ∈ R100×500 , different starting points

Convex Optimization Boyd and Vandenberghe 10.16

 Newton’s method with equality constraints
I requires x (0)  0, Ax (0) = b

105

100
f (x (k) ) − p★

10 −5

10 −10
0 5 10 15 20
k
Convex Optimization Boyd and Vandenberghe 10.17
 Newton method applied to dual problem
I requires AT 𝜈 (0)  0

105

100
p★ − g(𝜈 (k) )

10 −5

10 −10
0 2 4 6 8 10
k
Convex Optimization Boyd and Vandenberghe 10.18
 Infeasible start Newton method
I requires x (0)  0

1010

105

kr(x (k) , 𝜈 (k) ) k 2 100

10 −5

10 −10

10 −15
0 5 10 15 20 25
k
Convex Optimization Boyd and Vandenberghe 10.19
 Complexity per iteration of three methods is identical

I for feasible Newton method, use block elimination to solve KKT system

diag(x) −2 AT diag(x) −1 1
    
Δx
=
A 0 w 0

reduces to solving A diag(x) 2 AT w = b

I for Newton system applied to dual, solve A diag(AT 𝜈) −2 AT Δ𝜈 = −b + A diag(AT 𝜈) −1 1
I for infeasible start Newton method, use block elimination to solve KKT system

diag(x) −2 AT diag(x) −1 1 − AT 𝜈
    
Δx
=
A 0 Δ𝜈 b − Ax

reduces to solving A diag(x) 2 AT w = 2Ax − b

I conclusion: in each case, solve ADAT w = h with D positive diagonal

Convex Optimization Boyd and Vandenberghe 10.20

 Example: Network flow optimization

I directed graph with n arcs, p + 1 nodes

I xi : flow through arc i; 𝜙i : strictly convex flow cost function for arc i
I incidence matrix à ∈ R (p+1) ×n defined as

 1 arc j leaves node i
Ãij = arc j enters node i


−1
 0 otherwise


I reduced incidence matrix A ∈ Rp×n is à with last row removed
I rank A = p if graph is connected
I flow conservation is Ax = b, b ∈ Rp is (reduced) source vector

Ín
I network flow optimization problem: minimize i=1 𝜙i (xi ) subject to Ax = b

Convex Optimization Boyd and Vandenberghe 10.21

 KKT system

I KKT system is
H AT v g
    
=−
A 0 w h
I H = diag(𝜙100 (x1 ), . . . , 𝜙n00 (xn )), positive diagonal
I solve via elimination:

AH −1 AT w = h − AH −1 g, v = −H −1 (g + AT w)

I sparsity pattern of AH −1 AT is given by graph connectivity

(AH −1 AT )ij ≠ 0 ⇐⇒ (AAT )ij ≠ 0

⇐⇒ nodes i and j are connected by an arc

Convex Optimization Boyd and Vandenberghe 10.22

 Analytic center of linear matrix inequality

I minimize − log det X subject to tr(Ai X) = bi , i = 1, . . . , p

I optimality conditions
p
tr(Ai X ★) = bi ,
Õ
X ★  0, −(X ★) −1 + j Ai = 0,
𝜈★ i = 1, . . . , p
j=1

I Newton step ΔX at feasible X is defined by

p
tr(Ai ΔX) = 0,
Õ
X −1 (ΔX)X −1 + wj Ai = X −1 , i = 1, . . . , p
j=1

I follows from linear approximation (X + ΔX) −1 ≈ X −1 − X −1 (ΔX)X −1

I n(n + 1)/2 + p variables ΔX, w

Convex Optimization Boyd and Vandenberghe 10.23

 Solution by block elimination

Íp
I eliminate ΔX from first equation to get ΔX = X − j=1 wj XAj X

I substitute ΔX in second equation to get

p
tr(Ai XAj X)wj = bi ,
Õ
i = 1, . . . , p
j=1

I a dense positive definite set of linear equations with variable w ∈ Rp

I form and solve this set of equations to get w, then get ΔX from equation above

Convex Optimization Boyd and Vandenberghe 10.24

 Flop count

I find Cholesky factor L of X (1/3)n3

I form p products LT Aj L (3/2)pn3

I form p(p + 1)/2 inner products tr((LT Ai L)(LT Aj L)) to get coefficent matrix (1/2)p2 n2

I solve p × p system of equations via Cholesky factorization (1/3)p3

I flop count dominated by pn3 + p2 n2

I cf. naïve method, (n2 + p) 3

Convex Optimization Boyd and Vandenberghe 10.25

11. Interior-point methods
 Outline

Inequality constrained minimization

Logarithmic barrier and central path

Barrier method

Phase I methods

Complexity analysis

Generalized inequalities

Convex Optimization Boyd and Vandenberghe 11.1

 Inequality constrained minimization

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
Ax = b

we assume
I fi convex, twice continuously differentiable
I A ∈ Rp×n with rank A = p
I p★ is finite and attained
I problem is strictly feasible: there exists x̃ with

x̃ ∈ dom f0 , fi ( x̃) < 0, i = 1, . . . , m, Ax̃ = b

hence, strong duality holds and dual optimum is attained

Convex Optimization Boyd and Vandenberghe 11.2

 Examples

I LP, QP, QCQP, GP

I entropy maximization with linear inequality constraints

Ín
minimize i=1 xi log xi
subject to Fx  g, Ax = b

with dom f0 = Rn++

I differentiability may require reformulating the problem, e.g., piecewise-linear minimization

or ℓ∞ -norm approximation via LP

I SDPs and SOCPs are better handled as problems with generalized inequalities (see later)

Convex Optimization Boyd and Vandenberghe 11.3

 Outline

Inequality constrained minimization

Logarithmic barrier and central path

Barrier method

Phase I methods

Complexity analysis

Generalized inequalities

Convex Optimization Boyd and Vandenberghe 11.4

 Logarithmic barrier

I reformulation via indicator function:

Ím
minimize f0 (x) + i=1 I − (fi (x))
subject to Ax = b

where I− (u) = 0 if u ≤ 0, I− (u) = ∞ otherwise

I approximation via logarithmic barrier:
Ím
minimize f0 (x) − (1/t) i=1 log(−fi (x))
subject to Ax = b

I an equality constrained problem

I for t > 0, −(1/t) log(−u) is a smooth approximation of I−
I approximation improves as t → ∞

Convex Optimization Boyd and Vandenberghe 11.5

 I −(1/t) log u for three values of t, and I− (u)

10

−5
−3 −2 −1 0 1
u

Convex Optimization Boyd and Vandenberghe 11.6

 Logarithmic barrier function

I log barrier function for constraints f1 (x) ≤ 0, . . . , fm (x) ≤ 0

m
dom 𝜙 = {x | f1 (x) < 0, . . . , fm (x) < 0}
Õ
𝜙(x) = − log(−fi (x)),
i=1

I convex (from composition rules)

I twice continuously differentiable, with derivatives
m
Õ 1
∇𝜙(x) = ∇fi (x)
i=1
−fi (x)
m m
1 T
Õ 1
∇2 𝜙(x) ∇2 fi (x)
Õ
= ∇f i (x)∇fi (x) +
i=1
fi (x) 2 i=1
−f i (x)

Convex Optimization Boyd and Vandenberghe 11.7

 Central path
I for t > 0, define x★ (t) as the solution of

minimize tf0 (x) + 𝜙(x)

subject to Ax = b

(for now, assume x★ (t) exists and is unique for each t > 0)
I central path is {x★ (t) | t > 0}

example: central path for an LP

c

minimize cT x
subject to aTi x ≤ bi , i = 1, . . . , 6
x★ (10)
hyperplane cT x = cT x★ (t) is tangent to level curve of x★
𝜙 through x★ (t)

Convex Optimization Boyd and Vandenberghe 11.8

 Dual points on central path

I x = x★ (t) if there exists a w such that

m
1
∇fi (x) + AT w = 0,
Õ
t∇f0 (x) + Ax = b
i=1
−fi (x)

I therefore, x★ (t) minimizes the Lagrangian

m
𝜆★i (t)fi (x) + 𝜈★ (t) T (Ax − b)
Õ
L(x, 𝜆★ (t), 𝜈★ (t)) = f0 (x) +
i=1

where we define 𝜆★i (t) = 1/(−tfi (x★ (t)) and 𝜈★ (t) = w/t
I this confirms the intuitive idea that f0 (x★ (t)) → p★ if t → ∞:

p★ ≥ g(𝜆★ (t), 𝜈★ (t)) = L(x★ (t), 𝜆★ (t), 𝜈★ (t)) = f0 (x★ (t)) − m/t

Convex Optimization Boyd and Vandenberghe 11.9

 Interpretation via KKT conditions

x = x★ (t), 𝜆 = 𝜆★ (t), 𝜈 = 𝜈★ (t) satisfy

1. primal constraints: fi (x) ≤ 0, i = 1, . . . , m, Ax = b
2. dual constraints: 𝜆  0
3. approximate complementary slackness: −𝜆 i fi (x) = 1/t, i = 1, . . . , m
4. gradient of Lagrangian with respect to x vanishes:
m
𝜆i ∇fi (x) + AT 𝜈 = 0
Õ
∇f0 (x) +
i=1

difference with KKT is that condition 3 replaces 𝜆i fi (x) = 0

Convex Optimization Boyd and Vandenberghe 11.10

 Force field interpretation

I centering problem (for problem with no equality constraints)

minimize tf0 (x) − m

Í
i=1 log(−fi (x))

I force field interpretation

– tf0 (x) is potential of force field F0 (x) = −t∇f0 (x)

– − log(−fi (x)) is potential of force field Fi (x) = (1/fi (x))∇fi (x)

I forces balance at x★ (t):
m
Õ
F0 (x★ (t)) + Fi (x★ (t)) = 0
i=1

Convex Optimization Boyd and Vandenberghe 11.11

 Example: LP
I minimize cT x subject to aTi x ≤ bi , i = 1, . . . , m, with x ∈ Rn
I objective force field is constant: F0 (x) = −tc
I constraint force field decays as inverse distance to constraint hyperplane:
−ai 1
Fi (x) = , kFi (x)k 2 =
bi − aTi x dist(x, Hi )

where Hi = {x | aTi x = bi }

−c
−3c
t=1 t=3
Convex Optimization Boyd and Vandenberghe 11.12
 Outline

Inequality constrained minimization

Logarithmic barrier and central path

Barrier method

Phase I methods

Complexity analysis

Generalized inequalities

Convex Optimization Boyd and Vandenberghe 11.13

 Barrier method

given strictly feasible x, t := t (0) > 0, 𝜇 > 1, tolerance 𝜖 > 0.

repeat
1. Centering step. Compute x★ (t) by minimizing tf0 + 𝜙, subject to Ax = b.
2. Update. x := x★ (t).
3. Stopping criterion. quit if m/t < 𝜖.
4. Increase t. t := 𝜇t.

I terminates with f0 (x) − p★ ≤ 𝜖 (stopping criterion follows from f0 (x★ (t)) − p★ ≤ m/t)
I centering usually done using Newton’s method, starting at current x
I choice of 𝜇 involves a trade-off: large 𝜇 means fewer outer iterations, more inner
(Newton) iterations; typical values: 𝜇 = 10 or 20
I several heuristics for choice of t (0)

Convex Optimization Boyd and Vandenberghe 11.14

 Example: Inequality form LP
(m = 100 inequalities, n = 50 variables)

140
102

Newton iterations
120
100
duality gap

100
10 −2 80
60
10 −4 40
10 −6 𝜇 = 50 𝜇 = 150 𝜇=2 20
0
0 20 40 60 80 0 40 80 120 160 200
Newton iterations 𝜇

I starts with x on central path (t (0) = 1, duality gap 100)

I terminates when t = 108 (gap 10−6 )
I total number of Newton iterations not very sensitive for 𝜇 ≥ 10

Convex Optimization Boyd and Vandenberghe 11.15

 Example: Geometric program in convex form
(m = 100 inequalities and n = 50 variables)

minimize log 5k=1 exp(aT0k x + b0k )

Í 

subject to log 5k=1 exp(aTik x + bik ) ≤ 0,

Í 
i = 1, . . . , m

102

100
duality gap
10 −2

10 −4

10 −6 𝜇 = 150 𝜇 = 50 𝜇=2
0 20 40 60 80 100 120
Newton iterations
Convex Optimization Boyd and Vandenberghe 11.16
 Family of standard LPs
(A ∈ Rm×2m )
minimize cT x
subject to Ax = b, x0
m = 10, . . . , 1000; for each m, solve 100 randomly generated instances
35

Newton iterations
30

25

20

15 1
10 102 103
m
number of iterations grows very slowly as m ranges over a 100 : 1 ratio
Convex Optimization Boyd and Vandenberghe 11.17
 Outline

Inequality constrained minimization

Logarithmic barrier and central path

Barrier method

Phase I methods

Complexity analysis

Generalized inequalities

Convex Optimization Boyd and Vandenberghe 11.18

 Phase I methods

I barrier method needs strictly feasible starting point, i.e., x with

fi (x) < 0, i = 1, . . . , m, Ax = b

I (like the infeasible start Newton method, more sophisticated interior-point methods do not
require a feasible starting point)
I phase I method forms an optimization problem that
– is itself strictly feasible
– finds a strictly feasible point for original problem, if one exists
– certifies original problem as infeasible otherwise

I phase II uses barrier method starting from strictly feasible point found in phase I

Convex Optimization Boyd and Vandenberghe 11.19

 Basic phase I method

I introduce slack variable s in phase I problem

minimize (over x, s) s
subject to fi (x) ≤ s, i = 1, . . . , m
Ax = b

with optimal value p̄★

– if p̄★ < 0, original inequalities are strictly feasible
– if p̄★ > 0, original inequalities are infeasible
– p̄★ = 0 is an ambiguous case

I start phase I problem with

– any x̃ in problem domain with Ax̃ = b
– s = 1 + maxi fi ( x̃)

Convex Optimization Boyd and Vandenberghe 11.20

 Sum of infeasibilities phase I method

I minimize sum of slacks, not max:

minimize 1T s
subject to s  0, fi (x) ≤ si , i = 1, . . . , m
Ax = b

I will find a strictly feasible point if one exists

I for infeasible problems, produces a solution that satisfies many (but not all) inequalities

I can weight slacks to set priorities (in satifying constraints)

Convex Optimization Boyd and Vandenberghe 11.21

 Example

I infeasible set of 100 linear inequalities in 50 variables

I left: basic phase I solution; satisfies 39 inequalities
I right: sum of infeasibilities phase I solution; satisfies 79 inequalities

60 60

40 40
number

number
20 20

0 0
−1 −0.5 0 0.5 1 1.5 −1 −0.5 0 0.5 1 1.5
bi − aTi xmax bi − aTi xsum

Convex Optimization Boyd and Vandenberghe 11.22

 Example: Family of linear inequalities

I Ax  b + 𝛾Δb; strictly feasible for 𝛾 > 0, infeasible for 𝛾 < 0

I use basic phase I, terminate when s < 0 or dual objective is positive
I number of iterations roughly proportional to log(1/|𝛾|)

100 100 100

Newton iterations

Newton iterations
Newton iterations

80 Infeasible Feasible 80 80

60 60 60

40 40 40

20 20 20

0 0 0
−1 −0.5 0 0.5 1 −100 −10 −2 −10 −4 −10 −6 10 −6 10 −4 10 −2 100
𝛾 𝛾 𝛾

Convex Optimization Boyd and Vandenberghe 11.23

 Outline

Inequality constrained minimization

Logarithmic barrier and central path

Barrier method

Phase I methods

Complexity analysis

Generalized inequalities

Convex Optimization Boyd and Vandenberghe 11.24

 Number of outer iterations

I in each iteration duality gap is reduced by exactly the factor 𝜇

I number of outer (centering) iterations is exactly

log(m/(𝜖t (0) ))
 

log 𝜇

plus the initial centering step (to compute x★ (t (0) ))

I we will bound number of Newton steps per centering iteration using self-concordance
analysis

Convex Optimization Boyd and Vandenberghe 11.25

 Complexity analysis via self-concordance

same assumptions as on slide 11.2, plus:

I sublevel sets (of f0 , on the feasible set) are bounded
I tf0 + 𝜙 is self-concordant with closed sublevel sets

second condition
I holds for LP, QP, QCQP
I may require reformulating the problem, e.g.,
Ín Ín
minimize i=1 xi log xi −→ minimize i=1 xi log xi
subject to Fx  g subject to Fx  g, x  0

I needed for complexity analysis; barrier method works even when self-concordance
assumption does not apply

Convex Optimization Boyd and Vandenberghe 11.26

 Newton iterations per centering step
I we compute x+ = x★ (𝜇t), by minimizing 𝜇tf0 (x) + 𝜙(x) starting from x = x★ (t)
I from self-concordance theory,
𝜇tf0 (x) + 𝜙(x) − 𝜇tf0 (x+ ) − 𝜙(x+ )
#Newton iterations ≤ +c
𝛾
I 𝛾, c are constants (that depend only on Newton algorithm parameters)
I we will bound numerator 𝜇tf0 (x) + 𝜙(x) − 𝜇tf0 (x+ ) − 𝜙(x+ )

I with 𝜆i = 𝜆★i (t) = −1/(tfi (x)), we have −fi (x) = 1/(t𝜆i ), so

m
Õ m
Õ
𝜙(x) = − log(−fi (x)) = log(t𝜆i )
i=1 i=1
so
m
Õ m

Õ
𝜙(x) − 𝜙(x+ ) = log(t𝜆i ) + log(−fi (x+ )) = log(−𝜇t𝜆i fi (x+ )) − m log 𝜇
i=1 i=1
Convex Optimization Boyd and Vandenberghe 11.27
 Ím
using log u ≤ u − 1 we have 𝜙(x) − 𝜙(x+ ) ≤ −𝜇t i=1 𝜆 i fi (x
+) − m − m log 𝜇, so

𝜇tf0 (x) + 𝜙(x) − 𝜇tf0 (x+ ) − 𝜙(x+ )

Õm
≤ 𝜇tf0 (x) − 𝜇tf0 (x+ ) − 𝜇t 𝜆i fi (x+ ) − m − m log 𝜇
i=1
m
!
T
Õ
= 𝜇tf0 (x) − 𝜇t f0 (x ) +
+
𝜆i fi (x ) + 𝜈 (Ax − b) − m − m log 𝜇
+ +

i=1
= 𝜇tf0 (x) − 𝜇tL(x+ , 𝜆, 𝜈) − m − m log 𝜇
≤ 𝜇tf0 (x) − 𝜇tg(𝜆, 𝜈) − m − m log 𝜇
= m(𝜇 − 1 − log 𝜇)

using L(x+ , 𝜆, nu) ≥ g(𝜆, 𝜈) in second last line and f0 (x) − g(𝜆, 𝜈) = m/t in last line

Convex Optimization Boyd and Vandenberghe 11.28

 Total number of Newton iterations

log(m/(t (0) 𝜖)) m(𝜇 − 1 − log 𝜇)

  
#Newton iterations ≤ N = +c
log 𝜇 𝛾
5 104

4 104

3 104
N versus 𝜇 for typical values of 𝛾, c;
N

m
2 104 m = 100, initial duality gap t (0) 𝜖
= 105
1 104

0
1 1.1 1.2
𝜇
I confirms trade-off in choice of 𝜇
I in practice, #iterations is in the tens; not very sensitive for 𝜇 ≥ 10

Convex Optimization Boyd and Vandenberghe 11.29

 Polynomial-time complexity of barrier method

√
I for 𝜇 = 1 + 1/ m:
√ m/t (0)
  
N=O m log
𝜖
√
I number of Newton iterations for fixed gap reduction is O( m)
I multiply with cost of one Newton iteration (a polynomial function of problem dimensions),
to get bound on number of flops
I this choice of 𝜇 optimizes worst-case complexity; in practice we choose 𝜇 fixed and larger

Convex Optimization Boyd and Vandenberghe 11.30

 Outline

Inequality constrained minimization

Logarithmic barrier and central path

Barrier method

Phase I methods

Complexity analysis

Generalized inequalities

Convex Optimization Boyd and Vandenberghe 11.31

 Generalized inequalities

minimize f0 (x)
subject to fi (x) Ki 0, i = 1, . . . , m
Ax = b

I f0 convex, fi : Rn → Rki , i = 1, . . . , m, convex with respect to proper cones Ki ∈ Rki

I we assume
– fi twice continuously differentiable
– A ∈ Rp×n with rank A = p
– p★ is finite and attained
– problem is strictly feasible; hence strong duality holds and dual optimum is attained

I examples of greatest interest: SOCP, SDP

Convex Optimization Boyd and Vandenberghe 11.32

 Generalized logarithm for proper cone

𝜓 : Rq → R is generalized logarithm for proper cone K ⊆ Rq if:

I dom 𝜓 = int K and ∇2 𝜓(y) ≺ 0 for y K 0
I 𝜓(sy) = 𝜓(y) + 𝜃 log s for y K 0, s > 0 (𝜃 is the degree of 𝜓)

examples
I nonnegative orthant K = Rn+ : 𝜓(y) = ni=1 log yi , with degree 𝜃 = n
Í

I positive semidefinite cone K = Sn+ : 𝜓(Y) = log det Y, with degree 𝜃 = n

I second-order cone K = {y ∈ Rn+1 | (y12 + · · · + yn2 ) 1/2 ≤ yn+1 }:

2
𝜓(y) = log(yn+1 − y12 − · · · − yn2 ) with degree (𝜃 = 2)

Convex Optimization Boyd and Vandenberghe 11.33

 Properties
I (without proof): for y K 0,

∇𝜓(y) K ∗ 0, yT ∇𝜓(y) = 𝜃
I nonnegative orthant Rn+ : 𝜓(y) = ni=1 log yi
Í

∇𝜓(y) = (1/y1 , . . . , 1/yn ), yT ∇𝜓(y) = n

I positive semidefinite cone Sn+ : 𝜓(Y) = log det Y

∇𝜓(Y) = Y −1 , tr(Y ∇𝜓(Y)) = n

I second-order cone K = {y ∈ Rn+1 | (y12 + · · · + yn2 ) 1/2 ≤ yn+1 }:
 −y1 
 
2  .. 
yT ∇𝜓(y) = 2
 . 
∇𝜓(y) = 2 ,
yn+1 − y12 − · · · − yn2  −yn 

 yn+1 
 
Convex Optimization Boyd and Vandenberghe 11.34
 Logarithmic barrier and central path

logarithmic barrier for f1 (x) K1 0, …, fm (x) Km 0:

m
dom 𝜙 = {x | fi (x) ≺Ki 0, i = 1, . . . , m}
Õ
𝜙(x) = − 𝜓i (−fi (x)),
i=1

I 𝜓i is generalized logarithm for Ki , with degree 𝜃 i

I 𝜙 is convex, twice continuously differentiable

central path: {x★ (t) | t > 0} where x★ (t) is solution of

minimize tf0 (x) + 𝜙(x)

subject to Ax = b

Convex Optimization Boyd and Vandenberghe 11.35

 Dual points on central path
x = x★ (t) if there exists w ∈ Rp ,
m
Dfi (x) T ∇𝜓i (−fi (x)) + AT w = 0
Õ
t∇f0 (x) +
i=1

(Dfi (x) ∈ Rki ×n is derivative matrix of fi )

I therefore, x★ (t) minimizes Lagrangian L(x, 𝜆★ (t), 𝜈★ (t)), where

1 w
𝜆★i (t) = ∇𝜓i (−fi (x★ (t))), 𝜈★ (t) =
t t
I from properties of 𝜓i : 𝜆★i (t) K ∗ 0, with duality gap
i

m
Õ
f0 (x (t)) − g(𝜆 (t), 𝜈 (t)) = (1/t)
★ ★ ★
𝜃i
i=1

Convex Optimization Boyd and Vandenberghe 11.36

 Example: Semidefinite programming
(with Fi ∈ Sp )
minimize cT x
subject to F (x) = ni=1 xi Fi + G  0
Í

I logarithmic barrier: 𝜙(x) = log det(−F (x) −1 )

I central path: x★ (t) minimizes tcT x − log det(−F (x)); hence

tci − tr(Fi F (x★ (t)) −1 ) = 0, i = 1, . . . , n

I dual point on central path: Z ★ (t) = −(1/t)F (x★ (t)) −1 is feasible for

maximize tr(GZ)
subject to tr(Fi Z) + ci = 0, i = 1, . . . , n
Z0

I duality gap on central path: cT x★ (t) − tr(GZ ★ (t)) = p/t

Convex Optimization Boyd and Vandenberghe 11.37
 Barrier method

given strictly feasible x, t := t (0) > 0, 𝜇 > 1, tolerance 𝜖 > 0.

repeat
1. Centering step. Compute x★ (t) by minimizing tf0 + 𝜙, subject to Ax = b.
2. Update. x := x★ (t).
3. Stopping criterion. quit if ( i 𝜃 i )/t < 𝜖.
Í
4. Increase t. t := 𝜇t.

I only difference is duality gap m/t on central path is replaced by

Í
i 𝜃 i /t
I number of outer iterations:
log(( i 𝜃 i )/(𝜖t (0) ))
 Í 

log 𝜇
I complexity analysis via self-concordance applies to SDP, SOCP

Convex Optimization Boyd and Vandenberghe 11.38

 Example: SOCP

(50 variables, 50 SOC constraints in R6 )

102
120

Newton iterations
100
duality gap

10 −2 80

10 −4
40

10 −6 𝜇 = 50 𝜇 = 200 𝜇=2
0
0 20 40 60 80 20 60 100 140 180
Newton iterations 𝜇

Convex Optimization Boyd and Vandenberghe 11.39

 Example: SDP

(100 variables, LMI constraint in S100 )

140
102

Newton iterations
100 100
duality gap

10 −2
60
10 −4

10 −6 𝜇 = 150 𝜇 = 50 𝜇=2 20

0 20 40 60 80 100 0 20 40 60 80 100 120

Newton iterations 𝜇

Convex Optimization Boyd and Vandenberghe 11.40

 Example: Family of SDPs
(A ∈ Sn , x ∈ Rn )
minimize 1T x
subject to A + diag(x)  0
n = 10, . . . , 1000; for each n solve 100 randomly generated instances
35

Newton iterations
30

25

20

15
101 102 103
n
Convex Optimization Boyd and Vandenberghe 11.41
 Primal-dual interior-point methods

I more efficient than barrier method when high accuracy is needed

I update primal and dual variables, and 𝜅, at each iteration; no distinction between inner
and outer iterations
I often exhibit superlinear asymptotic convergence
I search directions can be interpreted as Newton directions for modified KKT conditions

I can start at infeasible points

I cost per iteration same as barrier method

Convex Optimization Boyd and Vandenberghe 11.42

12. Conclusions
 Modeling

mathematical optimization
I problems in engineering design, data analysis and statistics, economics, management, …,
can often be expressed as mathematical optimization problems
I techniques exist to take into account multiple objectives or uncertainty in the data

tractability
I roughly speaking, tractability in optimization requires convexity
I algorithms for nonconvex optimization find local (suboptimal) solutions, or are very
expensive
I surprisingly many applications can be formulated as convex problems

Convex Optimization Boyd and Vandenberghe 12.1

 Theoretical consequences of convexity

I local optima are global

I extensive duality theory
– systematic way of deriving lower bounds on optimal value
– necessary and sufficient optimality conditions
– certificates of infeasibility
– sensitivity analysis
I solution methods with polynomial worst-case complexity theory
(with self-concordance)

Convex Optimization Boyd and Vandenberghe 12.2

 Practical consequences of convexity

(most) convex problems can be solved globally and efficiently

I interior-point methods require 20 – 80 steps in practice
I basic algorithms (e.g., Newton, barrier method, …) are easy to implement and work well
for small and medium size problems (larger problems if structure is exploited)
I high-quality solvers (some open-source) are available
I high level modeling tools like CVXPY ease modeling and problem specification

Convex Optimization Boyd and Vandenberghe 12.3

 How to use convex optimization

to use convex optimization in some applied context

I use rapid prototyping, approximate modeling
– start with simple models, small problem instances, inefficient solution methods
– if you don’t like the results, no need to expend further effort on more accurate models or
efficient algorithms
I work out, simplify, and interpret optimality conditions and dual
I even if the problem is quite nonconvex, you can use convex optimization
– in subproblems, e.g., to find search direction
– by repeatedly forming and solving a convex approximation at the current point

Convex Optimization Boyd and Vandenberghe 12.4

 Further topics

some topics we didn’t cover:

I methods for very large scale problems
I subgradient calculus, convex analysis
I localization, subgradient, proximal and related methods
I distributed convex optimization
I applications that build on or use convex optimization
these are all covered in EE364b.

Convex Optimization Boyd and Vandenberghe 12.5

 Related classes

I EE364b — convex optimization II (Pilanci)

I EE364m — mathematics of convexity (Duchi)
I CS261, CME334, MSE213 — theory and algorithm analysis (Sidford)
I AA222 — algorithms for nonconvex optimization (Kochenderfer)
I CME307 — linear and conic optimization (Ye)

Convex Optimization Boyd and Vandenberghe 12.6