CPSC 542F WINTER 2017: Lecture Notes

CPSC 542F WINTER 2017
Convex Analysis and Optimization
Lecture Notes
S UMMER R ESEARCH I NTERNSHIP, U NIVERSITY OF W ESTERN O NTARIO
GITHUB . COM /L AURETH T E X/C LUSTERING
This research was done under the supervision of Dr. Pauline Barmby with the financial support of
the MITACS Globalink Research Internship Award within a total of 12 weeks, from June 16th to
September 5th of 2014.
First release, August 2014

Contents
1 Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1 Convexity 5
1.1.1 Cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Convex Functions 6
1.2.1 Epigraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Support Function 8
1.4 Operations Preserve Convexity of Functions 9
1. Convex Sets
1.1 Convexity
1.1.1 Cone
Definition 1.1.1 — Cone. A set K ∈ Rn , when x ∈ K implies αx ∈ K.
A non convex cone can be hyper-plane.

For convex cone x + y ∈ K, ∀x, y ∈ K.
Cone don’t need to be "pointed". e.g.
Direct sums of cones C1 +C2 = {x = x1 + x2 |x1 ∈ C1 , x2 ∈ C2 }.
Example 1.1 S1n {X|X = X n , λ (x) ≥ 0}

A matrix with positive eigenvalues.
Operations preserving convexity

Intersection C ∩i∈I Ci
Linear map Let A : Rn → Rn be a linear map. If C ∈ Rn is convex, so is A(C) = {Ax∀x ∈ C}
Inverse image A−1 (D) = {x ∈ R|Ax ∈ D}
Operations that induce convexity

Convex hull on S = ∩{C|S ∈ C,Cisconvex}
Example 1.2 Co{x1 , x2 , · · · , xm } = {∑m

i=1 αi xi |α ∈ δm }
For a convex set x ∈ C ⇒ x = ∑ αi xi .
Theorem 1.1.1 — Carathéodory’s theorem. If a point x ∈ Rd lies in the convex hull of a set
P, there is a subset P0 of P consisting of d + 1 or fewer points such that x lies in the convex hull of
P0 . Equivalently, x lies in an r-simplex with vertices in P.
6 Chapter 1. Convex Sets
1.2 Convex Functions

Definition 1.2.1 — Convex function. Let C ∈ Rn be convex, f : C → R is convex on f if
x, y ∈ C ×C. ∀α ∈ (0, 1), f (αx + (1 − α)y) ≤ f (αx) + f ((1 − α)y)
Definition 1.2.2 — Strictly Convex function. Let C ∈ Rn be convex, f : C → R is strictly convex

on f if x, y ∈ C ×C. ∀α ∈ (0, 1), f (αx + (1 − α)y)h f (αx) + f ((1 − α)y)
Definition 1.2.3 — Strongly convex. f : C → R is strongly convex with modules u ≥ 0 if

f − 21 u|| · ||2 is convex.
Interpretation: There is a convex quadratic 21 u|| · ||2 that lower bounds f.

Example 1.3 minx∈C f (x) ↔ min f¯(x) Useful to turn this into an unconstrained problem.
(
f (x) i f x ∈ C
f¯(x) =
∞ elsewhere

Definition 1.2.4 A function f : Rn → R ∪ ∞ R̄ is convex if x, y ∈ Rn × Rn , ∀x, y, f¯(αx + (1 −

α)y) ≤ f (αx) + f ((1 − α)y)
Definition 1 is equivalent to definition 2 if f (x) = ∞.

Example 1.4 f (x) = sup j∈J f j (x)
1.2.1 Epigraph
Definition 1.2.5 — Epigraph. For f : Rn → R̄, its epigraph epi( f ) ∈ Rn+1 isthesetepi( f ){(x, α)| f (x) ∈
α}
Next: a function is convex i.f.f. its epigraph is convex.

Definition 1.2.6 A function f : C → R,C ∈ Rn is convex if ∀x, y ∈ C, f (ax + (1 − a)x) ≤ a f (x) +
(1 − a) f (x) ∀a ∈ (0, 1).
Strict convex: x 6= y ⇒ f (ax + (1 − a)x) ≤ a f (x) + (1 − a) f (x)
R f is convex ⇒ − f is concave.
Level set: Sα f = {x| f (x) ≤ α}.

Sα f is convex ⇔ f is convex.
Definition 1.2.7 — Strongly convex. f : C → R is strongly convex with modules µ if ∀x, y ∈ C,

1
∀α ∈ (0, 1), f (ax + (1 − a)x) ≤ a f (x) + (1 − a) f (x) − 2µ α(1 − α)kx − yk2 .
R
• f is 2nd-differentiable, f ix convex ⇐⇒ ∇2 f (x)i0.
• f is strongly convex ⇐⇒ ∇2 f (x)iµI ⇐⇒ x ≥ µ
1.2 Convex Functions 7
Definition 1.2.8 — 2. f : Rn → R̄ is convexif x, y ∈ R, α ∈ (0, 1), f (ax + (1 − a)x) ≤ a f (x) +

(1 − a) f (x).
The effective domain of f is dom f = {x| f (x)h+∞}

(
0 x ∈C
Example 1.5 — ludcator function. δc (x) = .
+∞ elsewhere
domδc (x) = C
Definition 1.2.9 — Epigraph. The epigraph of f is epi f = {(x, α)| f (x) ≤ α}
The graph of epi f is {(x, f (x)|x ∈ dom f }.

Definition 1.2.10 — III. A function f : Rn → R̄ is
Theorem 1.2.1 f : Rn → R̄ is convex ⇐⇒ ∀x, y ∈ Rn , α ∈ (0, 1), f (ax + (1 − a)x) ≤ a f (x) +

(1 − a) f (x).
Proof. ⇒ take x, y ∈ dom f , (x, f (x)) ∈ epi f ,(y, f (y)) ∈ epi f .
Example 1.6 — Distance. Distance to a convexset dc (x) = inf{kz − xk|z ∈ C}. Take any two
sequence {yk }and{ȳk } ⊂ C s.t. kyk − xk → dc (x), kȳk − x̄k → dc (x̄). zk = αyk + (1 − α)ȳk .
dc (αx + (1 − α)x̄) ≤ kzk − αx − (1 − α)x̄k

= kα(yk − x) + (1 − α)(ȳk − x̄)k
≤ αkyk − xk + (1 − α)kȳk − x̄k
Take k → ∞, dc (αx + (1 − α)x̄) ≤ αd(x) + (1 − α)d(x̄)
Example 1.7 — Eigenvalues. Let X ∈ Sn := {n × nsymmetricmatrix}. λ1 (x) ≥ λ2 (X) ≥ . . . ≥

λn (x).
fk (x) = ∑n1 λi (x).
Equivalent characterization
fk (x) = max{∑ vTi Xvi |vi ⊥ v j , i 6= j}

i
= max{tr(V T XV |V T V = Ik }
max{tr(VV T X)}by circularity
Note hA, Bi = tr(A, B) is true for symmetric matrix.

hA, Ai = |A|2F = ∑i A2ii
1.3 Support Function
Take a set C ∈ Rn , not necessarily convex.The support function is σC = Rn → R̄. σC (x) =
∑{hx, ui|u ∈ C}.

Fact 1.3.1 The support function binds the supporting hyper-plane.
Supporting functions are
• Positively homogeneous
σC (αx) = ασC (x)∀αi0
σC (αx) = supu∈C hαx, ui = α supu∈C hx, ui = ασC (x)
• Sub-linear( a special case of convex, linear combination holds ∀α.
σC (αx + (1 − α)y) = supu∈C hαx + (1 − α)y, ui ≤ α supu∈C hx, ui + (1 − α) supu∈C hy, ui
Example 1.8 — L2-norm. kxk = supu∈C {hx, ui, u ∈ Rn }.

1
kxk p = sup{hx, ui, u ∈ Bq } where p + 1q = 1. Bq = {kxkq ≤ 1}.
The norm is
• Positive homogeneous
• sub-linear
• If 0 ∈ C, σC is non-negative.
• If C is central-symmetric, σC (0) = 0 and σC (x) = σC (−x)
Fact 1.3.2 — Epigraph of a support function. epiσC = {(x,t)|σC (x) ≤ t}. Suppose (x,t) ∈ epiσC .
Take any α > 0. α(x,t) = (αx, αt).
ασC (x) = ασC (x) ≤ αt. α(x, c) ∈ epiσC
1.4 Operations Preserve Convexity of Functions 9
1.4 Operations Preserve Convexity of Functions

• Positive affine transformation
f1 , f2 , . . . , fk ∈ cvxRn .
f = α1 f1 + α2 f2 + . . . + αk fk
• Supremum of functions. Let { fi }i∈I be arbitrary family of functions. If ∃x sup j∈J f j (x) < ∞ ⇔
f (x) = sup j∈J f j (x)
• Composition with linear map.

f ∈ cvxRn , A : Rn → Rm is a linear map. f ◦ A(x) = f (Ax) ∈ cvxRn
f ◦ A(x) = f (A(αx + (1 − α)y))

= f (Aαx + (1 − α)Ay)
≤ α f (Ax) + (a − α) f (Ay)

CPSC 542F WINTER 2017: Lecture Notes

Uploaded by

Copyright:

Available Formats

CPSC 542F WINTER 2017: Lecture Notes

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CPSC 542F WINTER 2017: Lecture Notes

Uploaded by

Copyright:

Available Formats

CPSC 542F WINTER 2017

Convex Analysis and Optimization

GITHUB . COM /L AURETH T E X/C LUSTERING

First release, August 2014

A non convex cone can be hyper-plane.

Example 1.1 S1n {X|X = X n , λ (x) ≥ 0}

Operations preserving convexity

Operations that induce convexity

Example 1.2 Co{x1 , x2 , · · · , xm } = {∑m

For a convex set x ∈ C ⇒ x = ∑ αi xi .

1.2 Convex Functions

Definition 1.2.2 — Strictly Convex function. Let C ∈ Rn be convex, f : C → R is strictly convex

Definition 1.2.3 — Strongly convex. f : C → R is strongly convex with modules u ≥ 0 if

Interpretation: There is a convex quadratic 21 u|| · ||2 that lower bounds f.

Definition 1.2.4 A function f : Rn → R ∪ ∞ R̄ is convex if x, y ∈ Rn × Rn , ∀x, y, f¯(αx + (1 −

Definition 1 is equivalent to definition 2 if f (x) = ∞.

Next: a function is convex i.f.f. its epigraph is convex.

Level set: Sα f = {x| f (x) ≤ α}.

Definition 1.2.7 — Strongly convex. f : C → R is strongly convex with modules µ if ∀x, y ∈ C,

Definition 1.2.8 — 2. f : Rn → R̄ is convexif x, y ∈ R, α ∈ (0, 1), f (ax + (1 − a)x) ≤ a f (x) +

The effective domain of f is dom f = {x| f (x)h+∞}

Definition 1.2.9 — Epigraph. The epigraph of f is epi f = {(x, α)| f (x) ≤ α}

The graph of epi f is {(x, f (x)|x ∈ dom f }.

Theorem 1.2.1 f : Rn → R̄ is convex ⇐⇒ ∀x, y ∈ Rn , α ∈ (0, 1), f (ax + (1 − a)x) ≤ a f (x) +

Proof. ⇒ take x, y ∈ dom f , (x, f (x)) ∈ epi f ,(y, f (y)) ∈ epi f .

dc (αx + (1 − α)x̄) ≤ kzk − αx − (1 − α)x̄k

Take k → ∞, dc (αx + (1 − α)x̄) ≤ αd(x) + (1 − α)d(x̄)

Example 1.7 — Eigenvalues. Let X ∈ Sn := {n × nsymmetricmatrix}. λ1 (x) ≥ λ2 (X) ≥ . . . ≥

fk (x) = max{∑ vTi Xvi |vi ⊥ v j , i 6= j}

Note hA, Bi = tr(A, B) is true for symmetric matrix.

1.3 Support Function

Take a set C ∈ Rn , not necessarily convex.The support function is σC = Rn → R̄. σC (x) =

∑{hx, ui|u ∈ C}.

Supporting functions are

Example 1.8 — L2-norm. kxk = supu∈C {hx, ui, u ∈ Rn }.

1.4 Operations Preserve Convexity of Functions

• Composition with linear map.

f ◦ A(x) = f (A(αx + (1 − α)y))

You might also like