6CCS3OME – Optimisation Methods

Basic concepts in convexity,

convex sets, convex functions,
Convex optimisation

Yali Du

Department of Informatics, King’s College London

Recall the lecture

• What can you recall from the lecture?

• Leave your answers at pollev.com/yalidu885

2
Recall the lecture

• What can you recall from the lecture?

• Leave your answers at pollev.com/yalidu885

3
Quiz: convexity basics

4
 Quiz: convexity basics
Given a convex optimisation problem, answer Yes or No to the
questions below.
1. Is local optimum always global optimum?
2. Is optimal solution unique?
3. Does the solution always exist?
4. If f(x) is convex, is ( − ) always convex?
5. If f(x) is convex, Is − ( ) always convex?
6. If C1 ⊆ R, C2 ⊆ R are two convex set, is C1 ∪ C2 always
a convex set?
7. If C1 ⊆ R, C2 ⊆ R are two convex set, is C1 ∩ C2 always
a convex set?

5
𝑓
𝑓
𝑥
𝑥
 Quiz: convexity basics
Given a convex optimisation problem, answer Yes or No to the
questions below.
1. Is local optimum always global optimum? (Yes)
2. Is optimal solution unique? (No)
3. Does the solution always exist? (No)
4. If f(x) is convex, is ( − ) always convex? (Yes)
5. If f(x) is convex, Is − ( ) always convex? (No)
6. If C1 ⊆ R, C2 ⊆ R are two convex set, is C1 ∪ C2 always
a convex set? (No)
7. If C1 ⊆ R, C2 ⊆ R are two convex set, is C1 ∩ C2 always
a convex set? (Yes)

6
𝑓
𝑓
𝑥
𝑥
 Recap: Convex Sets and functions
convex set: contains line segment between any two points in
the set. That is, if , ∈ , then + (1 − ) ∈ for all
∈ [0,1]
A function : → is convex if is a convex set and

( + (1 − ) ) ≤ ( ) + (1 − ) ( )
for all , ∈ and ∈ [0,1]

7
Convex Set Not a Convex Set
𝑓
𝑡
𝑥
𝑡
𝑦
𝑡
𝑓
𝑥
𝑡
𝑓
𝑦
𝑓
𝐶
𝑅
𝑡
𝑥
𝑡
𝑦
𝐶
𝑡
𝑡
𝑥
𝐶
𝑥
𝑦
𝑦
𝐶
𝐶
𝑥𝑦𝑥𝑦
𝑛
 Recap: Convex Sets and functions
1st-order condition: differentiable with convex domain is convex iff

( ) ≥ ( ) + ∇ ( )( − ) for all , ∈

i.e. first-order approximation of is global underestimator

2nd-order conditions: for twice differentiable with convex domain
is convex if and only if

∇2 ( ) ≽ 0 for all ∈
i.e. the hessian is positive semi-definite (all of the eigenvalues are nonnegative).

• is strictly convex if is convex and the inequalities strictly hold for all , ∈ ,
≠ and ∈ [0,1]
• Every local optimum of a convex optimization problem is a global optimum

8
𝑓
𝑥
𝑓
𝑦
𝑓
𝑥
𝑓
𝑥
𝑦
𝑥
𝑓
𝑓
𝑡
𝑓
𝑓𝑓
𝑥
𝑦
𝑥
𝑥
𝐶𝑥
𝑦
𝑦
𝐶
𝐶
𝐶
Topics
• Examples of convex set
• Examples of convex function
• Convex Optimality

9
 x1, x2 ∈ C, 0 ≤ θ ≤ 1
Example =⇒ θx1 + (1 − θ

• Is any
etween this a convex set?
amples (onetwo pointstwo
convex, in the set
nonconvex sets)

=⇒ θx1 + (1 − θ)x2 ∈ C

x sets)

ex sets

10
 x1, x2 ∈ C, 0 ≤ θ ≤ 1
Example =⇒ θx1 + (1 − θ

• Is any
etween this a convex set?
amples (onetwo pointstwo
convex, in the set
nonconvex sets)

=⇒ θx1 + (1 − θ)x2 ∈ C

x sets)
(Yes) (No)

ex sets

(No) (Yes)
11
Polyhedra
Polyhedra
• The feasible region for linear programme
solution set of finitely many linear inequalities and equalities

Ax ≼ b, Cx = d

(A ∈ Rm×n, C ∈ Rp×n, ≼ is componentwise inequality)

a1 a2

P
a5
a3

a4

polyhedron is intersection of finite number of halfspaces and hyperplanes

12
 Convex function
• Given ( ), can you verify whether they are i) convex, ii)
strictly convex, iii) concave, etc?

13
𝑓
𝑥
 Convex Functions
• Which of the following functions are convex, strictly convex,
concave?
1. exp( )
2. log( )
3. co ( )
2
4.
3
5. max( , 0)
6.
7. | |

(1 0)
01
8. ∥Ax − b∥22, x ∈ R 2, =
𝐴
14
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
𝑠
𝑥
 Convex Functions
• Which of the following functions are convex, strictly convex,
concave?
• Convex in green, concave in red
1. exp( ) (strictly convex)
2. log( )
3. cos(x) (neither convex nor concave)
2
4. (strictly convex)
3
5. max( , 0)
6.
7. | | (convex)

(1 0)
0 1
8. ∥Ax − b∥22, x ∈ R 2, A := (strictly convex)

15
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
 Example: the max function
• Max function: ( ) = max , ∈ .
𝑖
16
𝑖
𝑓
𝑥
𝑥
𝑥
𝑅
𝑛
 Example: composite function
• Would a composite of two convex functions convex?
2
• Example: Consider two convex functions ( ) = and
( ) = 2 − 1. Then, the composition
( ◦ )( ) = 4 − 2 2 + 1.

17
𝑓
𝑔
𝑓
𝑥
𝑥
𝑔
𝑥
𝑥
𝑥
𝑥
𝑥
 Example: composite function
• Would a composite of two convex functions convex?
2
• Example: Consider two convex functions ( ) = and
( ) = 2 − 1. Then, the composition
( ◦ )( ) = 4 − 2 2 + 1.

18
𝑓
𝑔
𝑓
𝑥
𝑥
𝑔
𝑥
𝑥
𝑥
𝑥
𝑥
Quiz: convex functions

19
 Quiz: convex functions
• Which of the following functions are convex, strictly convex, concave?
1. 5x + 1
2. exp(−x)
3. sin(x)
4. f (x1, x2) = x1x2, x1 ≥ 0,x2 ≥ 0.
5. max(ax + b,0), x ∈ R, a < 0, b ∈ R
6. max(ax + b,0), x ∈ R, a > 0, b ∈ R

(1 0)
01
7. ∥Ax − b∥22, x ∈ R 2, =

A. Convex B. convex+ strictly convex

C. concave D. convex + concave
E. none of these properties
𝐴
20
 Quiz: convex functions
• Which of the following functions are convex, strictly convex, concave?
1. 5x + 1 (D)
2. exp(−x) (B)
3. sin(x) (E)
4. f (x1, x2) = x1x2, x1 ≥ 0,x2 ≥ 0. (E)
5. max(ax + b,0), x ∈ R, a < 0, b ∈ R(A)
6. max(ax + b,0), x ∈ R, a > 0, b ∈ R(A)

(1 0)
01
7. ∥Ax − b∥22, x ∈ R 2, = (B)

A. Convex B. convex+ strictly convex

C. concave D. convex + concave
E. none of these properties
𝐴
21
22
 Convex optimization
minimizex1,x2 f0(x) = x12 + x22
subject to f1(x) = x1 / (1 + x22) ≤ 0
h1(x) = (x1 + x2) = 0
2

• Is this a convex problem?

• 0 is convex; feasible set {( 1, 2) 1 = − 2 ≤ 0} is convex

• Not a convex problem (according to our definition): 1 is not convex, h1 is

not affine

• Equivalent (but not identical) to the convex problem

minimizex1,x2 x12 + x22
subject to x1 ≤ 0
x1 + x2 = 0

23
𝑥
𝑥
𝑥
𝑥
𝑓
𝑓
 • Given a differentiable function : ℝ → ℝ, how do you find its
global minimum?

24
𝑓
 • Given a differentiable function : ℝ → ℝ, how do you find its
global minimum?
• The global minimum can only occur in one of three places:

• An such that ′( ) = 0

• In the lim
→∞
( )

lim ( )
• In the →−∞

• Note that in the last two cases, the global minimum need not be
achievable at any finite

25
𝑥
𝑥
𝑓
𝑥
𝑥𝑓

𝑥
𝑓
𝑓
𝑥
𝑥
Local Optima
• Critical points can be local minima, local maxima, or saddle
points

26
Local Optima
• Critical points can be local minima, local maxima, or saddle
points

source:27Wikipedi
 Optimality criterion for differentiable 0
Optimality criterion for differentiable f0
• convex problem: is optimal if and only if it is feasible and
mal if and only if it is feasible and
∇f(x)(y − x) ≥ 0 for all feasible
∇f0(x)T (y − x) ≥ 0 for all feasible y
• unconstrained problem: is optimal if and only if
x ∈ dom f, ∇f(x) = 0

In words: all feasible directions

−∇f0(x)
X
x from are aligned with gradient
∇ ( )

Note athat
o, ∇f0(x)• defines supporting ( ) ( to−feasible
if ∇ hyperplane ) <set0 X
foratsome
x ∈ , then it must
correspond to a descent direction, i.e., if we take a small enough
tion problems
step in this direction, the function will 4–9
decrease in value
𝑓
28
𝑓
𝑥
𝑦
𝑥
𝑦
𝐶
𝑦𝑥
𝑥
𝑓
𝑥
𝑥
𝑇
 Least Squares Regression

∑(
+ ).
2
Let us denote = −
=1

∑ =1 (
• =2 − + )

∑ =1 ∑ =1
• =0⟹ = −

∑ =1 (
• = 2 − + )(− )

∑ =1 ( + )
=0⟹ =
• ∑ =1 2

• Note: for complicated cases, it is not easy to have closed form solutions.
𝑖
𝑖
𝑖
𝑥
𝜕
𝑎
𝜕
𝜕
𝜕
𝑎
𝑏
𝑏
𝑛
𝑖
𝑎
𝑖
𝑖
𝑖
𝑖
𝑖
𝑖
𝑖
𝑓
𝑦
𝑎
𝑥
𝑏
29
𝑏
𝑦
𝑎
𝑥
𝑏
𝑖
𝑖
𝑖
𝑖
𝑖
𝑖
𝑖
𝑖
𝜕
𝑓
𝑎
𝑦
𝑥
𝑎
𝑥
𝑏
𝑦
𝑥
𝑦
𝑏
𝑥
𝜕
𝜕
𝜕
𝑓
𝑓
𝑓
𝑛
𝑛
𝑛
𝑛
𝑛
𝑛
 n
X 2
= w0 + w T xi yi = f (w0 , w) .

Least Squares Regression

⇤
i=1

4 LGT2 for convex basics

• Given data points 1 , …, ∈ and y1, …, ∈ , find
If x 2 Rk , a 2 Rk , let us drop b.
the best fit line
n
X 2
min f = yi a> xi (1)
i=1

Xn
@f
=2 yi a> xi · ( xi )
@a i=1

0 1
x>
1
B .. C >
Xn⇥k =@ . A , Yn⇥1 = [y1 , . . . , yn ] (2)
x>
n n⇥k
(3)

@f 1
= 0 ) a = X >X XY
@a
matrix inverse is expensive!! We could use gradient descent!
Matrix inverse is expensive!!
We could use gradient descent!
𝑛
30
𝑛
𝑥
𝑥
𝑅
𝑦
𝑅
𝑘
Summary
• Convex sets & Key properties
• Convex functions & Operations preserving convexity
• Convex optimization and optimality

31
32