MAEG4070 Engineering Optimization

Lecture 4 Convex Sets & Convex Functions

Yue Chen
MAE, CUHK
email: [email protected]
Sep 16, 2021

1
Content of this course (tentative)
Linear programming
Lecture 2
Linear
Dual Theory – Part I
Lecture 3
Linearization
technique Programming
Lecture 7 Unconstrained optimization Lecture 13
Lecture 5 & 6

Optimization Non-Linear Constrained optimization

Lecture 8 & 9
Lecture 1 Lecture 4
Dual Theory – Part II Engineering examples
Lecture 10
Lecture 14

Distributed optimization Lecture 11

Multi-objective optimization Lecture 12

Theory
2
Affine Sets

line passing through points 𝑥𝑥1 and 𝑥𝑥2

𝑦𝑦 = 𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑥𝑥2 , ∀𝜃𝜃 ∈ ℝ

line segment between points 𝑥𝑥1 and 𝑥𝑥2

𝑦𝑦 = 𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑥𝑥2 , ∀𝜃𝜃 ∈ [0,1]

Affine set: the set that contains all line through any two distinct points in
the set 𝒞𝒞
∀𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞, 𝜃𝜃 ∈ ℝ ⇒ 𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒞𝒞

3
Pictures from Google image
Affine Sets

Example: solution set of linear equations 𝒳𝒳 = {𝑥𝑥|𝐴𝐴𝐴𝐴 = 𝑏𝑏} is an affine set.

Proof: Given any two points 𝑥𝑥1 ∈ 𝒳𝒳 and 𝑥𝑥2 ∈ 𝒳𝒳, for any 𝜃𝜃 ∈ ℝ,
then 𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 represents a point on the line crossing 𝑥𝑥1 and 𝑥𝑥2 .

Since 𝐴𝐴𝐴𝐴1 = 𝑏𝑏 and 𝐴𝐴𝑥𝑥2 = 𝑏𝑏, we have

𝐴𝐴 𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑥𝑥2 = 𝜃𝜃𝐴𝐴𝐴𝐴1 + 1 − 𝜃𝜃 𝐴𝐴𝐴𝐴2 = 𝑏𝑏

Therefore, 𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒳𝒳.

4
Affine Sets

Example: solution set of linear equations 𝒳𝒳 = {𝑥𝑥|𝐴𝐴𝐴𝐴 = 𝑏𝑏}

Affine combination of points 𝑥𝑥1 , … , 𝑥𝑥𝑛𝑛 is

𝑥𝑥 = 𝜃𝜃1 𝑥𝑥1 + ⋯ + 𝜃𝜃𝑛𝑛 𝑥𝑥𝑛𝑛 , 𝜃𝜃1 + ⋯ + 𝜃𝜃𝑛𝑛 = 1

Affine hull of set 𝒞𝒞 is

• The smallest affine set that contains 𝒞𝒞
• Set of all affine combinations of points in 𝒞𝒞

5
Pictures from Google image
Convex Sets

Convex set: the set that contains all line segment between any two
distinct points in the set 𝒞𝒞 1>=Theta >=0
∀𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞, 𝜃𝜃 ∈ [0,1] ⇒ 𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒞𝒞
Points inside Line segment (1>=theta>=0)
Set C area within Set C area

Y
N
Intuitive explanation: in a convex set, you can see everywhere
wherever you stand
Try it yourself: Are the following sets convex?

Y N N
6
Pictures from Google image
Convex Sets

Convex combination of points 𝑥𝑥1 , … , 𝑥𝑥𝑛𝑛 is

𝑥𝑥 = 𝜃𝜃1 𝑥𝑥1 + ⋯ + 𝜃𝜃𝑛𝑛 𝑥𝑥𝑛𝑛 , 𝜃𝜃1 + ⋯ + 𝜃𝜃𝑛𝑛 = 1, 𝜃𝜃𝑘𝑘 ≥ 0, ∀𝑘𝑘 = 1, … , 𝑛𝑛

Convex hull of set 𝒞𝒞 is

• The smallest convex set that contains 𝒞𝒞
• Set of all convex combinations of points in 𝒞𝒞

7
Pictures from Google image
Cones
𝜃𝜃1 𝑥𝑥1 ∈ 𝒞𝒞 (cone)
Cone: if for every 𝑥𝑥 ∈ 𝒞𝒞 and 𝜃𝜃 ≥ 0, we have 𝜃𝜃𝑥𝑥 ∈ 𝒞𝒞 𝜃𝜃2 𝑥𝑥2 ∈ 𝒞𝒞 (cone)
∀𝑥𝑥 ∈ 𝒞𝒞, 𝜃𝜃 ≥ 0 ⇒ 𝜃𝜃𝑥𝑥 ∈ 𝒞𝒞 0.5𝜃𝜃1 𝑥𝑥1 + 0.5𝜃𝜃2 𝑥𝑥2 ∈ 𝐶𝐶 (convex)
2(0.5𝜃𝜃1 𝑥𝑥1 + 0.5𝜃𝜃2 𝑥𝑥2 ) ∈ 𝐶𝐶 (cone)
Convex cone: if 𝒞𝒞 is convex and also a cone
∀𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞, 𝜃𝜃1 , 𝜃𝜃2 ≥ 0 ⇒ 𝜃𝜃1 𝑥𝑥1 + 𝜃𝜃2 𝑥𝑥2 ∈ 𝒞𝒞

8
Pictures from Google image
Cones

Conic combination of points 𝑥𝑥1 , … , 𝑥𝑥𝑛𝑛 is

𝑥𝑥 = 𝜃𝜃1 𝑥𝑥1 + ⋯ + 𝜃𝜃𝑛𝑛 𝑥𝑥𝑛𝑛 , 𝜃𝜃𝑘𝑘 ≥ 0, ∀𝑘𝑘 = 1, … , 𝑛𝑛

Conic hull of set 𝒞𝒞 is

• The smallest cone that contains 𝒞𝒞
• Set of all conic combinations of points in 𝒞𝒞

9
Pictures from Google image
Comparison of affine set, convex set, cone

Affine set: ∀𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞, 𝜃𝜃 ∈ ℝ ⇒ 𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒞𝒞

Convex set: ∀𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞, 𝜃𝜃 ∈ [0,1] ⇒ 𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒞𝒞
Cone: ∀𝑥𝑥 ∈ 𝒞𝒞, 𝜃𝜃 ≥ 0 ⇒ 𝜃𝜃𝑥𝑥 ∈ 𝒞𝒞

Try to identify: are these affine set, convex set, or cone?

cone convex set affine set cone

10
Pictures from Google image
Polyhedron

Hyperplane: set of the form 𝑥𝑥 𝑎𝑎𝑇𝑇 𝑥𝑥 = 𝑏𝑏, 𝑎𝑎 ≠ 0}

Hyperplane is affine and convex

Halfspace: set of the form 𝑥𝑥 𝑎𝑎𝑇𝑇 𝑥𝑥 ≤ 𝑏𝑏, 𝑎𝑎 ≠ 0}

halfspace is convex

11
Pictures from Google image
Polyhedron

Hyperplane: set of the form 𝑥𝑥 𝑎𝑎𝑇𝑇 𝑥𝑥 = 𝑏𝑏, 𝑎𝑎 ≠ 0}

Hyperplane is affine and convex

Proof: We have proved that 𝑆𝑆 = 𝑥𝑥 𝑎𝑎𝑇𝑇 𝑥𝑥 = 𝑏𝑏, 𝑎𝑎 ≠ 0} is an affine set.

Next, let’s prove it is convex. Suppose we have two vectors 𝑥𝑥1 , 𝑥𝑥2 , then for any
𝜃𝜃 ∈ [0,1], we have
𝑎𝑎 𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑥𝑥2 = 𝜃𝜃𝑎𝑎𝑥𝑥1 + 1 − 𝜃𝜃 𝑎𝑎𝑥𝑥2 = 𝜃𝜃𝑏𝑏 + 1 − 𝜃𝜃 𝑏𝑏 = 𝑏𝑏

Therefore, we have 𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑥𝑥2 ∈ 𝑆𝑆.

12
Pictures from Google image
Polyhedron
Halfspace: set of the form 𝑥𝑥 𝑎𝑎𝑇𝑇 𝑥𝑥 ≤ 𝑏𝑏, 𝑎𝑎 ≠ 0}

halfspace is convex

Proof: Suppose we have two vectors 𝑥𝑥1 , 𝑥𝑥2 , then for any 𝜃𝜃 ∈ [0,1], we have

𝑎𝑎 𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑥𝑥2 = 𝜃𝜃𝑎𝑎𝑥𝑥1 + 1 − 𝜃𝜃 𝑎𝑎𝑥𝑥2 ≤ 𝜃𝜃𝑏𝑏 + 1 − 𝜃𝜃 𝑏𝑏 = 𝑏𝑏

Therefore, we have 𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑥𝑥2 ∈ 𝑆𝑆. not affine because we need 𝜃𝜃 ≥ 0

13
Pictures from Google image
Polyhedron

A polyhedron is defined as the solution set of linear equalities and

inequalities
𝑃𝑃 = {𝑥𝑥|𝑎𝑎𝑗𝑗𝑇𝑇 𝑥𝑥 ≤ 𝑏𝑏𝑗𝑗 , 𝑗𝑗 = 1, … , 𝑚𝑚, 𝑐𝑐𝑘𝑘𝑇𝑇 𝑥𝑥 = 𝑑𝑑𝑘𝑘 , 𝑘𝑘 = 1, … , 𝑝𝑝}
Or in a compact form
𝑃𝑃 = {𝑥𝑥|𝐴𝐴𝐴𝐴 ≤ 𝑏𝑏, 𝐶𝐶𝐶𝐶 = 𝑑𝑑}

Polyhedron is intersection of finite number of halfspaces and hyperplanes

14
Pictures from Google image
How to prove a set 𝓒𝓒 is convex?

To prove a set 𝒞𝒞 is convex, we can

two points x1 and x2
• Apply definition
∀𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞, 𝜃𝜃 ∈ [0,1] ⇒ 𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒞𝒞

• Show that 𝒞𝒞 is obtained from simple convex sets by operations that

preserve convexity, e.g.
 Intersection
 Affine mapping
 Perspective mapping
 Linear-fractional mapping
 epigraph

15
Example - 1 x1 and x2 is similar to x and y coordinates of a point
y (y1, y2) is a point
Prove that the set 𝑆𝑆 = 𝑥𝑥1 , 𝑥𝑥2 𝑥𝑥1 + 𝑥𝑥2 ≤ 6, −𝑥𝑥1 + 2𝑥𝑥2 ≤ 1} is convex.

Proof: Given any 𝑥𝑥 = 𝑥𝑥1 , 𝑥𝑥2 ∈ 𝑆𝑆, 𝑦𝑦 = (𝑦𝑦1 , 𝑦𝑦2 ) ∈ 𝑆𝑆. For any 𝜃𝜃 ∈ [0,1], we have
𝜃𝜃𝑥𝑥 + 1 − 𝜃𝜃 𝑦𝑦 = (𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑦𝑦1 , 𝜃𝜃𝑥𝑥2 + 1 − 𝜃𝜃 𝑦𝑦2 )

Then we check 𝜃𝜃𝑥𝑥 + 1 − 𝜃𝜃 𝑦𝑦 is in 𝑆𝑆

𝜃𝜃𝑥𝑥1 + 1 − 𝜃𝜃 𝑦𝑦1 + 𝜃𝜃𝑥𝑥2 + 1 − 𝜃𝜃 𝑦𝑦2 = 𝜃𝜃 𝑥𝑥1 + 𝑥𝑥2 + (1 − 𝜃𝜃)(𝑦𝑦1 + 𝑦𝑦2 ) ≤ 6

−𝜃𝜃𝑥𝑥1 − 1 − 𝜃𝜃 𝑦𝑦1 + 2𝜃𝜃𝑥𝑥2 + 2 1 − 𝜃𝜃 𝑦𝑦2 ≤ 𝜃𝜃 −𝑥𝑥1 + 2𝑥𝑥2 + (1 − 𝜃𝜃)(−𝑦𝑦1 + 2𝑦𝑦2 ) ≤ 1

Therefore,𝜃𝜃𝑥𝑥 + 1 − 𝜃𝜃 𝑦𝑦 ∈ 𝑆𝑆.

16
Example -2 Positive semidefinite cone

Let 𝑺𝑺𝑛𝑛 be a set of symmetric 𝑛𝑛 × 𝑛𝑛 matrices, define

Convex cone (try to prove it)

Then we have definition of a positive semidefinite cone

17
Pictures from Google image
Example -2 Positive semidefinite cone

Proof: 1) Convex

18
Example -2 Positive semidefinite cone

Proof: 2) Cone

19
Example – 3 Intersection of two sets

Suppose we have two convex sets 𝒞𝒞 and 𝒮𝒮. Then, let’s prove 𝒞𝒞 ∩ 𝒮𝒮 is a convex set.

Proof:
Given any 𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞 ∩ 𝒮𝒮, then we have 𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒞𝒞 and 𝑥𝑥1 , 𝑥𝑥2 ∈ 𝒮𝒮
Since 𝒞𝒞 and 𝒮𝒮 are all convex sets, then given any 𝜃𝜃 ∈ 0,1
We have
𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒞𝒞
𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒮𝒮
Therefore,
𝜃𝜃𝑥𝑥1 + (1 − 𝜃𝜃)𝑥𝑥2 ∈ 𝒞𝒞 ∩ 𝒮𝒮

20
Convex function

Function f : ℝ𝑛𝑛 → ℝ is convex if dom(f) is a convex set, and the following

inequality holds

If we change ≤ into ≥, then it is concave

21
Pictures from Google image
Convex function

Function f : ℝ𝑛𝑛 → ℝ is strictly convex if dom(f) is a convex set, and the

following inequality holds

2
Function f is strongly convex if ∃𝛼𝛼 > 0: 𝑓𝑓 𝑥𝑥 − 𝛼𝛼 𝑥𝑥 2 is convex

f is (strictly, strongly) concave if –f is (strictly, strongly) convex

f(x) - x^2 [a convex function],
then will become strongly convex

About Gradient

stronger
22
Convex function
.
.

Apart from proving the convexity by definition, in the following, we provide

.

two conditions, i.e. first-order condition & second-order condition

1. Suppose f is differentiable and ∇𝑓𝑓(𝑥𝑥) exists at each 𝑥𝑥 ∈ 𝑑𝑑𝑑𝑑𝑑𝑑(𝑓𝑓) About Hessian Matrix
d^2f d^2f
2. First-order condition f with convex domain is convex iff dx1^2 … dx1 dxn

d^2f d^2f
…
height of f(y) must be taller dxn dx1 dxn^2
than the tangent height at y
position

height

23
x y Pictures from Google image
Convex function
Simple proof:

24
Convex function

× 𝒕𝒕 p.23
(z, f(z))
× (𝟏𝟏 − 𝒕𝒕)

x y

25
Convex function

Suppose f is twice differentiable and the Hessian 𝐻𝐻(𝑥𝑥) exists at every

𝑥𝑥 ∈ 𝑑𝑑𝑑𝑑𝑑𝑑(𝑓𝑓).

Second-order condition function f with convex domain is

• convex iff

• Strictly convex iff

• Strongly convex iff

26
Example
Convex functions
• Affine: 𝑎𝑎𝑎𝑎 + 𝑏𝑏 on ℝ for any a and b
• Quadratic function: 𝑎𝑎𝑥𝑥 2 + 𝑏𝑏𝑏𝑏 + 𝑐𝑐 on ℝ for any 𝑎𝑎 ≥ 0
• Exponential: 𝑒𝑒 𝑎𝑎𝑎𝑎 on ℝ for any a
• Negative entropy: 𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥(𝑥𝑥) on ℝ++

Try to prove 𝑓𝑓 𝑥𝑥 = 𝑎𝑎𝑥𝑥 2 + 𝑏𝑏𝑏𝑏 + 𝑐𝑐 on ℝ for any 𝑎𝑎 ≥ 0 is convex.

Proof:
𝑓𝑓 ′ 𝑥𝑥 = 2𝑎𝑎𝑎𝑎 + 𝑏𝑏, 𝑓𝑓 ′′ 𝑥𝑥 = 2𝑎𝑎 ≥ 0
According to the second-order condition, it is convex.

27
Epigraph
The graph of a function 𝑓𝑓: ℝ𝑛𝑛 → ℝ is defined as
𝑥𝑥, 𝑓𝑓 𝑥𝑥 𝑥𝑥 ∈ 𝑑𝑑𝑑𝑑𝑑𝑑 𝑓𝑓 } ⊆ ℝ𝑛𝑛+1

The epigraph of a function 𝑓𝑓: ℝ𝑛𝑛 → ℝ is defined as

𝑒𝑒𝑒𝑒𝑒𝑒 𝑓𝑓 = 𝑥𝑥, 𝑡𝑡 𝑥𝑥 ∈ 𝑑𝑑𝑑𝑑𝑑𝑑 𝑓𝑓 , 𝑓𝑓(𝑥𝑥) ≤ 𝑡𝑡} ⊆ ℝ𝑛𝑛+1

𝑓𝑓(𝑥𝑥) is convex if and only if 𝑒𝑒𝑒𝑒𝑒𝑒 𝑓𝑓 is a convex set.

28
Pictures from Google image
How to prove a function 𝒇𝒇(𝒙𝒙) is convex?

To prove a function 𝑓𝑓(𝑥𝑥) is convex, we can

• Verify definition

• For twice differentiable functions, apply second-order condition

• Show that 𝑓𝑓(𝑥𝑥) is obtained from simple convex functions by operations that
preserve convexity, e.g.
 Nonnegative weighted sum
 Composition with affine function
 Pointwise maximum
 Composition
 Minimization
 Perspective
29
Example - 1

Prove that 𝑓𝑓 𝑥𝑥1 , 𝑥𝑥2 = 𝑥𝑥12 − 2𝑥𝑥1 𝑥𝑥2 + 4𝑥𝑥22 + 3𝑥𝑥1 is convex.

Proof: The gradient of 𝑓𝑓 𝑥𝑥1 , 𝑥𝑥2 = 𝑥𝑥12 − 2𝑥𝑥1 𝑥𝑥2 + 4𝑥𝑥22 + 3𝑥𝑥1 is

𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕
= 2𝑥𝑥1 − 2𝑥𝑥2 + 3, = −2𝑥𝑥1 + 8𝑥𝑥2
𝜕𝜕𝑥𝑥1 𝜕𝜕𝑥𝑥2

The Hessian is
2 −2
𝐻𝐻 𝑥𝑥 =
−2 8
𝐻𝐻(𝑥𝑥) is positive semi-definite. Therefore, 𝑓𝑓 𝑥𝑥1 , 𝑥𝑥2 is a convex function.

30
Example – 2 Pointwise maximum of convex functions

Suppose the maximum is the 𝑖𝑖 ∗ term

max{𝑓𝑓𝑖𝑖 (𝑥𝑥)} ≥ 𝑓𝑓𝑖𝑖 ∗ (𝑥𝑥)
𝑖𝑖
max{𝑓𝑓𝑖𝑖 (𝑦𝑦)} ≥ 𝑓𝑓𝑖𝑖 ∗ (𝑦𝑦)
𝑖𝑖

Example: piecewise-linear functions

31
Example – 3 Minimization

32
Thanks！

33