Convex functions
Renu M. R.
March 5, 2020
1 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)
2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)
f is strictly convex if a strict inequality holds for x 6= y and
0 < α < 1.
2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)
f is strictly convex if a strict inequality holds for x 6= y and
0 < α < 1.
f is concave if −f is convex.
2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)
f is strictly convex if a strict inequality holds for x 6= y and
0 < α < 1.
f is concave if −f is convex.
This inequality also goes by the name Jensen’s inequality
2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)
f is strictly convex if a strict inequality holds for x 6= y and
0 < α < 1.
f is concave if −f is convex.
This inequality also goes by the name Jensen’s inequality
Examples?
2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)
f is strictly convex if a strict inequality holds for x 6= y and
0 < α < 1.
f is concave if −f is convex.
This inequality also goes by the name Jensen’s inequality
Examples?
How to check for convexity?
2 / 11
First order condition
3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?
3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?
Let f be differentiable, f is convex iff,
3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?
Let f be differentiable, f is convex iff,
3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?
Let f be differentiable, f is convex iff,
1 Domain of f (C) is a convex set, and
3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?
Let f be differentiable, f is convex iff,
1 Domain of f (C) is a convex set, and
2 ∀ x, y ∈ C,
f (y) ≥ f (x) + (y − x)T ∇f (x).
3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?
Let f be differentiable, f is convex iff,
1 Domain of f (C) is a convex set, and
2 ∀ x, y ∈ C,
f (y) ≥ f (x) + (y − x)T ∇f (x).
Proof?
3 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x) 0, ∀x ∈ C.
4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x) 0, ∀x ∈ C.
Symbol indicates psd.
4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x) 0, ∀x ∈ C.
Symbol indicates psd.
If Hessian is pd ∀x, then f is strictly convex.
4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x) 0, ∀x ∈ C.
Symbol indicates psd.
If Hessian is pd ∀x, then f is strictly convex.
Is the converse true for strictly convex functions?
4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x) 0, ∀x ∈ C.
Symbol indicates psd.
If Hessian is pd ∀x, then f is strictly convex.
Is the converse true for strictly convex functions? No.
4 / 11
Problems
[P1] Check if the following functions (which we saw in the
previous classes) are convex?
1 f (x) =k Ax − b k2 ,
A ∈ Rm×n , m > n. A is full rank.
3 3 x
2 f (x) = x − x + y − y, where x = .
y
x6
3 f (x) = 2x2 − 1.05x4 + + xy + y 2 .
6
[P2] When is the following function convex? (x ∈ Rn , P is
symmetric)
1
f (x) = xT P x + qT x + r.
2
[P3] Show that linear and affine functions are convex.
5 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.
Proof.
6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.
Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ ) 0.
6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.
Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ ) 0.
f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,
6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.
Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ ) 0.
f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,
S = {x | k x − x∗ k< }.
6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.
Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ ) 0.
f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,
S = {x | k x − x∗ k< }.
Let y ∈ C and z = x∗ + α(y − x∗ ), 0 ≤ α ≤ 1.
6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.
Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ ) 0.
f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,
S = {x | k x − x∗ k< }.
Let y ∈ C and z = x∗ + α(y − x∗ ), 0 ≤ α ≤ 1.
Choose α small enough so that z ∈ S,
6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.
Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ ) 0.
f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,
S = {x | k x − x∗ k< }.
Let y ∈ C and z = x∗ + α(y − x∗ ), 0 ≤ α ≤ 1.
Choose α small enough so that z ∈ S,
Use convexity to argue that x∗ is global a minimum
6 / 11
Coercive function
7 / 11
Coercive function
A continuous function f : Rn → R is coercive if
lim f (x) → ∞.
kxk→∞
7 / 11
Coercive function
A continuous function f : Rn → R is coercive if
lim f (x) → ∞.
kxk→∞
Check if the following functions are coercive.
1 f (x, y) = x2 + y 2
2 f (x, y) = x2 − 2xy + y 2
3 f (x, y) = x4 − 4xy + y 4
7 / 11
Coercive function
A continuous function f : Rn → R is coercive if
lim f (x) → ∞.
kxk→∞
Check if the following functions are coercive.
1 f (x, y) = x2 + y 2
2 f (x, y) = x2 − 2xy + y 2
3 f (x, y) = x4 − 4xy + y 4
Let a continuous function f (x) be coercive. Then f (x) has
at least one global minimum.
7 / 11
Additional problems
Check if the following functions are convex
P1 eax a, x ∈ R
P2 xa a, x ∈ R
P3 log x 0 6= x ∈ R
P4 x log x x≥0
P5 f (x) = max xi x ∈ Rn
i=1···n
P6 f (x, y) = x2 /y y > 0.
8 / 11
Additional problems
P7 Use the notion of convexity to prove
1 the AM-GM inequality
2 that for a convex function f , f (E(X)) ≤ E(f (X)), where X is a
discrete random variable and E(X) is the expectation of X.
P8 Use the AM-GM inequality to solve the following:
maximize f (x) = x1 x2 x3
subject to x1 x2 + 2x2 x3 + 2x1 x3 = S.
9 / 11
Proof slide
10 / 11
Proof
First order condition
11 / 11
Proof
First order condition
Say f (x) is convex.
11 / 11
Proof
First order condition
Say f (x) is convex.
Use the definition of directional derivative to prove the inequality.
11 / 11
Proof
First order condition
Say f (x) is convex.
Use the definition of directional derivative to prove the inequality.
Say, the inequality is true.
11 / 11
Proof
First order condition
Say f (x) is convex.
Use the definition of directional derivative to prove the inequality.
Say, the inequality is true.
Consider three points, x, y, and z, where y is a convex
combination of x and z. Using the gradient at y, show that f is
convex.
11 / 11
Proof
First order condition
Say f (x) is convex.
Use the definition of directional derivative to prove the inequality.
Say, the inequality is true.
Consider three points, x, y, and z, where y is a convex
combination of x and z. Using the gradient at y, show that f is
convex.
Second order condition
11 / 11
Proof
First order condition
Say f (x) is convex.
Use the definition of directional derivative to prove the inequality.
Say, the inequality is true.
Consider three points, x, y, and z, where y is a convex
combination of x and z. Using the gradient at y, show that f is
convex.
Second order condition
Use Taylor series expansion and use the first order condition.
Back to first order conditions.
11 / 11