An Introduction to Non-smooth Optimization

Lecture 01 - Mathematical Background

Jingwei LIANG
Institute of Natural Sciences, Shanghai Jiao Tong University

Email: [email protected]
Office: Room 355, No. 6 Science Building
Outline

1 Vector spaces

2 Convex sets

3 Non-expansive operators

4 Fejér monotonicity

5 Convex functions

6 Differentiability

7 Convex minimization problem

Vector
Let Rn be the n-dimensional real vector space, a column vector of Rn is denoted by a ∈ Rn , with
 
a1
a2 
a =  . .
 
 .. 
an
The number ai is called the i’th element/component of the vector a .

NB: By default we refer vector as column vector.

Vector spaces Definitions 2/22

Matrix
A matrix with m rows and n columns is called an m × n matrix and denoted by
 
a1,1 a1,2 . . . a1,n
 a2,1 a2,2 . . . a2,n 
m×n
A= . ..  ∈ R .
 
.. ..
 .. . . . 
am,1 am,2 . . . am,n

The identity matrix of size n is a diagonal matrix

 
1
 1 
Id n =  .
 
.. 
 . 
1 n×n

Vector spaces Definitions 3/22

Vector norm

Definition - Vector inner product

Let x , y ∈ Rn , their inner product or dot product returns a scalar
n
X
hxx | y i = xi yi .
i=1

Alternative notation
x Ty .

Given any x , y ∈ Rn , their distance is

p
||xx − y || = hxx − y | x − y i.

Vector spaces Norms 4/22

Vector norm

Definition - Vector p-norm

Let x ∈ Rn be a vector and p ≥ 1, then the p-norm (also called `p -norm) of x is defined by
n
!1/p
X
||xx ||p = |xi |p .
i=1

Vector spaces Norms 4/22

Vector norm

Definition - Vector p-norm

Let x ∈ Rn be a vector and p ≥ 1, then the p-norm (also called `p -norm) of x is defined by
n
!1/p
X
||xx ||p = |xi |p .
i=1

A norm must satisfies

Positivity: ||xx ||p ≥ 0, ||xx ||p = 0 if and only if x = 0.
Homogeneity: ||rxx ||p = |r|||xx ||p , r ∈ R.
Triangle inequality: ||xx + y ||p ≤ ||xx ||p + ||yy ||p .

Vector spaces Norms 4/22

Vector norm

Example - `2 -norm (Euclidean norm)

Let p = 2 we obtain the Euclidean norm of x
qP
n √
||xx ||2 = |x |2 = x T x .
i=1 i

||xx || without subscript 2 is also used to denote `2 -norm.

Vector spaces Norms 4/22

Vector norm

Example - `2 -norm (Euclidean norm)

Let p = 2 we obtain the Euclidean norm of x
qP
n √
||xx ||2 = |x |2 = x T x .
i=1 i

||xx || without subscript 2 is also used to denote `2 -norm.

Example - `1 -norm
Let p = 1 we obtain the `1 -norm of x
Pn
||xx ||1 = i=1
|xi |.

Vector spaces Norms 4/22

Vector norm

Example - `2 -norm (Euclidean norm)

Let p = 2 we obtain the Euclidean norm of x
qP
n √
||xx ||2 = |x |2 = x T x .
i=1 i

||xx || without subscript 2 is also used to denote `2 -norm.

Example - `1 -norm
Let p = 1 we obtain the `1 -norm of x
Pn
||xx ||1 = i=1
|xi |.

Example - `∞ -norm
The infinity norm of x is defined by
||xx ||∞ = max |xi |.
i=1,...,n

Vector spaces Norms 4/22

Vector norm

Definition - Vector inner product

Let x , y ∈ Rn , their inner product or dot product returns a scalar
n
X
hxx | y i = xi yi .
i=1

Theorem - Cauchy-Schwarz inequality

For any two vectors x and y in Rn , the Cauchy-Schwarz inequality
|hxx | y i| ≤ ||xx ||||yy ||
holds. Furthermore, equality holds if and only if x = αyy for some α ∈ R.

Vector spaces Norms 4/22

Dual norm

Definition - Dual norm

Let || · || be a norm defined on Rn , the associated dual norm, denoted by || · ||∗ is defined as
||vv ||∗ = sup hvv | x i : ||xx || ≤ 1 .


Vector spaces Norms 5/22

Dual norm

Definition - Dual norm

Let || · || be a norm defined on Rn , the associated dual norm, denoted by || · ||∗ is defined as
||vv ||∗ = sup hvv | x i : ||xx || ≤ 1 .


𝑣
The dual of the Euclidean norm is the Euclidean
norm
n o
sup hvv | x i : ||xx ||2 ≤ 1 = ||vv ||2 .

Vector spaces Norms 5/22

Dual norm

Definition - Dual norm

Let || · || be a norm defined on Rn , the associated dual norm, denoted by || · ||∗ is defined as
||vv ||∗ = sup hvv | x i : ||xx || ≤ 1 .


𝑣
The dual of the `1 -norm is the `∞ -norm
n o
sup hvv | x i : ||xx ||1 ≤ 1 = ||vv ||∞ .
Recall that in R2
hvv | x i = v1 x1 + v2 x2 .

Vector spaces Norms 5/22

Dual norm

Definition - Dual norm

Let || · || be a norm defined on Rn , the associated dual norm, denoted by || · ||∗ is defined as
||vv ||∗ = sup hvv | x i : ||xx || ≤ 1 .


𝑣
The dual of the `∞ -norm is the `1 -norm
n o
sup hvv | x i : ||xx ||∞ ≤ 1 = ||vv ||1 .
Recall that in R2
hvv | x i = v1 x1 + v2 x2 .

Vector spaces Norms 5/22

Dual norm

Proposition - Dual norm

Given p, q ≥ 1, `p -norm and `q -norm are dual of each other if
1 1
+ = 1.
p q

Vector spaces Norms 5/22

Dual norm

Proposition - Dual norm

Given p, q ≥ 1, `p -norm and `q -norm are dual of each other if
1 1
+ = 1.
p q

Theorem - generalized Cauchy-Schwarz inequality

Given any nonzero x ∈ Rn and v ∈ Rn , there holds
hvv | x /||xx ||i ≤ sup hvv | y i : ||yy || ≤ 1 = ||vv ||∗ =⇒

hvv | x i ≤ ||vv ||∗ ||xx ||
which holds for all v and x .

The inequality is tight in the sense that, for any x there exists a v such that the equality
holds, and vice versa.

Vector spaces Norms 5/22

Convex sets
Definition, closedness
Convex sets

Definition - Convex set

A subset S of Rn is convex if for any x , y ∈ S and λ ∈ [0, 1], there holds
λxx + (1 − λ)yy ∈ S.
λxx + (1 − λ)yy is called the convex combination of x and y .

Convex sets 6/22

Convex sets

Definition - Convex set

A subset S of Rn is convex if for any x , y ∈ S and λ ∈ [0, 1], there holds
λxx + (1 − λ)yy ∈ S.
λxx + (1 − λ)yy is called the convex combination of x and y .

Example - Hyper plane and half space

Given a ∈ Rn and b ∈ R,
Hyper plane
def
H = x : aTx = b .


Half space
def
H = x : aT x ≤ b .


Convex sets 6/22

Convex sets

Proposition - Some properties


Let S be a convex set, then βS = βxx : x ∈ S is convex.
Let Si , i = 1, 2, ..., m be a family of convex sets, then
\
Si
i=1,2,...,m
is convex.
Let S1 , S2 be two convex sets, then
S1 + S2 and S1 − S2
are convex.

Convex sets 6/22

Open and closed sets

Definition - Interior point

An element x ∈ S ⊂ Rn is called an interior point of S if there ∃ > 0 for which
n o
y : ||yy − x || ≤  ⊂ S.

The interior of S, i.e. int(S), denotes the set of all interior points of S.

Convex sets 7/22

Open and closed sets

Definition - Interior point

An element x ∈ S ⊂ Rn is called an interior point of S if there ∃ > 0 for which
n o
y : ||yy − x || ≤  ⊂ S.

The interior of S, i.e. int(S), denotes the set of all interior points of S.

A set S is open if int(S) = S, it is closed if

Rn \ S = x ∈ Rn : x ∈

/S
is open.

Convex sets 7/22

Open and closed sets

Definition - Interior point

An element x ∈ S ⊂ Rn is called an interior point of S if there ∃ > 0 for which
n o
y : ||yy − x || ≤  ⊂ S.

The interior of S, i.e. int(S), denotes the set of all interior points of S.

A set S is open if int(S) = S, it is closed if

Rn \ S = x ∈ Rn : x ∈

/S
is open.

The closure and boundary of S are defined as

cl(S) = Rn \ int(Rn \ S) and bd(S) = cl(S) \ int(S).

Convex sets 7/22

Non-expansive operators
Non-expansiveness and fixed-point
Non-expansive operator

Definition - Non-expansive operator

Let S be a non-empty subset of Rn and let F : S → Rn . Then F is non-expansive if it is Lipschitz
continuous with constant 1, i.e.
(∀xx , y ∈ S) ||F(xx ) − F(yy )|| ≤ ||xx − y ||.

Non-expansive operators 8/22

Non-expansive operator

Definition - Non-expansive operator

Let S be a non-empty subset of Rn and let F : S → Rn . Then F is non-expansive if it is Lipschitz
continuous with constant 1, i.e.
(∀xx , y ∈ S) ||F(xx ) − F(yy )|| ≤ ||xx − y ||.

Definition - Firmly non-expansive operator

F is firmly non-expansive if
(∀xx , y ∈ S) ||F(xx ) − F(yy )||2 + ||(IId − F)(xx ) − (IId − F)(yy )||2 ≤ ||xx − y ||2 .

The following are equivalent

F is firmly non-expansive.
Id − F is firmly non-expansive.
2F − Id is non-expansive.

Non-expansive operators 8/22

Non-expansive operator

Definition - Non-expansive operator

Let S be a non-empty subset of Rn and let F : S → Rn . Then F is non-expansive if it is Lipschitz
continuous with constant 1, i.e.
(∀xx , y ∈ S) ||F(xx ) − F(yy )|| ≤ ||xx − y ||.

Definition - Averaged non-expansiveness

Let S be a non-empty subset of Rn and let F : S → Rn . Then F is α-averaged non-expansive if
there exist α ∈]0, 1[ and a non-expansive operator R such that
F = (1 − α)IId + αR.

Non-expansive operators 8/22

Fixed-points of non-expansive operator

Definition - Non-expansive operator

Let S be a non-empty convex subset of Rn and F : S → Rn be a non-expansive operator, the
set of fixed points of F, denoted by fix(F), is defined by
fix(F) = x ∈ S : x = F(xx ) .
def


Non-expansive operators 9/22

Fixed-points of non-expansive operator

Definition - Non-expansive operator

Let S be a non-empty convex subset of Rn and F : S → Rn be a non-expansive operator, the
set of fixed points of F, denoted by fix(F), is defined by
fix(F) = x ∈ S : x = F(xx ) .
def


Proposition - Convexity
Let S be a non-empty closed convex subset of Rn and let F : S → Rn be non-expansive, then
the set of fixed points fix(F) is closed and convex.

Non-expansive operators 9/22

Fixed-points of non-expansive operator

Definition - Non-expansive operator

Let S be a non-empty convex subset of Rn and F : S → Rn be a non-expansive operator, the
set of fixed points of F, denoted by fix(F), is defined by
fix(F) = x ∈ S : x = F(xx ) .
def


Proposition - Convexity
Let S be a non-empty closed convex subset of Rn and let F : S → Rn be non-expansive, then
the set of fixed points fix(F) is closed and convex.

Theorem - Browder-Göhde-Kirk
Let S be a non-empty bounded closed convex subset of Rn and F : S → S be a non-expansive
operator. Then
fix(F) 6= ∅.

Non-expansive operators 9/22

Fejér monotonicity
Fejér monotonicity, fixed-point iteration
Sequence and limits
A number x? ∈ R is called the limit of the sequence {x(k) }k∈N if for any positive  > 0 there
exists a number k > 0 such that for all k ≥ k, there holds
|x(k) − x? | < .
That is, x(k) ∈ [x? − , x? + ] for all k ≥ k. In this case, we write
x? = lim x(k)
k→+∞
or
x(k) → x? .

A sequence that has a limit is called a convergent sequence.

Extension to sequences in Rn .

Fejér monotonicity 10/22

Sequence and limits

Limit of convergent sequence A convergent sequence has only one limit.

Fejér monotonicity 10/22

Sequence and limits

Limit of convergent sequence A convergent sequence has only one limit.

Boundedness and convergence Every convergent sequence is bounded.

Fejér monotonicity 10/22

Sequence and limits

Limit of convergent sequence A convergent sequence has only one limit.

Boundedness and convergence Every convergent sequence is bounded.

Monotonicity and convergence Every monotone bounded sequence in R is convergent.

Fejér monotonicity 10/22

Sequence and limits

Limit of convergent sequence A convergent sequence has only one limit.

Boundedness and convergence Every convergent sequence is bounded.

Monotonicity and convergence Every monotone bounded sequence in R is convergent.

Subsequence and convergence Any subsequence of a convergent sequence is convergent.

Fejér monotonicity 10/22

Sequence and limits

Limit of convergent sequence A convergent sequence has only one limit.

Boundedness and convergence Every convergent sequence is bounded.

Monotonicity and convergence Every monotone bounded sequence in R is convergent.

Subsequence and convergence Any subsequence of a convergent sequence is convergent.

Bolzano-Weierstrass Any bounded sequence has a convergent subsequence.

Fejér monotonicity 10/22

Fejér monotonicity

Definition - Fejér monotonicity

Let S be a non-empty subset of Rn and let {xx (k) }k∈N be a sequence in Rn . Then {xx (k) }k∈N is
Fejér monotone with respect to S if
(∀xx ∈ S)(k ∈ N) ||xx (k+1) − x || ≤ ||xx (k) − x ||.

Fejér monotonicity 11/22

Fejér monotonicity

Definition - Fejér monotonicity

Let S be a non-empty subset of Rn and let {xx (k) }k∈N be a sequence in Rn . Then {xx (k) }k∈N is
Fejér monotone with respect to S if
(∀xx ∈ S)(k ∈ N) ||xx (k+1) − x || ≤ ||xx (k) − x ||.

Theorem - Fejér monotonicity and convergence

Let S be a nonempty subset of Rn and let {xx (k) }k∈N be a sequence in Rn . Suppose that
{xx (k) }k∈N is Fejér monotone with respect to S, then
{xx (k) }k∈N is bounded. For every x ∈ S, {||xx (k) − x ||}k∈N converges.
If every sequential cluster point of {xx (k) }k∈N belongs to S, then
{xx (k) }k∈N converges to a point in S.

Fejér monotonicity 11/22

Fixed-point iteration

Definition - Fixed-point iteration

Let S be a nonempty closed convex subset of Rn , let operator F : S → S be non-expansive such
that fix(F) 6= ∅. Let x (0) ∈ S, and set
(∀k ∈ N) x (k+1) = F(xx (k) ).
Suppose that x (k) − F(xx (k) ) → 0, then
{xx (k) }k∈N converges to a point in fix(F).

Only non-expansiveness does not guarantee convergence.

Fejér monotonicity 12/22

Fixed-point iteration

Theorem - Groetsch
Let S be a nonempty closed convex subset of Rn , let operator FP : S → S be non-expansive such
that fix(F) 6= ∅. Let {λk }k∈N be a sequence in [0, 1] such that k λk (1 − λk ) = +∞, and let
x (0) ∈ S. Set
(∀k ∈ N) x (k+1) = x (k) + λk F(xx (k) ) − x (k) .

Then the following hold

{xx (k) }k∈N is Fejér monotone with respect to fix(F).
{F(xx (k) ) − x (k) }k∈N converges to 0 .
{xx (k) }k∈N converges to a point in fix(F).

When F is α-averaged non-expansive, then for {λk }k∈N , the condition changes to
λk ∈ [0, 1/α] and
X 1 
λk − λk = +∞.
α
k

Fejér monotonicity 12/22

Convex functions
Convex functions
Functions
Let S ⊂ Rn , a function F is a mapping from S to [−∞, +∞], i.e.
F : S → [−∞, +∞].
The domain of F is n o
dom(F) = x ∈ S : F(xx ) < +∞ .
def

The graph of F is n o
gra(F) = (xx , v) ∈ S × R : F(xx ) = v .
def

The epi graph of F is

n o
epi(F) = (xx , v) ∈ S × R : F(xx ) ≤ v .
def

The sub-level set of F is

n o
lev≤v (F) = x ∈ S : F(xx ) ≤ v .
def

Convex functions 13/22

Functions

+∞
epi(F )

gra(F )

x1 lev≤v (F ) x2 dom(F )

Convex functions 13/22

Closed function

Definition - Extended real line function

An extended real-valued function is a function defined over the entire underlying space that can
take any real value, as well as the infinite values −∞ and +∞.

Convex functions 14/22

Closed function

Definition - Extended real line function

An extended real-valued function is a function defined over the entire underlying space that can
take any real value, as well as the infinite values −∞ and +∞.

Example - Indicator function

Let S ⊂ Rn be a set, the indicator function of S is an extended real-valued function given by
(
0 : x ∈ S,
ιS (xx ) =
+∞ : x ∈/ S.

Convex functions 14/22

Closed function

Definition - Extended real line function

An extended real-valued function is a function defined over the entire underlying space that can
take any real value, as well as the infinite values −∞ and +∞.

Example - Indicator function

Let S ⊂ Rn be a set, the indicator function of S is an extended real-valued function given by
(
0 : x ∈ S,
ιS (xx ) =
+∞ : x ∈/ S.

Definition - Closed function

A function F : Rn → [−∞, +∞] is closed if
epi-graph is closed.
sub-level set is closed.

Convex functions 14/22

Convex functions

Definition - Convex function

Let S ⊂ Rn be a non-empty convex set, a function F : S → R is said to be convex if for any
x , y ∈ S and any λ ∈ (0, 1), there holds


F λxx + (1 − λ)yy ≤ λF(xx ) + (1 − λ)F(yy ).

If −F is convex, then F is said to be concave.

Convex functions 15/22

Convex functions

Definition - Convex function

Let S ⊂ Rn be a non-empty convex set, a function F : S → R is said to be convex if for any
x , y ∈ S and any λ ∈ (0, 1), there holds


F λxx + (1 − λ)yy ≤ λF(xx ) + (1 − λ)F(yy ).

If −F is convex, then F is said to be concave.

Example - Examples on R

Absolute value function F(x) = |x| is closed and convex.

The function F(x) = − log(x) is closed and convex.

Convex functions 15/22

Convex functions

Definition - Convex function

Let S ⊂ Rn be a non-empty convex set, a function F : S → R is said to be convex if for any
x , y ∈ S and any λ ∈ (0, 1), there holds


F λxx + (1 − λ)yy ≤ λF(xx ) + (1 − λ)F(yy ).

If −F is convex, then F is said to be concave.

Definition - Strong convexity

Function F : Rn → R is strongly convex if dom(F) is convex, there exists α > 0 such that
α
F(xx ) − ||xx ||2
2
is convex.

Convex functions 15/22

Convex functions

Let F : S → R be a convex function and β > 0, then βF is convex.

Convex functions 15/22

Convex functions

Let F : S → R be a convex function and β > 0, then βF is convex.

Let F1 , F2 : S → R be convex functions, then F1 + F2 is convex.

Sum of finitely many convex functions ki=1 Fi ...

P

Convex functions 15/22

Convex functions

Let F : S → R be a convex function and β > 0, then βF is convex.

Let F1 , F2 : S → R be convex functions, then F1 + F2 is convex.

Sum of finitely many convex functions ki=1 Fi ...

P

be a convex function and (αi )ri=1 ∈]0, 1[ such that

P
LetPF : S → RP i αi = 1, then
F( i αix i ) ≤ i αi F(xx i ).

Convex functions 15/22

Convex functions

Let F : S → R be a convex function and β > 0, then βF is convex.

Let F1 , F2 : S → R be convex functions, then F1 + F2 is convex.

Sum of finitely many convex functions ki=1 Fi ...

P

be a convex function and (αi )ri=1 ∈]0, 1[ such that

P
LetPF : S → RP i αi = 1, then
F( i αix i ) ≤ i αi F(xx i ).

Let S ⊂ R be a non-empty convex set and F : S → R a convex function, then F is continuous

along the interior of S.

Convex functions 15/22

Convex functions

Let F : S → R be a convex function and β > 0, then βF is convex.

Let F1 , F2 : S → R be convex functions, then F1 + F2 is convex.

Sum of finitely many convex functions ki=1 Fi ...

P

be a convex function and (αi )ri=1 ∈]0, 1[ such that

P
LetPF : S → RP i αi = 1, then
F( i αix i ) ≤ i αi F(xx i ).

Let S ⊂ R be a non-empty convex set and F : S → R a convex function, then F is continuous

along the interior of S.

Let F : S → R be a convex function and any α ∈ R, then the sub-level set is convex.

Convex functions 15/22

Convex functions

Let F : S → R be a convex function and β > 0, then βF is convex.

Let F1 , F2 : S → R be convex functions, then F1 + F2 is convex.

Sum of finitely many convex functions ki=1 Fi ...

P

be a convex function and (αi )ri=1 ∈]0, 1[ such that

P
LetPF : S → RP i αi = 1, then
F( i αix i ) ≤ i αi F(xx i ).

Let S ⊂ R be a non-empty convex set and F : S → R a convex function, then F is continuous

along the interior of S.

Let F : S → R be a convex function and any α ∈ R, then the sub-level set is convex.

Definition - Γ0 (Rn )
The set of all proper, closed and convex functions on Rn is denoted as Γ0 (Rn ).
Convex functions 15/22
Differentiability
Gradient, sub-differential
Gradient

Definition - Directional derivative

Let S be a nonempty subset of Rn , F : Rn → R, and x ∈ dom(F). The directional derivative of F
at x in the direction y is
F(xx + αyy ) − F(xx )
∇y F(xx ) = lim ,
α↓0 α
provided that the limits exists.

Differentiability 16/22
Gradient

Definition - Directional derivative

Let S be a nonempty subset of Rn , F : Rn → R, and x ∈ dom(F). The directional derivative of F
at x in the direction y is
F(xx + αyy ) − F(xx )
∇y F(xx ) = lim ,
α↓0 α
provided that the limits exists.

Definition - Gradient
Let S be a subset of Rn , F : S → R, and suppose that F is differentiable at x ∈ S. Then, there
exists a unique vector ∇F(xx ) ∈ Rn such that such
(∀yy ∈ Rn with ||yy || = 1) ∇y F(xx ) = hyy | ∇F(xx )i.

Differentiability 16/22
Gradient
The (Gâteaux) gradient of F at x ∈ S ⊂ dom(F) is an n-dimensional vector
 ∂F(xx) 
∂x1
 ∂F(xx) 
 
 ∂x 
∇F(xx ) =  . 2  ∈ Rn ,
 . 
 . 
∂F(xx )
∂xn
where the partial derivative is defined by
∂F(xx ) F(xx + αeei ) − F(xx )
= lim .
∂xi α↓0 α

Differentiability 16/22
Gradient
The (Gâteaux) gradient of F at x ∈ S ⊂ dom(F) is an n-dimensional vector
 ∂F(xx) 
∂x1
 ∂F(xx) 
 
 ∂x 
∇F(xx ) =  . 2  ∈ Rn ,
 . 
 . 
∂F(xx )
∂xn
where the partial derivative is defined by
∂F(xx ) F(xx + αeei ) − F(xx )
= lim .
∂xi α↓0 α

Proposition - Characterization of convexity

Let S ⊂ Rn be an open set and F : S → R be convex and smooth differentiable, then
F(yy ) ≥ F(xx ) + h∇F(xx ) | y − x i.
hyy − x | ∇F(yy ) − ∇F(xx )i ≥ 0.

Differentiability 16/22
Subdifferentiability

Definition - Subdifferential
Let R : Rn →] − ∞, +∞] be proper convex. The subdifferential of R at is the set-valued
operator
n o
∂R : Rn ⇒ Rn : x → v ∈ Rn : (∀yy ∈ Rn ) hyy − x | v i + R(xx ) ≤ R(yy ) .
Let x ∈ Rn , then R is subdifferentiable at x if ∂R(xx ) 6= ∅.
The elements of ∂R(xx ) are the subgradients of R at x .

Differentiability 17/22
Subdifferentiability

Definition - Subdifferential
Let R : Rn →] − ∞, +∞] be proper convex. The subdifferential of R at is the set-valued
operator
n o
∂R : Rn ⇒ Rn : x → v ∈ Rn : (∀yy ∈ Rn ) hyy − x | v i + R(xx ) ≤ R(yy ) .
Let x ∈ Rn , then R is subdifferentiable at x if ∂R(xx ) 6= ∅.
The elements of ∂R(xx ) are the subgradients of R at x .

Example - Indicator function

Let S be a non-empty convex subset of Rn . Then
v ∈ Rn : suphvv | S − x i ≤ 0 : x ∈ S,
(
∂ιS (xx ) = NS (xx ) =
∅ : o.w.

Differentiability 17/22
Subdifferentiability

Definition - Subdifferential
Let R : Rn →] − ∞, +∞] be proper convex. The subdifferential of R at is the set-valued
operator
n o
∂R : Rn ⇒ Rn : x → v ∈ Rn : (∀yy ∈ Rn ) hyy − x | v i + R(xx ) ≤ R(yy ) .
Let x ∈ Rn , then R is subdifferentiable at x if ∂R(xx ) 6= ∅.
The elements of ∂R(xx ) are the subgradients of R at x .

Proposition - Convexity of subdifferential

Let R : Rn →] − ∞, +∞] be proper convex and x ∈ dom(R). Then
dom(∂R) ⊂ dom(R).
∂R(xx ) is closed and convex.

Differentiability 17/22
Convex minimization problem
Minimizers, and characterizations
Global and local minimum

Definition - Infimum and minimum

Let Φ : S → [−∞, +∞] and let C be a subset of S.
The infimum of Φ over C is inf Φ(C); it is also denoted by
inf Φ(xx ).
x ∈C
Φ achieves its infimum over C if there exists y ∈ C such that
Φ(yy ) = inf Φ(C).
In this case, we write
Φ(yy ) = min Φ(C) or Φ(yy ) = min Φ(xx )
x ∈C
and call min Φ(C) the minimum of Φ over C.

Convex minimization problem 18/22

Global and local minimum

Definition - Global and local minimum

Let Φ : Rn →]−∞, +∞] be proper, and let x ∈ Rn
x is a (global) minimizer of Φ if
Φ(xx ) = inf Φ(Rn )
and Φ(xx ) = min Φ(Rn ) ∈ R.
The set of minimizers of Φ is denoted by Argmin(Φ).
If Argmin(Φ) is a singleton, its unique element is denoted by argminn Φ(xx ).
x ∈R
Let S be a subset of R such that S ∩ dom(Φ) 6= ∅
n

A minimizer of Φ over S is a minimizer of Φ + ιS .

If ∃ρ > 0 such that x is a minimizer of Φ over B(xx ; ρ), then x is a local minimizer of Φ.

Convex minimization problem 18/22

Global and local minimum

Definition - Global and local minimum

Let Φ : Rn →]−∞, +∞] be proper, and let x ∈ Rn
x is a (global) minimizer of Φ if
Φ(xx ) = inf Φ(Rn )
and Φ(xx ) = min Φ(Rn ) ∈ R.
The set of minimizers of Φ is denoted by Argmin(Φ).
If Argmin(Φ) is a singleton, its unique element is denoted by argminn Φ(xx ).
x ∈R
Let S be a subset of R such that S ∩ dom(Φ) 6= ∅
n

A minimizer of Φ over S is a minimizer of Φ + ιS .

If ∃ρ > 0 such that x is a minimizer of Φ over B(xx ; ρ), then x is a local minimizer of Φ.

Theorem - Convexity and local minimizer

Let Φ : Rn →]−∞, +∞] be proper convex. Then every local minimizer of Φ is a minimizer.

Convex minimization problem 18/22

Fermat’s rule

Theorem - Fermat’s rule

Let Φ : Rn →] − ∞, +∞] be proper. Then
n o
Argmin(Φ) = zer(∂Φ) = x ∈ Rn : 0 ∈ ∂Φ(xx ) .
def

Convex minimization problem 19/22

Characterization of minimizers

Problem - A non-smooth problem

Let F ∈ Γ0 (Rn ), K : Rn → Rm be non-zero bounded linear and R ∈ Γ0 (Rm )
n o
min Φ(xx ) = F(xx ) + R(KKx ) ,
def

Convex minimization problem 20/22

Characterization of minimizers

Problem - A non-smooth problem

Let F ∈ Γ0 (Rn ), K : Rn → Rm be non-zero bounded linear and R ∈ Γ0 (Rm )
n o
min Φ(xx ) = F(xx ) + R(KKx ) ,
def

In general,
6 ∂F + K ∗ ◦ ∂R ◦ K .


∂Φ = ∂ F + R ◦ K =

Convex minimization problem 20/22

Characterization of minimizers

Problem - A non-smooth problem

Let F ∈ Γ0 (Rn ), K : Rn → Rm be non-zero bounded linear and R ∈ Γ0 (Rm )
n o
min Φ(xx ) = F(xx ) + R(KKx ) ,
def

In general,
6 ∂F + K ∗ ◦ ∂R ◦ K .


∂Φ = ∂ F + R ◦ K =

Suppose dom(R) ∩ K dom(F) 6= ∅,

∂F + K ∗ ◦ ∂R ◦ K ⊂ ∂ F + R ◦ K .

Convex minimization problem 20/22

Characterization of minimizers

Proposition - Characterization of minimizers

Let F ∈ Γ0 (Rn ), K : Rn → Rm be non-zero bounded and R ∈ Γ0 (Rm ). Then the following holds
zer ∂F + K ∗ ◦ ∂R ◦ K ⊂ Argmin(F + R ◦ K ).

Suppose Argmin(F + R ◦ K ) 6= ∅ and

ri dom(R) ∩ ri K dom(F) 6= ∅.

Then
Argmin(F + R ◦ K ) = zer ∂F + K ∗ ◦ ∂R ◦ K =


6 ∅.

Let x ? ∈ Argmin(F + R ◦ K ), the corresponding optimality condition reads

0 ∈ ∂F(xx ? ) + K ∗ ∂R(KKx ? ).

Convex minimization problem 20/22

Monotone operators

Definition - Set-valued operator

An operator A : Rn ⇒ Rn is set-valued if for every x ∈ Rn , A(xx ) is a subset of Rn . Its graph is
defined by n o
gra(A) = (xx , u ) ∈ Rn × Rn : u ∈ A(xx ) .

Convex minimization problem 21/22

Monotone operators

Definition - Set-valued operator

An operator A : Rn ⇒ Rn is set-valued if for every x ∈ Rn , A(xx ) is a subset of Rn . Its graph is
defined by n o
gra(A) = (xx , u ) ∈ Rn × Rn : u ∈ A(xx ) .

Definition - Monotone operator

Let A : Rn ⇒ Rn . Then A is monotone if
∀(xx , u ) ∈ gra(A) (yy , v ) ∈ gra(A)



hxx − y | u − v i ≥ 0.
It is moreover maximally monotone if gra(A) cannot be contained properly by the graph of
another monotone operator B.

Convex minimization problem 21/22

Monotone operators

Definition - Monotone operator

Let A : Rn ⇒ Rn . Then A is monotone if
∀(xx , u ) ∈ gra(A) (yy , v ) ∈ gra(A)



hxx − y | u − v i ≥ 0.
It is moreover maximally monotone if gra(A) cannot be contained properly by the graph of
another monotone operator B.

Theorem - Moreau
Let R ∈ Γ0 (Rn ). Then ∂R is maximally monotone.

Convex minimization problem 21/22

Monotone operators

Definition - Monotone operator

Let A : Rn ⇒ Rn . Then A is monotone if
∀(xx , u ) ∈ gra(A) (yy , v ) ∈ gra(A)



hxx − y | u − v i ≥ 0.
It is moreover maximally monotone if gra(A) cannot be contained properly by the graph of
another monotone operator B.

Theorem - Moreau
Let R ∈ Γ0 (Rn ). Then ∂R is maximally monotone.

Proposition -
Let A : Rn ⇒ Rn be maximally monotone and x ∈ Rn . Then A(xx ) is closed and convex.

Convex minimization problem 21/22

References

Amir Beck: First-order methods in optimization, Vol. 25. SIAM, 2017.

Heinz H. Bauschke and Patrick L. Combettes: Convex analysis and monotone operator theory
in Hilbert spaces, Vol. 408. New York: Springer, 2011.

References 22/22