Scribd
Upload
98 views14 pages

CS 726: Nonlinear Optimization 1 Lecture 2: Background Material

This document provides a summary of the background material presented in Lecture 2 of CS 726: Nonlinear Optimization 1 at UW-Madison. The lecture covered topics including continuity, norms, polyhedral sets, convexity of sets and functions. Key concepts like effective domain, epigraph, and the definition of a convex function were defined. Examples of convex functions such as linear functions and norms were also provided.

Uploaded by

Harris
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
98 views14 pages

CS 726: Nonlinear Optimization 1 Lecture 2: Background Material

This document provides a summary of the background material presented in Lecture 2 of CS 726: Nonlinear Optimization 1 at UW-Madison. The lecture covered topics including continuity, norms, polyhedral sets, convexity of sets and functions. Key concepts like effective domain, epigraph, and the definition of a convex function were defined. Examples of convex functions such as linear functions and norms were also provided.

Uploaded by

Harris
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

CS 726: Nonlinear Optimization 1

Lecture 2 : Background Material

Michael C. Ferris

Computer Sciences Department


University of Wisconsin-Madison

January 27 2021

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 1 / 13


Announcements

In addition to the Appendix of the textbook for the course, there are
two handouts on Canvas that could be useful. They are
“MathBackground.pdf” (alternatively [Bertsekas(2015), Appendix A])
and [Bertsekas(2015), Appendix B].

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 2 / 13


Continuity
A function f (x) is continuous at a point x if f (x) = lim f (t).
t!x
Define epif = {(x, )| f (x)}.
Then f is lower semi-continuous if epif is a closed set in Rn+1 . (We
called this a closed function in class but will use both interchangeably.)
Intuitively, this is saying f (x)  lim f (t).
t!x
4

2
f (x)

1
2 1 0 1 2
x

The function f (x) is lower semi-continuous if f (0) = 1 but not if f (0) = 1.


Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 3 / 13
Norms

The norm of x, denoted ||x||, satisfies:


1 ||x|| 0 8x and ||x|| = 0 , x = 0
2 || x|| = | | ||x|| 8 2 R
3 ||x + y ||  ||x|| + ||y || 8x, y .
Some notes:
Norms are continuous functions
Norms are convex functions (see below)
All norms are equivalent in Rn because
9c1 , c2 > 0 : c1 ⇤ ||x||b  ||x||a  c2 ⇤ ||x||b

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 4 / 13


Some common norms

P
n
||x||1 = |xi |
i=1
s
P
n
||x||2 = xi2
i=1

||x||1 = max |xi |


i=1,2,...,n

Note that ||x||1 and ||x||1 can be described using a finite number of linear
constraints while ||x||2 cannot be.
We can define the norm of a matrix using a vector norm by letting:
||A|| = sup ||Ax||
||x|| .
x6=0
Using that, the condition number of A is: K (A) = ||A|| ⇤ ||A 1 ||.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 5 / 13


Polyhedral Sets

A set C = {x|Ax  b} is called polyhedral.


Sets with equality constraints are still polyhedral
We will call a bounded polyhedral set a polytope

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 6 / 13


Convexity of a set
Definition
Set C is convex if 8x, y 2 C , 8↵ 2 [0, 1] : ↵x + (1 ↵)y 2 C

Prove the following statements:


1 C = {x|Ax  b} is convex.
T
2 Suppose that Ci , i 2 I are convex in Rn . Then i2I Ci is convex.
Remark: Unions of convex sets are not convex:

Unions of convex sets are not convex.

3 Suppose that C1 , C2 are convex. Then C1 + C2 is also convex.


Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 7 / 13
More to prove:
1 Suppose that C1 , C2 are convex. Then
C1 ⇥ C2 = {(x, y )|x 2 C1 , y 2 C2 } is also convex.
2 Suppose that A 2 Rm⇥n . Then
C 2 Rn is convex ) AC = {y |y = Ax, x 2 C } is convex in Rm

When A is [1,0], AC is the projection of C.

3 Suppose C is convex. Then the closure of C : Cl(C ) is also convex.


Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 8 / 13
Convex Function
Suppose f : S ⇢ Rn ! R. Extend the domain to Rn :
(
f (x), x 2 S
f¯ : Rn ! R̄ = R [ {+1} such that f¯(x) =
+1, else
In terms of convex functions, it is convenient to just consider extended
real-valued functions. For example,
min f (x) s.t. x 2 C ) min f¯(x) s.t. x 2 Rn
where (
f (x), if x 2 C
f¯(x) =
+1, if x 2
/C
or (
0, if x 2 C
f¯(x) = f (x) + IC (x), where Ic (x) =
+1, if x 2
/C
This is essentially the same as used in [Boyd and Vandenberghe(2004)]
and also in the lecture notes [Burke(2018)].
Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 9 / 13
Definitions
E↵ective domain: dom(f ) = {x|f (x) < +1}
Epigraph: f : Rn ! R̄, epi(f ) = {(x, µ) 2 Rn+1 |f (x)  µ} (4)
Convex function: f is convex if epi(f ) is a convex subset of Rn+1

Epigraph of a function.

Note that dom(f ) = [I 0]epi(f ) so is convex when f is.


Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 10 / 13
Main theorem
Theorem
f : R ! R̄. Then f is convex , 8 2 [0, 1], 8x, y

f ( x + (1 )y )  f (x) + (1 )f (y ).

Proof.
()) Suppose epi(f ) is convex. Let x, y be given and 0 < < 1. If x or
y2/ dom(f ), RHS= +1. So trivial. (Convention: positive number
⇥(+1) = +1.)
Else, suppose x, y 2 dom(f ), then (x, f (x)), (y , f (y )) 2 epi(f )
By convexity of epi(f ),

(x, f (x)) + (1 )(y , f (y ))


= ( x + (1 )y , f (x) + (1 )f (y )) 2 epi(f )
) f ( x + (1 )y )  f (x) + (1 )f (y )

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 11 / 13


Cont.

Proof.
(() Suppose (x, µ), (y , ) 2 epi(f ), 0 < < 1.
RTP: (x, µ) + (1 )(y , ) 2 epi(f ).

f ( x + (1 )y )  f (x) + (1 )f (y )
 µ + (1 )
) ( x + (1 )y , µ + (1 ) ) 2 epi(f )
) (x, µ) + (1 )(y , ) 2 epi(f )

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 12 / 13


Examples

1 C T x is a convex function.
Remark: Linear functions are the only functions that are both convex
and concave.
2 kxk is convex by triangular inequality.
3 e x and x 2 are convex.
4
(
1
x , if x > 0
f (x) = is convex.
+1, else

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 13 / 13


D. P. Bertsekas.
Convex Optimization Algorithms.
Athena Scientific, 2015.
S. Boyd and L. Vandenberghe.
Convex Optimization.
Cambridge University Press, 2004.
ISBN 9780521833783.
URL
http://www.informaworld.com/openurl?genre=article{&}doi=
10.1080/10556781003625177{&}magic=crossref.
J. V. Burke.
Nonlinear Optimization.
in preparation, 2018.
S. J. Wright and B. Recht.
Optimization for Data Analysis.
in proof, 2020.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 13 / 13

You might also like