EE61012: Convex Optimization for Control and Signal

Processing
Instructor: Prof. Ashish R. Hota

Class Hours: G Slot + S3(2) Slot. Wednesday: 11am - 11:55pm, Thursday:

12pm - 12:55pm, Thursday: 5pm - 5:55pm, Friday: 8am-8:55am
Venue: NR 413
Grading Scheme: 50 % Endsem, 30 % Midsem, 20 % Tutorial and Class
Tests
Preferred Mode of Contact: Send email to [email protected] with
subject containing [EE61012]. Do not forget to write your name and roll no.
Any email with a blank subject and without name and roll no. will be ignored.

1
 Content

Theory:
Formal definition of an optimization problem
Basic topology of sets and existence of optimal solutions
Gradient, Hessian, and optimality conditions for unconstrained problems
Convex sets and properties
Convex functions and properties
Convex optimization problems and their classifications
Separating Hyperplane Theorems, Theorems of the Alternative, LP Duality
Lagrangian duality and KKT optimality conditions
Algorithms:
First order gradient based algorithms under smoothness, strong convexity
Accelerated, stochastic and distributed gradient descent
Applications:
Regression, support vector machines, ML estimation, hypothesis testing
Stability analysis and controller synthesis for linear dynamical systems
Robust optimization

2
 References

Primary References:
Optimization Models by G.C. Calafiore and L. El Ghaoui, Cambridge Univer-
sity Press, 2014. Link: https://people.eecs.berkeley.edu/~elghaoui/
optmodbook.html
Convex Optimization by Stephen Boyd and L. Vandenberghe, Cambridge
University Press. Available online at: https://web.stanford.edu/~boyd/
cvxbook/
Algorithms for Convex Optimization by Nisheeth K. Vishnoi, Cambridge
University Press. Available online at: https://convex-optimization.
github.io
Advanced References on Theory:
Lectures on Modern Convex Optimization, Aharon Ben-Tal and Arkadi Ne-
mirovski, SIAM. Available online at: https://epubs.siam.org/doi/book/
10.1137/1.9780898718829
Convex Analysis and Optimization, Bertsekas, Athena Scientific. More in-
formation at: http://www.athenasc.com/convexity.html
Convex Analysis and Minimization Algorithms, Jean-Baptiste Hiriart-Urruty,
Claude Lemarechal, Springer. Available online at: https://link.springer.
com/book/10.1007/978-3-662-02796-7
Advanced References on Algorithms:
Optimization for Modern Data Analysis, Benjamin Recht and Stephen
J. Wright, Available online at: https://people.eecs.berkeley.edu/
~brecht/opt4ml_book/
Numerical Optimization by Jorge Nocedal, Stephen J. Wright, Springer.
Available online at: https://link.springer.com/book/10.1007/
978-0-387-40065-5

3
 Introductory Lectures on Convex Optimization A Basic Course, by Yurii
Nesterov. Available online at: https://link.springer.com/book/10.
1007/978-1-4419-8853-9
First-order Methods in Optimization, by Amir Beck, SIAM. For more infor-
mation: https://epubs.siam.org/doi/10.1137/1.9781611974997.
Advanced References on Applications in Control:
Linear Matrix Inequalities in System and Control Theory, by Stephen Boyd,
Laurent El Ghaoui, E. Feron, and V. Balakrishnan, Society for Industrial and
Applied Mathematics (SIAM), 1994. Available online at: https://web.
stanford.edu/~boyd/lmibook/
A Course in Robust Control Theory: A Convex Approach, Springer.
Available online at: https://link.springer.com/book/10.1007/
978-1-4757-3290-0
Predictive Control for Linear and Hybrid Systems, Cambridge Univer-
sity Press. More information at: http://www.mpc.berkeley.edu/
mpc-course-material

Advanced References on Applications in Signal Processing and Machine Learning:

Convex Optimization in Signal Processing and Communications, Cambridge
University Press. More information at: https://www.cambridge.org/in/
academic/subjects/engineering/communications-and-signal-processing/
convex-optimization-signal-processing-and-communications?format=
HB&isbn=9780521762229
Optimization for Machine Learning, by Suvrit Sra, Stephen J. Wright, Se-
bastian Nowozin, MIT Press. More information at: https://mitpress.
mit.edu/9780262537766/optimization-for-machine-learning/
Recent Special Issue of Proceedings of the IEEE: https://ieeexplore.
ieee.org/xpl/tocresult.jsp?isnumber=9241485&punumber=5

4
 Computing Resources

MATLAB Toolbox
YALMIP: https://yalmip.github.io/
CVX: http://cvxr.com/cvx/

Python Toolbox
CVXOPT: https://cvxopt.org/index.html
CVXPY: https://www.cvxpy.org/
PYOMO: http://www.pyomo.org/

Solvers
MOSEK: https://www.mosek.com/
Gurobi: https://www.gurobi.com/
IPOPT: https://github.com/coin-or/Ipopt
COIN-OR: https://github.com/coin-or/
For optimal control, Casadi: https://web.casadi.org/

5
 Preliminaries

See https://www.stat.cmu.edu/~ryantibs/convexopt/prerequisite_topics.
pdf for refresher.

Please also see the Appendices of Boyd’s Book and Chapter 2 of ACO Book.

6
 Optimization in Abstract Form

An optimization problem can be stated as

min f (x), (1)

x2X

where
x decision variable, often a vector in Rn
X set of feasible solutions, often a subset of Rn
– often specified in terms of equality and inequality constraints
X := x 2 Rn |gi (x)  0, hj (x) = 0, i 2 {1, 2, . . . , m}, j 2 {1, 2, . . . , p} .
f : Rn ! R cost function

Goal:
Find x⇤ 2 X that minimizes the cost function, i.e., f (x⇤ )  f (x) for every
x 2 X.
Optimal value: f ⇤ := inf x2X f (x)
Optimal solution: x⇤ 2 X if f (x⇤ ) = f ⇤ .

What is inf x2X f (x)?

7
 Infimum vs. Minimum

f ⇤ := inf x2X f (x) if f ⇤ is the greatest lower bound on the value of the
function f (x) over x 2 X.

For any ✏ > 0, there exists some x̄ 2 X such that f ⇤ < f (x̄) < f ⇤ + ✏.

There are two possibilities:

There exists x⇤ 2 X for which f (x⇤ ) = f ⇤ . Then, we say that x⇤ is the
optimal solution and f ⇤ := minx2X f (x) is the optimal value.
f (x) 6= f ⇤ for any x 2 X. We then say that the infimum is not attained for
this problem.
If |X| is finite, then infimum is always attained.
The set of optimal solutions is denoted by argmin, and we say

x⇤ 2 argminx2X f (x) = {y 2 X|f (y) = f ⇤ }.

Note that [argminx2X f (x)] ✓ X.

8
 Examples

Let f (x) = e x
and X = [0, 1). Find f ⇤ and x⇤ .

What if X = [0, 1]?

What if X = [0, 1)?

Moral of the story: Properties of feasibility set X is critical in existence of optimal

solution.

Now suppose X = [0, 1] and f (x) = x for x > 0 and f (x) = 1 for x = 0.

Moral of the story:

9
 Infeasible optimization problem

The problem is infeasible when X is an empty set.

In this case, f ⇤ := +1.

Example:

10
 Unbounded optimization problem

The problem is unbounded when f ⇤ = 1 over the feasibility set X.

Example:

11
 Basic Topology of Sets

Let B(x0 , r) := {x 2 Rn |||x x0 ||2  r} denote the ball around point x0 2 Rn

with radius r > 0.

Interior of the set X, denoted int(X):

Set X is called an open set if X = int(X).

Set X is called closed if and only if its complement is open.

Intersection of arbitrary number of closed sets is closed.

Examples of Open and Closed Sets’:

12
 Bounded and Compact Set

A set X is bounded if there exists B 2 (0, 1) such that for any x1 , x2 2 X,

||x1 x2 ||2  B.

A set X is compact if it is closed and bounded.

13
 Global and Local Optimum

Definition 1 (Global Optimum). A feasible solution x⇤ 2 X is a global

optimum if f (x⇤ )  f (x) for all x 2 X. In this case, f ⇤ = f (x⇤ ). The set of
global optima is denoted by

argminx2X f (x) := {z 2 X|f (z) = f ⇤ }.

Definition 2 (Local Optimum). A feasible solution x⇤ 2 X is a local opti-

mum if f (x⇤ )  f (x) for all x 2 B(x⇤ , r) for some r > 0.

Existence of Optimal Solution:

Theorem 1: Weierstrass Theorem
If the cost function f is continuous and the feasible region X is compact
(closed and bounded), then (at least one global) optimal solution x⇤ exists.

Example:

When X is not bounded, then the above theorem still holds when an ↵-sublevel
set of f , defined as
S↵ (f ) := {x 2 X|f (x)  ↵},
is non-empty and bounded for some ↵ 2 R.

14
 Notes

Given an optimization problem, first determine

the decision variable x and the space in which it resides
feasibility set X
cost function f : X ! R
Before attempting to solve the problem, check whether
f is continuous
X is non empty, or the problem is unbounded
X is closed, and bounded (or any sub-level set of X is bounded)

How to verify whether some x⇤ is indeed an optimal solution?

15
 Gradient (rf (x))

For a function f : Rn ! R, its gradient is defined as:

Compute gradient of
f (x) = x> a

f (x) = x> Ax

f (x) = ||Ax b||22

16
 Hessian (H(x))

For a function f : Rn ! R, its Hessian is defined as:

Compute Hessian of
f (x) = x> a

f (x) = x> Ax

f (x) = ||Ax b||22

17
 Directional Derivative and Descent Direction

Consider a function f : Rn ! R. Let d 2 Rn be the direction of interest.

Definition: The directional derivative of f at point x0 2 Rn along direction

d 2 Rn is defined as

Define (t) := f (x + td).

0
Compute (0) :

If the directional derivative is negative along direction d, then d is called a descent

direction of the function at point x0 .

18
 Necessary Condition of Optimality for Unconstrained
Problems

Theorem 2
If x⇤ is a local optimum for the problem minx2Rn f (x), then rf (x⇤ ) = 0.

Proof by contradiction:

19
 Sufficient Condition of Optimality for Unconstrained
Problems

Let f be twice continuously di↵erentiable over Rn .

Theorem 3
If for x⇤ 2 Rn , we have rf (x⇤ ) = 0 and the Hessian of the cost function
f at x⇤ is a positive definite matrix, then x⇤ is a local optimum for the
problem minx2Rn f (x).

20
 Least Sqaures Problem

Consider the following optimization problem:

minn ||Ax b||22 .

x2R

21
 Convex Sets

Definition 1. Given a collection of points x1 , x2 , . . . , xk , the combination

P1 xk 1 + 2 x2 + . . . + k xk is called Convex combination if i 0 and
i=1 i = 1.
A set X is a convex set if all convex combinations of its elements are in the
set.

Equivalently, X is a convex set if

for every x, y 2 X, x + (1 )y 2 X for any 2 [0, 1].

it contains all convex combinations of any two of its elements.

Are the following sets convex:

X1 = {x 2 R2 |x1 0, x2 0}.
X2 = {x 2 R2 |x1 x2 0}.

1
 Basic Examples of Convex Sets

Sets Defined by Linear Inequalities:

Hyperplane: H = {x 2 Rn |a> x = b} for some a 2 Rn , b 2 R.

Halfspaces: {x 2 Rn |a> x  b} for some a 2 Rn , b 2 R.

2
 Sets Defined by Norms

Consider the Ball Bp (c, R) := {x 2 Rn | ||x c||p  R} where

8⇣ ⌘ p1
< P p
||z||p := i2[n] |xi | , 1  p < 1,
:max
i2[n] |xi |, p = 1.

Recall that norm satisfies triangle inequality and positive homogeneity. We define
[n] := {1, 2, . . . , n}.

Proposition 1. Bp (c, R) is a convex set.

3
 Positive Semidefinite Matrices

Proposition 2. Set of symmetric positive semidefinite matrices, denoted by

Sn+ := {X 2 S n |X ⌫ 0n⇥n }, is a convex set.

25
 Operations that preserve convexity of sets

Proposition 3 (Intersection). If X1 , X2 , . . . , Xm are convex sets, then \i2[m] Xi

is a convex set.

Example: Polyhedron {x 2 Rn |Ax  b} for some A 2 Rm⇥n , b 2 Rm which is

an intersection of half-spaces.

26
 Operations that preserve convexity of sets

Proposition 4 (Affine Image). If X is a convex set, f (x) = Ax + b with

A 2 Rm⇥n , b 2 Rm , then the set f (X) := {y|y = Ax + b for some x 2 X}
is a convex set.

27
 Operations that preserve convexity of sets

Proposition 5 (Product). If X1 , X2 , . . . , Xm are convex sets, then

X := X1 ⇥ X2 ⇥ . . . ⇥ Xm := {(x1 , x2 , . . . , xm ) | xi 2 Xi , i 2 [m]}

is a convex set.

Proposition
P 6 (Weighted
P Sum). If X1 , X2 , . . . , Xm are convex sets, then
i2[m] ↵i Xi := {y | y = i2[m] ↵i xi , xi 2 Xi } is a convex set for ↵i 2 R.

Example:

28
 Operations that preserve convexity of sets

Proposition 7 (Inverse Affine Image). Let X 2 Rn be a convex set and

A : Rm ! Rn be an affine map with A(y) = Ay + b for matrix A and vector
b of suitable dimension. Then, the set A 1 (X) := {y 2 Rm | Ay + b 2 X} is
a convex set.

Problem: Let X1 and X2 be convex sets. Determine if X1 \ X2 is convex.

29
 Ellipsoid

Proposition 8. Let A be a symmetric positive definite matrix. Then, the set

E := {x 2 Rn |(x c)> A 1 (x c)  1} is convex.

30
 Convex Combination

Given a collection of points x1 , x2 , . . . P

, xk , the combination 1 x1 + 2 x2 +...+
k xk is called Convex if i 0 and ni=1 i = 1.

Equivalent Definition:
Definition 4 (Convex Set). A set is convex if it contains all convex combi-
nations of its points.

Definition 5 (Convex Hull). The convex hull of a set X 2 Rn is the set of

all convex combinations of its elements, i.e.,
8 9
< X X =
n
conv(X) := y 2 R | y = i xi , where i 0, i = 1, xi 2 X8i 2 [k], k 2 N .
: ;
i2[k] i2[k]

Proposition 9 (Convex Hull). The following are true.

conv(X) is a convex set (even when X is not).
If X is convex, then conv(X) = X.
For any set X, conv(X) is the smallest convex set containing X.

Example: Determine the convex hull of X = [0, 1] [ [2, 3].

31
 Combination of points

Given a collection of points x1 , x2 , . . . , xk , the combination 1 x1 + 2 x2 +...+

k xk is called

Pn
Convex if i 0 and i=1 i = 1.

Conic if i 0,
Pn
Affine if i=1 i = 1,

Linear if i 2 R.

A set is convex/ convex cone/ affine subspace/linear subspace if it contains all

convex/conic/affine/linear combinations of its elements.

Definition 6. A set X is a cone if for any x 2 X, ↵ 0, we have ↵x 2 X.

Note: Every cone must include the origin. Union of two cones is a cone.

32
 Projection

Definition 7 (Projection). The projection of a point x0 on a set X, denoted

projX (x0 ) is defined as

projX (x0 ) := argminx2X ||x x0 ||22 .

Theorem 4: Projection Theorem

If X is closed and convex, then projX (x0 ) exists and is unique.

Main idea:
Existence due to Weierstrass Theorem
Uniqueness via contradiction exploiting convexity

33
 Supporting Hyperplane

Consider a hyperplane H = {x 2 Rn | a> x = b} with a 6= 0. We define

H := {x 2 Rn | a> x  b}.

Definition 8 (Supporting Hyperplane). A hyperplane H is a supporting hy-

perplane for a convex set C at a boundary point z 2 C if z 2 C and C ✓ H .

Theorem 5: Supporting Hyperplane Theorem

If C is a convex set and z 2 C is a boundary point, then there exists a

supporting hyperplane for C at z.

34
 Separating Hyperplane

Definition 9 (Separating Hyperplane). Let X1 and X2 be two nonempty sets

in Rn . A hyperplane H = {x 2 Rn | a> x = b} with a 6= 0 is said to separate
X1 and X2 if
X1 ✓ H := {x 2 Rn | a> x  b},
X2 ✓ H + := {x 2 Rn | a> x b}.

Separation is said to be strict if X1 ⇢ {x 2 Rn | a> x  b0 }, X2 ⇢ {x 2 Rn |

a> x b00 } with b0 < b00 .

Equivalently
sup a> x  inf a> x
x2X1 x2X2

with the inequality being strict for strict separation.

35
 Separating Hyperplane Theorem

Theorem 6: Separating Hyperplane Theorem

Let X1 and X2 be convex sets with X1 \ X2 = . Then, there exists

a separating hyperplane for X1 and X2 . If X1 is closed and bounded,
and X2 is closed, then X1 and X2 can be strictly separated.
Let X be a closed convex set and x0 2 / X. Then, there exists a
hyperplane that strictly separates x0 and X.

We will prove the second statement. Main Idea:

||a||22
1. Let H = {x 2 Rn | a> x = b} with a = x0 projX (x0 ) and b = a> x0 2 .

2. Use properties of projection and convexity of X to verify that H is indeed

the separating hyperplane.

36
 Theorem of the Alternative (Farkas’ Lemma)

Lemma 1 (Farkas’ Lemma). Let A 2 Rm⇥n and b 2 Rm . Then, exactly one

of the following sets must be empty:
1. {x 2 Rn | Ax = b, x 0}
2. {y 2 Rm | A> y  0, b> y > 0}.

Insight: If unable to show a system of linear inequalities does not have a solution,
try to show that its alternative system does.

Main Idea:
1. Easy to show that if (2) is feasible, (1) is infeasible.
2. For the converse, suppose (1) is infeasible. Then, b 2 / cone(a1 , a2 , . . . , an )
where ai is the i-th column of A. Find a hyperplane separating b from
cone(a1 , a2 , . . . , an ) and show that (2) is feasible.

37
Proof

38
 Domain of a Function

We consider extended real-valued functions f : Rn ! R [ {1} =: R̄.

The (e↵ective) domain of f , denoted dom(f ), is the set {x 2 Rn | |f (x)| <

+1}.

Example: f (x) = x1 . What is dom(f )?

Pn
f (x) = i=1 xi log(xi ). What is dom(f )?

When dom(f ) 6= , we say that the function f is proper.

39
 Convex Functions

Definition 10 (Convex Function). A function f : Rn ! R̄ is convex if

1. dom(f ) ✓ Rn is a convex set, and
2. for every x, y 2 dom(f ), 2 [0, 1], we have f ( x + (1 )y)  f (x) +
(1 )f (y).

The Line segment joining (x, f (x)) and (y, f (y)) lies “above” the function.

Examples:
f (x) = x2
f (x) = ex
f (x) = a> x + b for x 2 Rn

40
 Example: Norms

Definition 11 (Norms). A function ⇡ : Rn ! R̄ is a norm if

⇡(x) 0, 8x and ⇡(x) = 0 if and only if x = 0,
⇡(↵x) = |↵|⇡(x) for all ↵ 2 R,
⇡(x + y)  ⇡(x) + ⇡(y).

Examples:
P 1
||x||p := ( ni=1 |xi |p ) p for p 1.
p
||x||Q := x> Qx where Q is a positive definite matrix.
P Pn
||A||F := ( m i=1
2 1/2
j=1 |Ai,j | ) Frobenius norm on Rm⇥n .

Proposition 10. A Norm is a convex function.

41
 Example: Indicator Function

Definition 12. Indicator function IC (x) of a set C is defined as

(
0, x 2 C,
IC (x) :=
1, x2 / C.

Proposition 11. Indicator function IC (x) is convex if the set C is a convex

set.

42
 Example: Support Function

Proposition 12. Support function of a set C is defined as IC⇤ (x) := supy2C x> y.
Support function of a set is always a convex function.

43
 Special Types of Convex Functions

Definition 13. A function f : Rn ! R̄ is

strictly convex if property (2) above holds with strict inequality for
2 (0, 1),
2
µ-strongly convex if f (x) µ ||x||
2 is convex, and
2

concave if f (x) is convex.

44
 Jensen’s Inequality

Pk f : R ! PkR̄, for any collection of

n
Proposition 13. For a convex function
Pk{x1 , x2 , . . . , xk }, we have f ( i=1 i xi ) 
points i=1 i f (xi ) when i 0
and i=1 i = 1.

Proof is straightforward via induction.

45
 Epigraph Characterization

Definition 14. A epigraph of a function f : Rn ! R̄ is defined as the set

epi(f ) := {(x, t) 2 Rn+1 |f (x)  t}.

Example: Norm cone: {(x, t)|||x||  t} is a convex set.

Proposition 14. Function f : Rn ! R̄ is convex in Rn if and only if its

epigraph is a convex set in Rn+1 .

46