1

-norm Methods for

Convex-Cardinality Problems
problems involving cardinality
the
1
-norm heuristic
convex relaxation and convex envelope interpretations
examples
recent results
Prof. S. Boyd, EE364b, Stanford University

1
-norm heuristics for cardinality problems
cardinality problems arise often, but are hard to solve exactly
a simple heuristic, that relies on
1
-norm, seems to work well
used for many years, in many elds
sparse design
LASSO, robust estimation in statistics
support vector machine (SVM) in machine learning
total variation reconstruction in signal processing, geophysics
compressed sensing
new theoretical results guarantee the method works, at least for a few
problems
Prof. S. Boyd, EE364b, Stanford University 1
Cardinality
the cardinality of x R
n
, denoted card(x), is the number of nonzero
components of x
card is separable; for scalar x, card(x) =

0 x = 0
1 x ,= 0
card is quasiconcave on R
n
+
(but not R
n
) since
card(x + y) mincard(x), card(y)
holds for x, y _ 0
but otherwise has no convexity properties
arises in many problems
Prof. S. Boyd, EE364b, Stanford University 2
General convex-cardinality problems
a convex-cardinality problem is one that would be convex, except for
appearance of card in objective or constraints
examples (with c, f convex):
convex minimum cardinality problem:
minimize card(x)
subject to x c
convex problem with cardinality constraint:
minimize f(x)
subject to x c, card(x) k
Prof. S. Boyd, EE364b, Stanford University 3
Solving convex-cardinality problems
convex-cardinality problem with x R
n
if we x the sparsity pattern of x (i.e., which entries are zero/nonzero)
we get a convex problem
by solving 2
n
convex problems associated with all possible sparsity
patterns, we can solve convex-cardinality problem
(possibly practical for n 10; not practical for n > 15 or so . . . )
general convex-cardinality problem is (NP-) hard
can solve globally by branch-and-bound
can work for particular problem instances (with some luck)
in worst case reduces to checking all (or many of) 2
n
sparsity patterns
Prof. S. Boyd, EE364b, Stanford University 4
Boolean LP as convex-cardinality problem
Boolean LP:
minimize c
T
x
subject to Ax _ b, x
i
0, 1
includes many famous (hard) problems, e.g., 3-SAT, traveling salesman
can be expressed as
minimize c
T
x
subject to Ax _ b, card(x) +card(1 x) n
since card(x) +card(1 x) n x
i
0, 1
conclusion: general convex-cardinality problem is hard
Prof. S. Boyd, EE364b, Stanford University 5
Sparse design
minimize card(x)
subject to x c
nd sparsest design vector x that satises a set of specications
zero values of x simplify design, or correspond to components that
arent even needed
examples:
FIR lter design (zero coecients reduce required hardware)
antenna array beamforming (zero coecients correspond to unneeded
antenna elements)
truss design (zero coecients correspond to bars that are not needed)
wire sizing (zero coecients correspond to wires that are not needed)
Prof. S. Boyd, EE364b, Stanford University 6
Sparse modeling / regressor selection
t vector b R
m
as a linear combination of k regressors (chosen from n
possible regressors)
minimize |Ax b|
2
subject to card(x) k
gives k-term model
chooses subset of k regressors that (together) best t or explain b
can solve (in principle) by trying all

n
k

choices
variations:
minimize card(x) subject to |Ax b|
2

minimize |Ax b|
2
+ card(x)
Prof. S. Boyd, EE364b, Stanford University 7
Sparse signal reconstruction
estimate signal x, given
noisy measurement y = Ax+v, v A(0,
2
I) (A is known; v is not)
prior information card(x) k
maximum likelihood estimate x
ml
is solution of
minimize |Ax y|
2
subject to card(x) k
Prof. S. Boyd, EE364b, Stanford University 8
Estimation with outliers
we have measurements y
i
= a
T
i
x + v
i
+ w
i
, i = 1, . . . , m
noises v
i
A(0,
2
) are independent
only assumption on w is sparsity: card(w) k
B = i [ w
i
,= 0 is set of bad measurements or outliers
maximum likelihood estimate of x found by solving
minimize

iB
(y
i
a
T
i
x)
2
subject to [B[ k
with variables x and B 1, . . . , m
equivalent to
minimize |y Ax w|
2
2
subject to card(w) k
Prof. S. Boyd, EE364b, Stanford University 9
Minimum number of violations
set of convex inequalities
f
1
(x) 0, . . . , f
m
(x) 0, x c
choose x to minimize the number of violated inequalities:
minimize card(t)
subject to f
i
(x) t
i
, i = 1, . . . , m
x c, t 0
determining whether zero inequalities can be violated is (easy) convex
feasibility problem
Prof. S. Boyd, EE364b, Stanford University 10
Linear classier with fewest errors
given data (x
1
, y
1
), . . . , (x
m
, y
m
) R
n
1, 1
we seek linear (ane) classier y sign(w
T
x + v)
classication error corresponds to y
i
(w
T
x + v) 0
to nd w, v that give fewest classication errors:
minimize card(t)
subject to y
i
(w
T
x
i
+ v) + t
i
1, i = 1, . . . , m
with variables w, v, t (we use homogeneity in w, v here)
Prof. S. Boyd, EE364b, Stanford University 11
Smallest set of mutually infeasible inequalities
given a set of mutually infeasible convex inequalities
f
1
(x) 0, . . . , f
m
(x) 0
nd smallest (cardinality) subset of these that is infeasible
certicate of infeasibility is g() = inf
x
(

m
i=1

i
f
i
(x)) 1, _ 0
to nd smallest cardinality infeasible subset, we solve
minimize card()
subject to g() 1, _ 0
(assuming some constraint qualications)
Prof. S. Boyd, EE364b, Stanford University 12
Portfolio investment with linear and xed costs
we use budget B to purchase (dollar) amount x
i
0 of stock i
trading fee is xed cost plus linear cost: card(x) +
T
x
budget constraint is 1
T
x + card(x) +
T
x B
mean return on investment is
T
x; variance is x
T
x
minimize investment variance (risk) with mean return R
min
:
minimize x
T
x
subject to
T
x R
min
, x _ 0
1
T
x + card(x) +
T
x B
Prof. S. Boyd, EE364b, Stanford University 13
Piecewise constant tting
t corrupted x
cor
by a piecewise constant signal x with k or fewer jumps
problem is convex once location (indices) of jumps are xed
x is piecewise constant with k jumps card(D x) k, where
D =

1 1
1 1
.
.
.
.
.
.
1 1

R
(n1)n
as convex-cardinality problem:
minimize | x x
cor
|
2
subject to card(D x) k
Prof. S. Boyd, EE364b, Stanford University 14
Piecewise linear tting
t x
cor
by a piecewise linear signal x with k or fewer kinks
as convex-cardinality problem:
minimize | x x
cor
|
2
subject to card(
2
x) k
where

2
=

1 2 1
1 2 1
.
.
.
.
.
.
.
.
.
1 2 1

Prof. S. Boyd, EE364b, Stanford University 15

1
-norm heuristic
replace card(z) with |z|
1
, or add regularization term |z|
1
to
objective
> 0 is parameter used to achieve desired sparsity
(when card appears in constraint, or as term in objective)
more sophisticated versions use

i
w
i
[z
i
[ or

i
w
i
(z
i
)
+
+

i
v
i
(z
i
)

,
where w, v are positive weights
Prof. S. Boyd, EE364b, Stanford University 16
Example: Minimum cardinality problem
start with (hard) minimum cardinality problem
minimize card(x)
subject to x c
(c convex)
apply heuristic to get (easy)
1
-norm minimization problem
minimize |x|
1
subject to x c
Prof. S. Boyd, EE364b, Stanford University 17
Example: Cardinality constrained problem
start with (hard) cardinality constrained problem (f, c convex)
minimize f(x)
subject to x c, card(x) k
apply heuristic to get (easy)
1
-constrained problem
minimize f(x)
subject to x c, |x|
1

or
1
-regularized problem
minimize f(x) + |x|
1
subject to x c
, adjusted so that card(x) k
Prof. S. Boyd, EE364b, Stanford University 18
Polishing
use
1
heuristic to nd x with required sparsity
x the sparsity pattern of x
re-solve the (convex) optimization problem with this sparsity pattern to
obtain nal (heuristic) solution
Prof. S. Boyd, EE364b, Stanford University 19
Interpretation as convex relaxation
start with
minimize card(x)
subject to x c, |x|

R
equivalent to mixed Boolean convex problem
minimize 1
T
z
subject to [x
i
[ Rz
i
, i = 1, . . . , n
x c, z
i
0, 1, i = 1, . . . , n
with variables x, z
Prof. S. Boyd, EE364b, Stanford University 20
now relax z
i
0, 1 to z
i
[0, 1] to obtain
minimize 1
T
z
subject to [x
i
[ Rz
i
, i = 1, . . . , n
x c
0 z
i
1, i = 1, . . . , n
which is equivalent to
minimize (1/R)|x|
1
subject to x c
the
1
heuristic
optimal value of this problem is lower bound on original problem
Prof. S. Boyd, EE364b, Stanford University 21
Interpretation via convex envelope
convex envelope f
env
of a function f on set c is the largest convex
function that is an underestimator of f on c
epi(f
env
) = Co(epi(f))
f
env
= (f

(with some technical conditions)

for x scalar, [x[ is the convex envelope of card(x) on [1, 1]
for x R
n
scalar, (1/R)|x|
1
is convex envelope of card(x) on
z [ |z|

R
Prof. S. Boyd, EE364b, Stanford University 22
Weighted and asymmetric
1
heuristics
minimize card(x) over convex set c
suppose we know lower and upper bounds on x
i
over c
x c = l
i
x
i
u
i
(best values for these can be found by solving 2n convex problems)
if u
i
< 0 or l
i
> 0, then card(x
i
) = 1 (i.e., x
i
,= 0) for all x c
assuming l
i
< 0, u
i
> 0, convex relaxation and convex envelope
interpretations suggest using
n

i=1

(x
i
)
+
u
i
+
(x
i
)

l
i

as surrogate (and also lower bound) for card(x)

Prof. S. Boyd, EE364b, Stanford University 23
Regressor selection
minimize |Ax b|
2
subject to card(x) k
heuristic:
minimize |Ax b|
2
+ |x|
1
nd smallest value of that gives card(x) k
x associated sparsity pattern (i.e., subset of selected regressors) and
nd x that minimizes |Ax b|
2
Prof. S. Boyd, EE364b, Stanford University 24
Example (6.4 in BV book)
A R
1020
, x R
20
, b R
10
dashed curve: exact optimal (via enumeration)
solid curve:
1
heuristic with polishing
0 1 2 3 4
0
2
4
6
8
10
Ax b
2
c
a
r
d
(
x
)
Prof. S. Boyd, EE364b, Stanford University 25
Sparse signal reconstruction
convex-cardinality problem:
minimize |Ax y|
2
subject to card(x) k

1
heuristic:
minimize |Ax y|
2
subject to |x|
1

(called LASSO)
another form: minimize |Ax y|
2
+ |x|
1
(called basis pursuit denoising)
Prof. S. Boyd, EE364b, Stanford University 26
Example
signal x R
n
with n = 1000, card(x) = 30
m = 200 (random) noisy measurements: y = Ax + v, v A(0,
2
I),
A
ij
A(0, 1)
left: original; right:
1
reconstruction with = 10
3
100 200 300 400 500 600 700 800 900 1000
1
0.8
0.6
0.4
0.2
0
0.2
0.4
0.6
0.8
1
100 200 300 400 500 600 700 800 900 1000
1
0.8
0.6
0.4
0.2
0
0.2
0.4
0.6
0.8
1
Prof. S. Boyd, EE364b, Stanford University 27

2
reconstruction; minimizes |Ax y|
2
+ |x|
2
, where = 10
3
left: original; right:
2
reconstruction
100 200 300 400 500 600 700 800 900 1000
1
0.8
0.6
0.4
0.2
0
0.2
0.4
0.6
0.8
1
100 200 300 400 500 600 700 800 900 1000
1
0.8
0.6
0.4
0.2
0
0.2
0.4
0.6
0.8
1
Prof. S. Boyd, EE364b, Stanford University 28
Some recent theoretical results
suppose y = Ax, A R
mn
, card(x) k
to reconstruct x, clearly need m k
if m n and A is full rank, we can reconstruct x without cardinality
assumption
when does the
1
heuristic (minimizing |x|
1
subject to Ax = y)
reconstruct x (exactly)?
Prof. S. Boyd, EE364b, Stanford University 29
recent results by Cand`es, Donoho, Romberg, Tao, . . .
(for some choices of A) if m (C log n)k,
1
heuristic reconstructs x
exactly, with overwhelming probability
C is absolute constant; valid As include
A
ij
A(0,
2
)
Ax gives Fourier transform of x at m frequencies, chosen from
uniform distribution
Prof. S. Boyd, EE364b, Stanford University 30