Foundations and Trends
R

in
Machine Learning
Vol. 3, No. 1 (2010) 1–122

c 2011 S. Boyd, N. Parikh, E. Chu, B. Peleato
and J. Eckstein
DOI: 10.1561/2200000016

Distributed Optimization and Statistical

Learning via the Alternating Direction
Method of Multipliers

Stephen Boyd1 , Neal Parikh2 , Eric Chu3

Borja Peleato4 and Jonathan Eckstein5

1
Electrical Engineering Department, Stanford University, Stanford, CA
94305, USA, [email protected]
2
Computer Science Department, Stanford University, Stanford, CA 94305,
USA, [email protected]
3
Electrical Engineering Department, Stanford University, Stanford, CA
94305, USA, [email protected]
4
Electrical Engineering Department, Stanford University, Stanford, CA
94305, USA, [email protected]
5
Management Science and Information Systems Department and
RUTCOR, Rutgers University, Piscataway, NJ 08854, USA,
[email protected]
 Contents

1 Introduction 3

2 Precursors 7
2.1 Dual Ascent 7
2.2 Dual Decomposition 9
2.3 Augmented Lagrangians and the Method of Multipliers 10

3 Alternating Direction Method of Multipliers 13

3.1 Algorithm 13
3.2 Convergence 15
3.3 Optimality Conditions and Stopping Criterion 18
3.4 Extensions and Variations 20
3.5 Notes and References 23

4 General Patterns 25

4.1 Proximity Operator 25

4.2 Quadratic Objective Terms 26
4.3 Smooth Objective Terms 30
4.4 Decomposition 31

5 Constrained Convex Optimization 33

5.1 Convex Feasibility 34

5.2 Linear and Quadratic Programming 36
6
1 -Norm Problems 38

6.1 Least Absolute Deviations 39

6.2 Basis Pursuit 41
6.3 General
1 Regularized Loss Minimization 42
6.4 Lasso 43
6.5 Sparse Inverse Covariance Selection 45

7 Consensus and Sharing 48

7.1 Global Variable Consensus Optimization 48
7.2 General Form Consensus Optimization 53
7.3 Sharing 56

8 Distributed Model Fitting 61

8.1 Examples 62
8.2 Splitting across Examples 64
8.3 Splitting across Features 66

9 Nonconvex Problems 73
9.1 Nonconvex Constraints 73
9.2 Bi-convex Problems 76

10 Implementation 78
10.1 Abstract Implementation 78
10.2 MPI 80
10.3 Graph Computing Frameworks 81
10.4 MapReduce 82

11 Numerical Examples 87

11.1 Small Dense Lasso 88

11.2 Distributed
1 Regularized Logistic Regression 92
11.3 Group Lasso with Feature Splitting 95
11.4 Distributed Large-Scale Lasso with MPI 97
11.5 Regressor Selection 100
12 Conclusions 103

Acknowledgments 105

A Convergence Proof 106

References 111
Abstract
Many problems of recent interest in statistics and machine learning
can be posed in the framework of convex optimization. Due to the
explosion in size and complexity of modern datasets, it is increasingly
important to be able to solve problems with a very large number of fea-
tures or training examples. As a result, both the decentralized collection
or storage of these datasets as well as accompanying distributed solu-
tion methods are either necessary or at least highly desirable. In this
review, we argue that the alternating direction method of multipliers
is well suited to distributed convex optimization, and in particular to
large-scale problems arising in statistics, machine learning, and related
areas. The method was developed in the 1970s, with roots in the 1950s,
and is equivalent or closely related to many other algorithms, such
as dual decomposition, the method of multipliers, Douglas–Rachford
splitting, Spingarn’s method of partial inverses, Dykstra’s alternating
projections, Bregman iterative algorithms for
1 problems, proximal
methods, and others. After briefly surveying the theory and history of
the algorithm, we discuss applications to a wide variety of statistical
and machine learning problems of recent interest, including the lasso,
sparse logistic regression, basis pursuit, covariance selection, support
vector machines, and many others. We also discuss general distributed
optimization, extensions to the nonconvex setting, and efficient imple-
mentation, including some details on distributed MPI and Hadoop
MapReduce implementations.
 1
Introduction

In all applied fields, it is now commonplace to attack problems through

data analysis, particularly through the use of statistical and machine
learning algorithms on what are often large datasets. In industry, this
trend has been referred to as ‘Big Data’, and it has had a significant
impact in areas as varied as artificial intelligence, internet applications,
computational biology, medicine, finance, marketing, journalism, net-
work analysis, and logistics.
Though these problems arise in diverse application domains, they
share some key characteristics. First, the datasets are often extremely
large, consisting of hundreds of millions or billions of training examples;
second, the data is often very high-dimensional, because it is now possi-
ble to measure and store very detailed information about each example;
and third, because of the large scale of many applications, the data is
often stored or even collected in a distributed manner. As a result, it
has become of central importance to develop algorithms that are both
rich enough to capture the complexity of modern data, and scalable
enough to process huge datasets in a parallelized or fully decentral-
ized fashion. Indeed, some researchers [92] have suggested that even
highly complex and structured problems may succumb most easily to
relatively simple models trained on vast datasets.

3
4 Introduction

Many such problems can be posed in the framework of convex opti-

mization. Given the significant work on decomposition methods and
decentralized algorithms in the optimization community, it is natural
to look to parallel optimization algorithms as a mechanism for solving
large-scale statistical tasks. This approach also has the benefit that one
algorithm could be flexible enough to solve many problems.
This review discusses the alternating direction method of multipli-
ers (ADMM), a simple but powerful algorithm that is well suited to
distributed convex optimization, and in particular to problems aris-
ing in applied statistics and machine learning. It takes the form of a
decomposition-coordination procedure, in which the solutions to small
local subproblems are coordinated to find a solution to a large global
problem. ADMM can be viewed as an attempt to blend the benefits
of dual decomposition and augmented Lagrangian methods for con-
strained optimization, two earlier approaches that we review in §2. It
turns out to be equivalent or closely related to many other algorithms
as well, such as Douglas-Rachford splitting from numerical analysis,
Spingarn’s method of partial inverses, Dykstra’s alternating projec-
tions method, Bregman iterative algorithms for
1 problems in signal
processing, proximal methods, and many others. The fact that it has
been re-invented in different fields over the decades underscores the
intuitive appeal of the approach.
It is worth emphasizing that the algorithm itself is not new, and that
we do not present any new theoretical results. It was first introduced
in the mid-1970s by Gabay, Mercier, Glowinski, and Marrocco, though
similar ideas emerged as early as the mid-1950s. The algorithm was
studied throughout the 1980s, and by the mid-1990s, almost all of the
theoretical results mentioned here had been established. The fact that
ADMM was developed so far in advance of the ready availability of
large-scale distributed computing systems and massive optimization
problems may account for why it is not as widely known today as we
believe it should be.
The main contributions of this review can be summarized as follows:
(1) We provide a simple, cohesive discussion of the extensive
literature in a way that emphasizes and unifies the aspects
of primary importance in applications.
 5

(2) We show, through a number of examples, that the algorithm

is well suited for a wide variety of large-scale distributed mod-
ern problems. We derive methods for decomposing a wide
class of statistical problems by training examples and by fea-
tures, which is not easily accomplished in general.

(3) We place a greater emphasis on practical large-scale imple-

mentation than most previous references. In particular, we
discuss the implementation of the algorithm in cloud com-
puting environments using standard frameworks and provide
easily readable implementations of many of our examples.

Throughout, the focus is on applications rather than theory, and a main

goal is to provide the reader with a kind of ‘toolbox’ that can be applied
in many situations to derive and implement a distributed algorithm of
practical use. Though the focus here is on parallelism, the algorithm
can also be used serially, and it is interesting to note that with no
tuning, ADMM can be competitive with the best known methods for
some problems.
While we have emphasized applications that can be concisely
explained, the algorithm would also be a natural fit for more compli-
cated problems in areas like graphical models. In addition, though our
focus is on statistical learning problems, the algorithm is readily appli-
cable in many other cases, such as in engineering design, multi-period
portfolio optimization, time series analysis, network flow, or scheduling.

Outline
We begin in §2 with a brief review of dual decomposition and the
method of multipliers, two important precursors to ADMM. This sec-
tion is intended mainly for background and can be skimmed. In §3,
we present ADMM, including a basic convergence theorem, some vari-
ations on the basic version that are useful in practice, and a survey of
some of the key literature. A complete convergence proof is given in
appendix A.
In §4, we describe some general patterns that arise in applications
of the algorithm, such as cases when one of the steps in ADMM can
6 Introduction

be carried out particularly efficiently. These general patterns will recur

throughout our examples. In §5, we consider the use of ADMM for some
generic convex optimization problems, such as constrained minimiza-
tion and linear and quadratic programming. In §6, we discuss a wide
variety of problems involving the
1 norm. It turns out that ADMM
yields methods for these problems that are related to many state-of-the-
art algorithms. This section also clarifies why ADMM is particularly
well suited to machine learning problems.
In §7, we present consensus and sharing problems, which provide
general frameworks for distributed optimization. In §8, we consider
distributed methods for generic model fitting problems, including reg-
ularized regression models like the lasso and classification models like
support vector machines.
In §9, we consider the use of ADMM as a heuristic for solving some
nonconvex problems. In §10, we discuss some practical implementation
details, including how to implement the algorithm in frameworks suit-
able for cloud computing applications. Finally, in §11, we present the
details of some numerical experiments.
 2
Precursors

In this section, we briefly review two optimization algorithms that are

precursors to the alternating direction method of multipliers. While
we will not use this material in the sequel, it provides some useful
background and motivation.

2.1 Dual Ascent

Consider the equality-constrained convex optimization problem
minimize f (x)
(2.1)
subject to Ax = b,
with variable x ∈ Rn , where A ∈ Rm×n and f : Rn → R is convex.
The Lagrangian for problem (2.1) is
L(x, y) = f (x) + y T (Ax − b)
and the dual function is
g(y) = inf L(x, y) = −f ∗ (−AT y) − bT y,
x

where y is the dual variable or Lagrange multiplier, and f ∗ is the convex

conjugate of f ; see [20, §3.3] or [140, §12] for background. The dual

7
8 Precursors

problem is

maximize g(y),

with variable y ∈ Rm . Assuming that strong duality holds, the optimal

values of the primal and dual problems are the same. We can recover
a primal optimal point x
from a dual optimal point y
as
x
= argmin L(x, y
),
x

provided there is only one minimizer of L(x, y
). (This is the case

if, e.g., f is strictly convex.) In the sequel, we will use the notation
argminx F (x) to denote any minimizer of F , even when F does not
have a unique minimizer.
In the dual ascent method, we solve the dual problem using gradient
ascent. Assuming that g is differentiable, the gradient ∇g(y) can be
evaluated as follows. We first find x+ = argminx L(x, y); then we have
∇g(y) = Ax+ − b, which is the residual for the equality constraint. The
dual ascent method consists of iterating the updates
xk+1 := argmin L(x, y k ) (2.2)
x
y k+1 := y k + αk (Axk+1 − b), (2.3)
where αk > 0 is a step size, and the superscript is the iteration counter.
The first step (2.2) is an x-minimization step, and the second step (2.3)
is a dual variable update. The dual variable y can be interpreted as
a vector of prices, and the y-update is then called a price update or
price adjustment step. This algorithm is called dual ascent since, with
appropriate choice of αk , the dual function increases in each step, i.e.,
g(y k+1 ) > g(y k ).
The dual ascent method can be used even in some cases when g is
not differentiable. In this case, the residual Axk+1 − b is not the gradi-
ent of g, but the negative of a subgradient of −g. This case requires a
different choice of the αk than when g is differentiable, and convergence
is not monotone; it is often the case that g(y k+1 ) > g(y k ). In this case,
the algorithm is usually called the dual subgradient method [152].
If αk is chosen appropriately and several other assumptions hold,
then xk converges to an optimal point and y k converges to an optimal
 2.2 Dual Decomposition 9

dual point. However, these assumptions do not hold in many applica-

tions, so dual ascent often cannot be used. As an example, if f is a
nonzero affine function of any component of x, then the x-update (2.2)
fails, since L is unbounded below in x for most y.

2.2 Dual Decomposition

The major benefit of the dual ascent method is that it can lead to a
decentralized algorithm in some cases. Suppose, for example, that the
objective f is separable (with respect to a partition or splitting of the
variable into subvectors), meaning that

N
f (x) = fi (xi ),
i=1

where x = (x1 , . . . , xN ) and the variables xi ∈ Rni are subvectors of x.

Partitioning the matrix A conformably as

A = [A1 · · · AN ] ,
N
so Ax = i=1 Ai xi , the Lagrangian can be written as

N

N
 
L(x, y) = Li (xi , y) = fi (xi ) + y T Ai xi − (1/N )y T b ,
i=1 i=1

which is also separable in x. This means that the x-minimization

step (2.2) splits into N separate problems that can be solved in parallel.
Explicitly, the algorithm is

xk+1
i := argmin Li (xi , y k ) (2.4)
xi

y k+1 := y k + αk (Axk+1 − b). (2.5)

The x-minimization step (2.4) is carried out independently, in parallel,

for each i = 1, . . . , N . In this case, we refer to the dual ascent method
as dual decomposition.
In the general case, each iteration of the dual decomposition method
requires a broadcast and a gather operation. In the dual update
step (2.5), the equality constraint residual contributions Ai xk+1 i are
10 Precursors

collected (gathered) in order to compute the residual Axk+1 − b. Once

the (global) dual variable y k+1 is computed, it must be distributed
(broadcast) to the processors that carry out the N individual xi mini-
mization steps (2.4).
Dual decomposition is an old idea in optimization, and traces back
at least to the early 1960s. Related ideas appear in well known work
by Dantzig and Wolfe [44] and Benders [13] on large-scale linear pro-
gramming, as well as in Dantzig’s seminal book [43]. The general idea
of dual decomposition appears to be originally due to Everett [69],
and is explored in many early references [107, 84, 117, 14]. The use
of nondifferentiable optimization, such as the subgradient method, to
solve the dual problem is discussed by Shor [152]. Good references on
dual methods and decomposition include the book by Bertsekas [16,
chapter 6] and the survey by Nedić and Ozdaglar [131] on distributed
optimization, which discusses dual decomposition methods and con-
sensus problems. A number of papers also discuss variants on standard
dual decomposition, such as [129].
More generally, decentralized optimization has been an active topic
of research since the 1980s. For instance, Tsitsiklis and his co-authors
worked on a number of decentralized detection and consensus problems
involving the minimization of a smooth function f known to multi-
ple agents [160, 161, 17]. Some good reference books on parallel opti-
mization include those by Bertsekas and Tsitsiklis [17] and Censor and
Zenios [31]. There has also been some recent work on problems where
each agent has its own convex, potentially nondifferentiable, objective
function [130]. See [54] for a recent discussion of distributed methods
for graph-structured optimization problems.

2.3 Augmented Lagrangians and the Method of Multipliers

Augmented Lagrangian methods were developed in part to bring
robustness to the dual ascent method, and in particular, to yield con-
vergence without assumptions like strict convexity or finiteness of f .
The augmented Lagrangian for (2.1) is

Lρ (x, y) = f (x) + y T (Ax − b) + (ρ/2)Ax − b22 , (2.6)

 2.3 Augmented Lagrangians and the Method of Multipliers 11

where ρ > 0 is called the penalty parameter. (Note that L0 is the

standard Lagrangian for the problem.) The augmented Lagrangian
can be viewed as the (unaugmented) Lagrangian associated with the
problem
minimize f (x) + (ρ/2)Ax − b22
subject to Ax = b.
This problem is clearly equivalent to the original problem (2.1), since
for any feasible x the term added to the objective is zero. The associated
dual function is gρ (y) = inf x Lρ (x, y).
The benefit of including the penalty term is that gρ can be shown to
be differentiable under rather mild conditions on the original problem.
The gradient of the augmented dual function is found the same way as
with the ordinary Lagrangian, i.e., by minimizing over x, and then eval-
uating the resulting equality constraint residual. Applying dual ascent
to the modified problem yields the algorithm
xk+1 := argmin Lρ (x, y k ) (2.7)
x
y k+1 := y k + ρ(Axk+1 − b), (2.8)
which is known as the method of multipliers for solving (2.1). This is
the same as standard dual ascent, except that the x-minimization step
uses the augmented Lagrangian, and the penalty parameter ρ is used
as the step size αk . The method of multipliers converges under far more
general conditions than dual ascent, including cases when f takes on
the value +∞ or is not strictly convex.
It is easy to motivate the choice of the particular step size ρ in
the dual update (2.8). For simplicity, we assume here that f is differ-
entiable, though this is not required for the algorithm to work. The
optimality conditions for (2.1) are primal and dual feasibility, i.e.,
Ax
− b = 0, ∇f (x
) + AT y
= 0,
respectively. By definition, xk+1 minimizes Lρ (x, y k ), so
0 = ∇x Lρ (xk+1 , y k )
 
= ∇x f (xk+1 ) + AT y k + ρ(Axk+1 − b)
= ∇x f (xk+1 ) + AT y k+1 .
12 Precursors

We see that by using ρ as the step size in the dual update, the iterate
(xk+1 , y k+1 ) is dual feasible. As the method of multipliers proceeds, the
primal residual Axk+1 − b converges to zero, yielding optimality.
The greatly improved convergence properties of the method of mul-
tipliers over dual ascent comes at a cost. When f is separable, the aug-
mented Lagrangian Lρ is not separable, so the x-minimization step (2.7)
cannot be carried out separately in parallel for each xi . This means that
the basic method of multipliers cannot be used for decomposition. We
will see how to address this issue next.
Augmented Lagrangians and the method of multipliers for con-
strained optimization were first proposed in the late 1960s by Hestenes
[97, 98] and Powell [138]. Many of the early numerical experiments on
the method of multipliers are due to Miele et al. [124, 125, 126]. Much
of the early work is consolidated in a monograph by Bertsekas [15],
who also discusses similarities to older approaches using Lagrangians
and penalty functions [6, 5, 71], as well as a number of generalizations.
 3
Alternating Direction Method of Multipliers

3.1 Algorithm
ADMM is an algorithm that is intended to blend the decomposability
of dual ascent with the superior convergence properties of the method
of multipliers. The algorithm solves problems in the form
minimize f (x) + g(z)
(3.1)
subject to Ax + Bz = c

with variables x ∈ Rn and z ∈ Rm , where A ∈ Rp×n , B ∈ Rp×m , and

c ∈ Rp . We will assume that f and g are convex; more specific assump-
tions will be discussed in §3.2. The only difference from the general
linear equality-constrained problem (2.1) is that the variable, called x
there, has been split into two parts, called x and z here, with the objec-
tive function separable across this splitting. The optimal value of the
problem (3.1) will be denoted by

p
= inf{f (x) + g(z) | Ax + Bz = c}.

As in the method of multipliers, we form the augmented Lagrangian

Lρ (x, z, y) = f (x) + g(z) + y T (Ax + Bz − c) + (ρ/2)Ax + Bz − c22 .

13
14 Alternating Direction Method of Multipliers

ADMM consists of the iterations

xk+1 := argmin Lρ (x, z k , y k ) (3.2)

x
z k+1 := argmin Lρ (xk+1 , z, y k ) (3.3)
z
y k+1 := y k + ρ(Axk+1 + Bz k+1 − c), (3.4)

where ρ > 0. The algorithm is very similar to dual ascent and the
method of multipliers: it consists of an x-minimization step (3.2), a
z-minimization step (3.3), and a dual variable update (3.4). As in the
method of multipliers, the dual variable update uses a step size equal
to the augmented Lagrangian parameter ρ.
The method of multipliers for (3.1) has the form

(xk+1 , z k+1 ) := argmin Lρ (x, z, y k )

x,z

y k+1
:= y + ρ(Axk+1 + Bz k+1 − c).
k

Here the augmented Lagrangian is minimized jointly with respect to

the two primal variables. In ADMM, on the other hand, x and z are
updated in an alternating or sequential fashion, which accounts for the
term alternating direction. ADMM can be viewed as a version of the
method of multipliers where a single Gauss-Seidel pass [90, §10.1] over
x and z is used instead of the usual joint minimization. Separating the
minimization over x and z into two steps is precisely what allows for
decomposition when f or g are separable.
The algorithm state in ADMM consists of z k and y k . In other words,
(z k+1 , y k+1 ) is a function of (z k , y k ). The variable xk is not part of the
state; it is an intermediate result computed from the previous state
(z k−1 , y k−1 ).
If we switch (re-label) x and z, f and g, and A and B in the prob-
lem (3.1), we obtain a variation on ADMM with the order of the x-
update step (3.2) and z-update step (3.3) reversed. The roles of x and
z are almost symmetric, but not quite, since the dual update is done
after the z-update but before the x-update.
 3.2 Convergence 15

3.1.1 Scaled Form

ADMM can be written in a slightly different form, which is often
more convenient, by combining the linear and quadratic terms in the
augmented Lagrangian and scaling the dual variable. Defining the resid-
ual r = Ax + Bz − c, we have

y T r + (ρ/2)r22 = (ρ/2)r + (1/ρ)y22 − (1/2ρ)y22

= (ρ/2)r + u22 − (ρ/2)u22 ,

where u = (1/ρ)y is the scaled dual variable. Using the scaled dual vari-
able, we can express ADMM as
 
xk+1 := argmin f (x) + (ρ/2)Ax + Bz k − c + uk 22 (3.5)
x
 
z k+1 := argmin g(z) + (ρ/2)Axk+1 + Bz − c + uk 22 (3.6)
z
uk+1 := uk + Axk+1 + Bz k+1 − c. (3.7)

Defining the residual at iteration k as rk = Axk + Bz k − c, we see that

k
uk = u0 + rj ,
j=1

the running sum of the residuals.

We call the first form of ADMM above, given by (3.2–3.4), the
unscaled form, and the second form (3.5–3.7) the scaled form, since it
is expressed in terms of a scaled version of the dual variable. The two
are clearly equivalent, but the formulas in the scaled form of ADMM
are often shorter than in the unscaled form, so we will use the scaled
form in the sequel. We will use the unscaled form when we wish to
emphasize the role of the dual variable or to give an interpretation
that relies on the (unscaled) dual variable.

3.2 Convergence
There are many convergence results for ADMM discussed in the liter-
ature; here, we limit ourselves to a basic but still very general result
that applies to all of the examples we will consider. We will make one
16 Alternating Direction Method of Multipliers

assumption about the functions f and g, and one assumption about

problem (3.1).

Assumption 1. The (extended-real-valued) functions f : Rn → R ∪

{+∞} and g : Rm → R ∪ {+∞} are closed, proper, and convex.

This assumption can be expressed compactly using the epigraphs of

the functions: The function f satisfies assumption 1 if and only if its
epigraph

epi f = {(x, t) ∈ Rn × R | f (x) ≤ t}

is a closed nonempty convex set.

Assumption 1 implies that the subproblems arising in the x-update
(3.2) and z-update (3.3) are solvable, i.e., there exist x and z, not neces-
sarily unique (without further assumptions on A and B), that minimize
the augmented Lagrangian. It is important to note that assumption 1
allows f and g to be nondifferentiable and to assume the value +∞.
For example, we can take f to be the indicator function of a closed
nonempty convex set C, i.e., f (x) = 0 for x ∈ C and f (x) = +∞ other-
wise. In this case, the x-minimization step (3.2) will involve solving a
constrained quadratic program over C, the effective domain of f .

Assumption 2. The unaugmented Lagrangian L0 has a saddle point.

Explicitly, there exist (x
, z
, y
), not necessarily unique, for which

L0 (x
, z
, y) ≤ L0 (x
, z
, y
) ≤ L0 (x, z, y
)

holds for all x, z, y.

By assumption 1, it follows that L0 (x
, z
, y
) is finite for any sad-
dle point (x
, z
, y
). This implies that (x
, z
) is a solution to (3.1),
so Ax
+ Bz
= c and f (x
) < ∞, g(z
) < ∞. It also implies that y

is dual optimal, and the optimal values of the primal and dual prob-
lems are equal, i.e., that strong duality holds. Note that we make no
assumptions about A, B, or c, except implicitly through assumption 2;
in particular, neither A nor B is required to be full rank.
 3.2 Convergence 17

3.2.1 Convergence
Under assumptions 1 and 2, the ADMM iterates satisfy the following:

• Residual convergence. rk → 0 as k → ∞, i.e., the iterates

approach feasibility.
• Objective convergence. f (xk ) + g(z k ) → p
as k → ∞, i.e.,
the objective function of the iterates approaches the optimal
value.
• Dual variable convergence. y k → y
as k → ∞, where y
is a
dual optimal point.

A proof of the residual and objective convergence results is given in

appendix A. Note that xk and z k need not converge to optimal values,
although such results can be shown under additional assumptions.

3.2.2 Convergence in Practice

Simple examples show that ADMM can be very slow to converge to
high accuracy. However, it is often the case that ADMM converges to
modest accuracy—sufficient for many applications—within a few tens
of iterations. This behavior makes ADMM similar to algorithms like
the conjugate gradient method, for example, in that a few tens of iter-
ations will often produce acceptable results of practical use. However,
the slow convergence of ADMM also distinguishes it from algorithms
such as Newton’s method (or, for constrained problems, interior-point
methods), where high accuracy can be attained in a reasonable amount
of time. While in some cases it is possible to combine ADMM with
a method for producing a high accuracy solution from a low accu-
racy solution [64], in the general case ADMM will be practically useful
mostly in cases when modest accuracy is sufficient. Fortunately, this
is usually the case for the kinds of large-scale problems we consider.
Also, in the case of statistical and machine learning problems, solving
a parameter estimation problem to very high accuracy often yields lit-
tle to no improvement in actual prediction performance, the real metric
of interest in applications.
18 Alternating Direction Method of Multipliers

3.3 Optimality Conditions and Stopping Criterion

The necessary and sufficient optimality conditions for the ADMM prob-
lem (3.1) are primal feasibility,
Ax
+ Bz
− c = 0, (3.8)
and dual feasibility,
0 ∈ ∂f (x
) + AT y
(3.9)
0 ∈ ∂g(z ) + B y .

T

(3.10)
Here, ∂ denotes the subdifferential operator; see, e.g., [140, 19, 99].
(When f and g are differentiable, the subdifferentials ∂f and ∂g can
be replaced by the gradients ∇f and ∇g, and ∈ can be replaced by =.)
Since z k+1 minimizes Lρ (xk+1 , z, y k ) by definition, we have that
0 ∈ ∂g(z k+1 ) + B T y k + ρB T (Axk+1 + Bz k+1 − c)
= ∂g(z k+1 ) + B T y k + ρB T rk+1
= ∂g(z k+1 ) + B T y k+1 .
This means that z k+1 and y k+1 always satisfy (3.10), so attaining opti-
mality comes down to satisfying (3.8) and (3.9). This phenomenon is
analogous to the iterates of the method of multipliers always being dual
feasible; see page 11.
Since xk+1 minimizes Lρ (x, z k , y k ) by definition, we have that
0 ∈ ∂f (xk+1 ) + AT y k + ρAT (Axk+1 + Bz k − c)
= ∂f (xk+1 ) + AT (y k + ρrk+1 + ρB(z k − z k+1 ))
= ∂f (xk+1 ) + AT y k+1 + ρAT B(z k − z k+1 ),
or equivalently,
ρAT B(z k+1 − z k ) ∈ ∂f (xk+1 ) + AT y k+1 .
This means that the quantity
sk+1 = ρAT B(z k+1 − z k )
can be viewed as a residual for the dual feasibility condition (3.9).
We will refer to sk+1 as the dual residual at iteration k + 1, and to
rk+1 = Axk+1 + Bz k+1 − c as the primal residual at iteration k + 1.
 3.3 Optimality Conditions and Stopping Criterion 19

In summary, the optimality conditions for the ADMM problem con-

sist of three conditions, (3.8–3.10). The last condition (3.10) always
holds for (xk+1 , z k+1 , y k+1 ); the residuals for the other two, (3.8) and
(3.9), are the primal and dual residuals rk+1 and sk+1 , respectively.
These two residuals converge to zero as ADMM proceeds. (In fact, the
convergence proof in appendix A shows B(z k+1 − z k ) converges to zero,
which implies sk converges to zero.)

3.3.1 Stopping Criteria

The residuals of the optimality conditions can be related to a bound on
the objective suboptimality of the current point, i.e., f (xk ) + g(z k ) −
p
. As shown in the convergence proof in appendix A, we have

f (xk ) + g(z k ) − p
≤ −(y k )T rk + (xk − x
)T sk . (3.11)

This shows that when the residuals rk and sk are small, the objective
suboptimality also must be small. We cannot use this inequality directly
in a stopping criterion, however, since we do not know x
. But if we
guess or estimate that xk − x
2 ≤ d, we have that

f (xk ) + g(z k ) − p
≤ −(y k )T rk + dsk 2 ≤ y k 2 rk 2 + dsk 2 .

The middle or righthand terms can be used as an approximate bound

on the objective suboptimality (which depends on our guess of d).
This suggests that a reasonable termination criterion is that the
primal and dual residuals must be small, i.e.,

rk 2 ≤ pri and sk 2 ≤ dual , (3.12)

where pri > 0 and dual > 0 are feasibility tolerances for the primal and
dual feasibility conditions (3.8) and (3.9), respectively. These tolerances
can be chosen using an absolute and relative criterion, such as
√
pri = p abs + rel max{Axk 2 , Bz k 2 , c2 },
√
dual = n abs + rel AT y k 2 ,

where abs > 0 is an absolute tolerance and rel > 0 is a relative toler-
√ √
ance. (The factors p and n account for the fact that the
2 norms are
in Rp and Rn , respectively.) A reasonable value for the relative stopping
20 Alternating Direction Method of Multipliers

criterion might be rel = 10−3 or 10−4 , depending on the application.

The choice of absolute stopping criterion depends on the scale of the
typical variable values.

3.4 Extensions and Variations

Many variations on the classic ADMM algorithm have been explored in
the literature. Here we briefly survey some of these variants, organized
into groups of related ideas. Some of these methods can give superior
convergence in practice compared to the standard ADMM presented
above. Most of the extensions have been rigorously analyzed, so the
convergence results described above are still valid (in some cases, under
some additional conditions).

3.4.1 Varying Penalty Parameter

A standard extension is to use possibly different penalty parameters ρk
for each iteration, with the goal of improving the convergence in prac-
tice, as well as making performance less dependent on the initial choice
of the penalty parameter. In the context of the method of multipliers,
this approach is analyzed in [142], where it is shown that superlinear
convergence may be achieved if ρk → ∞. Though it can be difficult to
prove the convergence of ADMM when ρ varies by iteration, the fixed-
ρ theory still applies if one just assumes that ρ becomes fixed after a
finite number of iterations.
A simple scheme that often works well is (see, e.g., [96, 169]):


 incr k if rk 2 > µsk 2
τ ρ
ρk+1 := ρk /τ decr if sk 2 > µrk 2 (3.13)


ρk otherwise,

where µ > 1, τ incr > 1, and τ decr > 1 are parameters. Typical choices
might be µ = 10 and τ incr = τ decr = 2. The idea behind this penalty
parameter update is to try to keep the primal and dual residual norms
within a factor of µ of one another as they both converge to zero.
The ADMM update equations suggest that large values of ρ place a
large penalty on violations of primal feasibility and so tend to produce
 3.4 Extensions and Variations 21

small primal residuals. Conversely, the definition of sk+1 suggests that

small values of ρ tend to reduce the dual residual, but at the expense of
reducing the penalty on primal feasibility, which may result in a larger
primal residual. The adjustment scheme (3.13) inflates ρ by τ incr when
the primal residual appears large compared to the dual residual, and
deflates ρ by τ decr when the primal residual seems too small relative
to the dual residual. This scheme may also be refined by taking into
account the relative magnitudes of pri and dual .
When a varying penalty parameter is used in the scaled form of
ADMM, the scaled dual variable uk = (1/ρ)y k must also be rescaled
after updating ρ; for example, if ρ is halved, uk should be doubled
before proceeding.

3.4.2 More General Augmenting Terms

Another idea is to allow for a different penalty parameter for each
constraint, or more generally, to replace the quadratic term (ρ/2)r22
with (1/2)rT P r, where P is a symmetric positive definite matrix. When
P is constant, we can interpret this generalized version of ADMM as
standard ADMM applied to a modified initial problem with the equality
constraints r = 0 replaced with F r = 0, where F T F = P .

3.4.3 Over-relaxation
In the z- and y-updates, the quantity Axk+1 can be replaced with

αk Axk+1 − (1 − αk )(Bz k − c),

where αk ∈ (0, 2) is a relaxation parameter ; when αk > 1, this technique

is called over-relaxation, and when αk < 1, it is called under-relaxation.
This scheme is analyzed in [63], and experiments in [59, 64] suggest that
over-relaxation with αk ∈ [1.5, 1.8] can improve convergence.

3.4.4 Inexact Minimization

ADMM will converge even when the x- and z-minimization steps
are not carried out exactly, provided certain suboptimality measures
22 Alternating Direction Method of Multipliers

in the minimizations satisfy an appropriate condition, such as being

summable. This result is due to Eckstein and Bertsekas [63], building
on earlier results by Gol’shtein and Tret’yakov [89]. This technique is
important in situations where the x- or z-updates are carried out using
an iterative method; it allows us to solve the minimizations only approx-
imately at first, and then more accurately as the iterations progress.

3.4.5 Update Ordering

Several variations on ADMM involve performing the x-, z-, and y-
updates in varying orders or multiple times. For example, we can divide
the variables into k blocks, and update each of them in turn, possibly
multiple times, before performing each dual variable update; see, e.g.,
[146]. Carrying out multiple x- and z-updates before the y-update can
be interpreted as executing multiple Gauss-Seidel passes instead of just
one; if many sweeps are carried out before each dual update, the result-
ing algorithm is very close to the standard method of multipliers [17,
§3.4.4]. Another variation is to perform an additional y-update between
the x- and z-update, with half the step length [17].

3.4.6 Related Algorithms

There are also a number of other algorithms distinct from but inspired
by ADMM. For instance, Fukushima [80] applies ADMM to a dual
problem formulation, yielding a ‘dual ADMM’ algorithm, which is
shown in [65] to be equivalent to the ‘primal Douglas-Rachford’ method
discussed in [57, §3.5.6]. As another example, Zhu et al. [183] discuss
variations of distributed ADMM (discussed in §7, §8, and §10) that
can cope with various complicating factors, such as noise in the mes-
sages exchanged for the updates, or asynchronous updates, which can
be useful in a regime when some processors or subsystems randomly
fail. There are also algorithms resembling a combination of ADMM
and the proximal method of multipliers [141], rather than the standard
method of multipliers; see, e.g., [33, 60]. Other representative publica-
tions include [62, 143, 59, 147, 158, 159, 42].
 3.5 Notes and References 23

3.5 Notes and References

ADMM was originally proposed in the mid-1970s by Glowinski and
Marrocco [86] and Gabay and Mercier [82]. There are a number of other
important papers analyzing the properties of the algorithm, including
[76, 81, 75, 87, 157, 80, 65, 33]. In particular, the convergence of ADMM
has been explored by many authors, including Gabay [81] and Eckstein
and Bertsekas [63].
ADMM has also been applied to a number of statistical prob-
lems, such as constrained sparse regression [18], sparse signal recov-
ery [70], image restoration and denoising [72, 154, 134], trace norm
regularized least squares minimization [174], sparse inverse covari-
ance selection [178], the Dantzig selector [116], and support vector
machines [74], among others. For examples in signal processing, see
[42, 40, 41, 150, 149] and the references therein.
Many papers analyzing ADMM do so from the perspective of max-
imal monotone operators [23, 141, 142, 63, 144]. Briefly, a wide variety
of problems can be posed as finding a zero of a maximal monotone
operator; for example, if f is closed, proper, and convex, then the sub-
differential operator ∂f is maximal monotone, and finding a zero of ∂f
is simply minimization of f ; such a minimization may implicitly contain
constraints if f is allowed to take the value +∞. Rockafellar’s proximal
point algorithm [142] is a general method for finding a zero of a max-
imal monotone operator, and a wide variety of algorithms have been
shown to be special cases, including proximal minimization (see §4.1),
the method of multipliers, and ADMM. For a more detailed review of
the older literature, see [57, §2].
The method of multipliers was shown to be a special case of the
proximal point algorithm by Rockafellar [141]. Gabay [81] showed that
ADMM is a special case of a method called Douglas-Rachford split-
ting for monotone operators [53, 112], and Eckstein and Bertsekas
[63] showed in turn that Douglas-Rachford splitting is a special case
of the proximal point algorithm. (The variant of ADMM that per-
forms an extra y-update between the x- and z-updates is equiva-
lent to Peaceman-Rachford splitting [137, 112] instead, as shown by
Glowinski and Le Tallec [87].) Using the same framework, Eckstein
24 Alternating Direction Method of Multipliers

and Bertsekas [63] also showed the relationships between a number of

other algorithms, such as Spingarn’s method of partial inverses [153].
Lawrence and Spingarn [108] develop an alternative framework show-
ing that Douglas-Rachford splitting, hence ADMM, is a special case
of the proximal point algorithm; Eckstein and Ferris [64] offer a more
recent discussion explaining this approach.
The major importance of these results is that they allow the pow-
erful convergence theory for the proximal point algorithm to apply
directly to ADMM and other methods, and show that many of these
algorithms are essentially identical. (But note that our proof of con-
vergence of the basic ADMM algorithm, given in appendix A, is self-
contained and does not rely on this abstract machinery.) Research on
operator splitting methods and their relation to decomposition algo-
rithms continues to this day [66, 67].
A considerable body of recent research considers replacing the
quadratic penalty term in the standard method of multipliers with a
more general deviation penalty, such as one derived from a Bregman
divergence [30, 58]; see [22] for background material. Unfortunately,
these generalizations do not appear to carry over in a straightforward
manner from non-decomposition augmented Lagrangian methods to
ADMM: There is currently no proof of convergence known for ADMM
with nonquadratic penalty terms.
 4
General Patterns

Structure in f , g, A, and B can often be exploited to carry out the

x- and z-updates more efficiently. Here we consider three general cases
that we will encounter repeatedly in the sequel: quadratic objective
terms, separable objective and constraints, and smooth objective terms.
Our discussion will be written for the x-update but applies to the z-
update by symmetry. We express the x-update step as
 
x+ = argmin f (x) + (ρ/2)Ax − v22 ,
x

where v = −Bz + c − u is a known constant vector for the purposes of

the x-update.

4.1 Proximity Operator

First, consider the simple case where A = I, which appears frequently
in the examples. Then the x-update is
 
x+ = argmin f (x) + (ρ/2)x − v22 .
x

As a function of v, the righthand side is denoted proxf,ρ (v) and is

called the proximity operator of f with penalty ρ [127]. In variational

25
26 General Patterns

analysis,
 
f˜(v) = inf f (x) + (ρ/2)x − v22
x

is known as the Moreau envelope or Moreau-Yosida regularization of f ,

and is connected to the theory of the proximal point algorithm [144].
The x-minimization in the proximity operator is generally referred to
as proximal minimization. While these observations do not by them-
selves allow us to improve the efficiency of ADMM, it does tie the
x-minimization step to other well known ideas.
When the function f is simple enough, the x-update (i.e., the prox-
imity operator) can be evaluated analytically; see [41] for many exam-
ples. For instance, if f is the indicator function of a closed nonempty
convex set C, then the x-update is
 
x+ = argmin f (x) + (ρ/2)x − v22 = ΠC (v),
x
where ΠC denotes projection (in the Euclidean norm) onto C. This holds
independently of the choice of ρ. As an example, if f is the indicator
function of the nonnegative orthant Rn+ , we have x+ = (v)+ , the vector
obtained by taking the nonnegative part of each component of v.

4.2 Quadratic Objective Terms

Suppose f is given by the (convex) quadratic function
f (x) = (1/2)xT P x + q T x + r,
where P ∈ Sn+ , the set of symmetric positive semidefinite n × n matri-
ces. This includes the cases when f is linear or constant, by setting P ,
or both P and q, to zero. Then, assuming P + ρAT A is invertible, x+
is an affine function of v given by
x+ = (P + ρAT A)−1 (ρAT v − q). (4.1)
In other words, computing the x-update amounts to solving a linear
system with positive definite coefficient matrix P + ρAT A and right-
hand side ρAT v − q. As we show below, an appropriate use of numerical
linear algebra can exploit this fact and substantially improve perfor-
mance. For general background on numerical linear algebra, see [47] or
[90]; see [20, appendix C] for a short overview of direct methods.
 4.2 Quadratic Objective Terms 27

4.2.1 Direct Methods

We assume here that a direct method is used to carry out the x-update;
the case when an iterative method is used is discussed in §4.3. Direct
methods for solving a linear system F x = g are based on first factoring
F = F1 F2 · · · Fk into a product of simpler matrices, and then computing
x = F −1 b by solving a sequence of problems of the form Fi zi = zi−1 ,
where z1 = F1−1 g and x = zk . The solve step is sometimes also called
a back-solve. The computational cost of factorization and back-solve
operations depends on the sparsity pattern and other properties of F .
The cost of solving F x = g is the sum of the cost of factoring F and
the cost of the back-solve.
In our case, the coefficient matrix is F = P + ρAT A and the right-
hand side is g = ρAT v − q, where P ∈ Sn+ and A ∈ Rp×n . Suppose we
exploit no structure in A or P , i.e., we use generic methods that work
for any matrix. We can form F = P + ρAT A at a cost of O(pn2 ) flops
(floating point operations). We then carry out a Cholesky factorization
of F at a cost of O(n3 ) flops; the back-solve cost is O(n2 ). (The cost of
forming g is negligible compared to the costs listed above.) When p is
on the order of, or more than n, the overall cost is O(pn2 ). (When p is
less than n in order, the matrix inversion lemma described below can
be used to carry out the update in O(p2 n) flops.)

4.2.2 Exploiting Sparsity

When A and P are such that F is sparse (i.e., has enough zero entries
to be worth exploiting), much more efficient factorization and back-
solve routines can be employed. As an extreme case, if P and A are
diagonal n × n matrices, then both the factor and solve costs are
O(n). If P and A are banded, then so is F . If F is banded with
bandwidth k, the factorization cost is O(nk 2 ) and the back-solve cost
is O(nk). In this case, the x-update can be carried out at a cost
O(nk 2 ), plus the cost of forming F . The same approach works when
P + ρAT A has a more general sparsity pattern; in this case, a permuted
Cholesky factorization can be used, with permutations chosen to reduce
fill-in.
28 General Patterns

4.2.3 Caching Factorizations

Now suppose we need to solve multiple linear systems, say, F x(i) = g (i) ,
i = 1, . . . , N , with the same coefficient matrix but different righthand
sides. This occurs in ADMM when the parameter ρ is not changed. In
this case, the factorization of the coefficient matrix F can be computed
once and then back-solves can be carried out for each righthand side.
If t is the factorization cost and s is the back-solve cost, then the total
cost becomes t + N s instead of N (t + s), which would be the cost if
we were to factor F each iteration. As long as ρ does not change, we
can factor P + ρAT A once, and then use this cached factorization in
subsequent solve steps. For example, if we do not exploit any structure
and use the standard Cholesky factorization, the x-update steps can
be carried out a factor n more efficiently than a naive implementation,
in which we solve the equations from scratch in each iteration.
When structure is exploited, the ratio between t and s is typically
less than n but often still significant, so here too there are performance
gains. However, in this case, there is less benefit to ρ not changing, so
we can weigh the benefit of varying ρ against the benefit of not recom-
puting the factorization of P + ρAT A. In general, an implementation
should cache the factorization of P + ρAT A and then only recompute
it if and when ρ changes.

4.2.4 Matrix Inversion Lemma

We can also exploit structure using the matrix inversion lemma, which
states that the identity

(P + ρAT A)−1 = P −1 − ρP −1 AT (I + ρAP −1 AT )−1 AP −1

holds when all the inverses exist. This means that if linear systems
with coefficient matrix P can be solved efficiently, and p is small, or
at least no larger than n in order, then the x-update can be computed
efficiently as well. The same trick as above can also be used to obtain
an efficient method for computing multiple updates: The factorization
of I + ρAP −1 AT can be cached and cheaper back-solves can be used
in computing the updates.
 4.2 Quadratic Objective Terms 29

As an example, suppose that P is diagonal and that p ≤ n. A naive

method for computing the update costs O(n3 ) flops in the first itera-
tion and O(n2 ) flops in subsequent iterations, if we store the factors of
P + ρAT A. Using the matrix inversion lemma (i.e., using the righthand
side above) to compute the x-update costs O(np2 ) flops, an improve-
ment by a factor of (n/p)2 over the naive method. In this case, the
dominant cost is forming AP −1 AT . The factors of I + ρAP −1 AT can
be saved after the first update, so subsequent iterations can be car-
ried out at cost O(np) flops, a savings of a factor of p over the first
update.
Using the matrix inversion lemma to compute x+ can also make
it less costly to vary ρ in each iteration. When P is diagonal, for
example, we can compute AP −1 AT once, and then form and factor
I + ρk AP −1 AT in iteration k at a cost of O(p3 ) flops. In other words,
the update costs an additional O(np) flops, so if p2 is less than or equal
to n in order, there is no additional cost (in order) to carrying out
updates with ρ varying in each iteration.

4.2.5 Quadratic Function Restricted to an Affine Set

The same comments hold for the slightly more complex case of a convex
quadratic function restricted to an affine set:

f (x) = (1/2)xT P x + q T x + r, dom f = {x | F x = g}.

Here, x+ is still an affine function of v, and the update involves solving

the KKT (Karush-Kuhn-Tucker) system

P + ρI F T xk+1 q − ρ(z k − uk )
+ = 0.
F 0 ν −g

All of the comments above hold here as well: Factorizations can be

cached to carry out additional updates more efficiently, and structure
in the matrices can be exploited to improve the efficiency of the factor-
ization and back-solve steps.
30 General Patterns

4.3 Smooth Objective Terms

4.3.1 Iterative Solvers
When f is smooth, general iterative methods can be used to carry
out the x-minimization step. Of particular interest are methods that
only require the ability to compute ∇f (x) for a given x, to multiply a
vector by A, and to multiply a vector by AT . Such methods can scale
to relatively large problems. Examples include the standard gradient
method, the (nonlinear) conjugate gradient method, and the limited-
memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm [113,
26]; see [135] for further details.
The convergence of these methods depends on the conditioning of
the function to be minimized. The presence of the quadratic penalty
term (ρ/2)Ax − v22 tends to improve the conditioning of the problem
and thus improve the performance of an iterative method for updating
x. Indeed, one method for adjusting the parameter ρ from iteration to
iteration is to increase it until the iterative method used to carry out
the updates converges quickly enough.

4.3.2 Early Termination

A standard technique to speed up the algorithm is to terminate the
iterative method used to carry out the x-update (or z-update) early,
i.e., before the gradient of f (x) + (ρ/2)Ax − v22 is very small. This
technique is justified by the convergence results for ADMM with inexact
minimization in the x- and z-update steps. In this case, the required
accuracy should be low in the initial iterations of ADMM and then
repeatedly increased in each iteration. Early termination in the x- or
z-updates can result in more ADMM iterations, but much lower cost
per iteration, giving an overall improvement in efficiency.

4.3.3 Warm Start

Another standard trick is to initialize the iterative method used in
the x-update at the solution xk obtained in the previous iteration.
This is called a warm start. The previous ADMM iterate often gives
a good enough approximation to result in far fewer iterations (of the
 4.4 Decomposition 31

iterative method used to compute the update xk+1 ) than if the iterative
method were started at zero or some other default initialization. This
is especially the case when ADMM has almost converged, in which case
the updates will not change significantly from their previous values.

4.3.4 Quadratic Objective Terms

Even when f is quadratic, it may be worth using an iterative method
rather than a direct method for the x-update. In this case, we can use
a standard (possibly preconditioned) conjugate gradient method. This
approach makes sense when direct methods do not work (e.g., because
they require too much memory) or when A is dense but a fast method
is available for multiplying a vector by A or AT . This is the case, for
example, when A represents the discrete Fourier transform [90].

4.4 Decomposition
4.4.1 Block Separability
Suppose x = (x1 , . . . , xN ) is a partition of the variable x into subvectors
and that f is separable with respect to this partition, i.e.,
f (x) = f1 (x1 ) + · · · + fN (xN ),

where xi ∈ R and N
ni
i=1 ni = N . If the quadratic term Ax2 is also
2

separable with respect to the partition, i.e., AT A is block diagonal

conformably with the partition, then the augmented Lagrangian Lρ is
separable. This means that the x-update can be carried out in parallel,
with the subvectors xi updated by N separate minimizations.

4.4.2 Component Separability

In some cases, the decomposition extends all the way to individual
components of x, i.e.,
f (x) = f1 (x1 ) + · · · + fn (xn ),
where fi : R → R, and AT A is diagonal. The x-minimization step can
then be carried out via n scalar minimizations, which can in some
cases be expressed analytically (but in any case can be computed very
efficiently). We will call this component separability.
32 General Patterns

4.4.3 Soft Thresholding

For an example that will come up often in the sequel, consider f (x) =
λx1 (with λ > 0) and A = I. In this case the (scalar) xi -update is
 
x+i := argmin λ|xi | + (ρ/2)(xi − vi ) .
2
xi

Even though the first term is not differentiable, we can easily compute
a simple closed-form solution to this problem by using subdifferential
calculus; see [140, §23] for background. Explicitly, the solution is

x+
i := Sλ/ρ (vi ),

where the soft thresholding operator S is defined as



a − κ
 a>κ
Sκ (a) = 0 |a| ≤ κ


a + κ a < −κ,

or equivalently,

Sκ (a) = (a − κ)+ − (−a − κ)+ .

Yet another formula, which shows that the soft thresholding operator
is a shrinkage operator (i.e., moves a point toward zero), is

Sκ (a) = (1 − κ/|a|)+ a (4.2)

(for a = 0). We refer to updates that reduce to this form as element-

wise soft thresholding. In the language of §4.1, soft thresholding is the
proximity operator of the
1 norm.
 5
Constrained Convex Optimization

The generic constrained convex optimization problem is

minimize f (x)
(5.1)
subject to x ∈ C,

with variable x ∈ Rn , where f and C are convex. This problem can be

rewritten in ADMM form (3.1) as

minimize f (x) + g(z)

subject to x − z = 0,

where g is the indicator function of C.

The augmented Lagrangian (using the scaled dual variable) is

Lρ (x, z, u) = f (x) + g(z) + (ρ/2)x − z + u22 ,

so the scaled form of ADMM for this problem is

 
xk+1 := argmin f (x) + (ρ/2)x − z k + uk 22
x
k+1
z := ΠC (xk+1 + uk )
uk+1 := uk + xk+1 − z k+1 .

33
34 Constrained Convex Optimization

The x-update involves minimizing f plus a convex quadratic function,

i.e., evaluation of the proximal operator associated with f . The z-
update is Euclidean projection onto C. The objective f need not be
smooth here; indeed, we can include additional constraints (i.e., beyond
those represented by x ∈ C) by defining f to be +∞ where they are vio-
lated. In this case, the x-update becomes a constrained optimization
problem over dom f = {x | f (x) < ∞}.
As with all problems where the constraint is x − z = 0, the primal
and dual residuals take the simple form
rk = xk − z k , sk = −ρ(z k − z k−1 ).
In the following sections we give some more specific examples.

5.1 Convex Feasibility

5.1.1 Alternating Projections
A classic problem is to find a point in the intersection of two closed
nonempty convex sets. The classical method, which dates back to the
1930s, is von Neumann’s alternating projections algorithm [166, 36, 21]:
xk+1 := ΠC (z k )
z k+1 := ΠD (xk+1 ),
where ΠC and ΠD are Euclidean projection onto the sets C and D,
respectively.
In ADMM form, the problem can be written as
minimize f (x) + g(z)
subject to x − z = 0,
where f is the indicator function of C and g is the indicator function
of D. The scaled form of ADMM is then
xk+1 := ΠC (z k − uk )
z k+1 := ΠD (xk+1 + uk ) (5.2)
u k+1 k
:= u + x k+1
−z k+1
,
so both the x and z steps involve projection onto a convex set, as in
the classical method. This is exactly Dykstra’s alternating projections
 5.1 Convex Feasibility 35

method [56, 9], which is far more efficient than the classical method
that does not use the dual variable u.
Here, the norm of the primal residual xk − z k 2 has a nice inter-
pretation. Since xk ∈ C and z k ∈ D, xk − z k 2 is an upper bound on
dist(C, D), the Euclidean distance between C and D. If we terminate
with rk 2 ≤ pri , then we have found a pair of points, one in C and
one in D, that are no more than pri far apart. Alternatively, the point
(1/2)(xk + z k ) is no more than a distance pri /2 from both C and D.

5.1.2 Parallel Projections

This method can be applied to the problem of finding a point in the
intersection of N closed convex sets A1 , . . . , AN in Rn by running the
algorithm in RnN with
C = A1 × · · · × AN , D = {(x1 , . . . , xN ) ∈ RnN | x1 = x2 = · · · = xN }.
If x = (x1 , . . . , xN ) ∈ RnN , then projection onto C is
ΠC (x) = (ΠA1 (x1 ), . . . , ΠAN (xN )),
and projection onto D is
ΠD (x) = (x, x, . . . , x),
N
where x = (1/N ) i=1 xi is the average of x1 , . . . , xN . Thus, each step
of ADMM can be carried out by projecting points onto each of the sets
Ai in parallel and then averaging the results:
xk+1
i := ΠAi (z k − uki )
1
k+1
N
z k+1 := (xi + uki )
N
i=1
uk+1
i := uki + xk+1
i − z k+1 .
Here the first and third steps are carried out in parallel, for i = 1, . . . , N .
(The description above involves a small abuse of notation in dropping
the index i from zi , since they are all the same.) This can be viewed as a
special case of constrained optimization, as described in §4.4, where the
indicator function of A1 ∩ · · · ∩ AN splits into the sum of the indicator
functions of each Ai .
36 Constrained Convex Optimization

We note for later reference a simplification of the ADMM algorithm

above. Taking the average (over i) of the last equation we obtain
uk+1 = uk + xk+1 − z k ,
combined with z k+1 = xk+1 + uk (from the second equation) we see
that uk+1 = 0. So after the first step, the average of ui is zero. This
means that z k+1 reduces to xk+1 . Using these simplifications, we arrive
at the simple algorithm
xk+1
i := ΠAi (xk − uki )
uk+1
i := uki + (xk+1
i − xk+1 ).
This shows that uki is the running sum of the the ‘discrepancies’ xki − xk
(assuming u0 = 0). The first step is a parallel projection onto the sets
Ci ; the second involves averaging the projected points.
There is a large literature on successive projection algorithms and
their many applications; see the survey by Bauschke and Borwein [10]
for a general overview, Combettes [39] for applications to image pro-
cessing, and Censor and Zenios [31, §5] for a discussion in the context
of parallel optimization.

5.2 Linear and Quadratic Programming

The standard form quadratic program (QP) is
minimize (1/2)xT P x + q T x
(5.3)
subject to Ax = b, x ≥ 0,
with variable x ∈ Rn ; we assume that P ∈ Sn+ . When P = 0, this
reduces to the standard form linear program (LP).
We express it in ADMM form as
minimize f (x) + g(z)
subject to x − z = 0,
where
f (x) = (1/2)xT P x + q T x, dom f = {x | Ax = b}
is the original objective with restricted domain and g is the indicator
function of the nonnegative orthant Rn+ .
 5.2 Linear and Quadratic Programming 37

The scaled form of ADMM consists of the iterations

 
xk+1 := argmin f (x) + (ρ/2)x − z k + uk 22
x
k+1 k+1
z := (x + uk )+
uk+1 := uk + xk+1 − z k+1 .

As described in §4.2.5, the x-update is an equality-constrained least

squares problem with optimality conditions

P + ρI AT xk+1 q − ρ(z k − uk )
+ = 0.
A 0 ν −b

All of the comments on efficient computation from §4.2, such as storing

factorizations so that subsequent iterations can be carried out cheaply,
also apply here. For example, if P is diagonal, possibly zero, the first
x-update can be carried out at a cost of O(np2 ) flops, where p is the
number of equality constraints in the original quadratic program. Sub-
sequent updates only cost O(np) flops.

5.2.1 Linear and Quadratic Cone Programming

More generally, any conic constraint x ∈ K can be used in place of the
constraint x ≥ 0, in which case the standard quadratic program (5.3)
becomes a quadratic conic program. The only change to ADMM is in
the z-update, which then involves projection onto K. For example, we
can solve a semidefinite program with x ∈ Sn+ by projecting xk+1 + uk
onto Sn+ using an eigenvalue decomposition.
 6

1 -Norm Problems

The problems addressed in this section will help illustrate why ADMM
is a natural fit for machine learning and statistical problems in particu-
lar. The reason is that, unlike dual ascent or the method of multipliers,
ADMM explicitly targets problems that split into two distinct parts, f
and g, that can then be handled separately. Problems of this form are
pervasive in machine learning, because a significant number of learning
problems involve minimizing a loss function together with a regulariza-
tion term or side constraints. In other cases, these side constraints are
introduced through problem transformations like putting the problem
in consensus form, as will be discussed in §7.1.
This section contains a variety of simple but important problems
involving
1 norms. There is widespread current interest in many of these
problems across statistics, machine learning, and signal processing, and
applying ADMM yields interesting algorithms that are state-of-the-art,
or closely related to state-of-the-art methods. We will see that ADMM
naturally lets us decouple the nonsmooth
1 term from the smooth loss
term, which is computationally advantageous. In this section, we focus on
the non-distributed versions of these problems for simplicity; the problem
of distributed model fitting will be treated in the sequel.

38
 6.1 Least Absolute Deviations 39

The idea of
1 regularization is decades old, and traces back to

Huber’s [100] work on robust statistics and a paper of Claerbout [38]
in geophysics. There is a vast literature, but some important modern
papers are those on total variation denoising [145], soft thresholding
[49], the lasso [156], basis pursuit [34], compressed sensing [50, 28, 29],
and structure learning of sparse graphical models [123].
Because of the now widespread use of models incorporating an
1
penalty, there has also been considerable research on optimization algo-
rithms for such problems. A recent survey by Yang et al. [173] com-
pares and benchmarks a number of representative algorithms, includ-
ing gradient projection [73, 102], homotopy methods [52], iterative
shrinkage-thresholding [45], proximal gradient [132, 133, 11, 12], aug-
mented Lagrangian methods [175], and interior-point methods [103].
There are other approaches as well, such as Bregman iterative algo-
rithms [176] and iterative thresholding algorithms [51] implementable
in a message-passing framework.

6.1 Least Absolute Deviations

A simple variant on least squares fitting is least absolute deviations,
in which we minimize Ax − b1 instead of Ax − b22 . Least absolute
deviations provides a more robust fit than least squares when the data
contains large outliers, and has been used extensively in statistics and
econometrics. See, for example, [95, §10.6], [171, §9.6], and [20, §6.1.2].
In ADMM form, the problem can be written as
minimize z1
subject to Ax − z = b,
so f = 0 and g =  · 1 . Exploiting the special form of f and g, and
assuming AT A is invertible, ADMM can be expressed as
xk+1 := (AT A)−1 AT (b + z k − uk )
z k+1 := S1/ρ (Axk+1 − b + uk )
uk+1 := uk + Axk+1 − z k+1 − b,
where the soft thresholding operator is interpreted elementwise. As in
§4.2, the matrix AT A can be factored once; the factors are then used
in cheaper back-solves in subsequent x-updates.
40
1 -Norm Problems

The x-update step is the same as carrying out a least squares fit

with coefficient matrix A and righthand side b + z k − uk . Thus ADMM
can be interpreted as a method for solving a least absolute deviations
problem by iteratively solving the associated least squares problem with
a modified righthand side; the modification is then updated using soft
thresholding. With factorization caching, the cost of subsequent least
squares iterations is much smaller than the initial one, often making
the time required to carry out least absolute deviations very nearly the
same as the time required to carry out least squares.

6.1.1 Huber Fitting

A problem that lies in between least squares and least absolute devia-
tions is Huber function fitting,

minimize g hub (Ax − b),

where the Huber penalty function g hub is quadratic for small arguments
and transitions to an absolute value for larger values. For scalar a, it
is given by

a2 /2 |a| ≤ 1
g hub (a) =
|a| − 1/2 |a| > 1

and extends to vector arguments as the sum of the Huber functions

of the components. (For simplicity, we consider the standard Huber
function, which transitions from quadratic to absolute value at the
level 1.)
This can be put into ADMM form as above, and the ADMM algo-
rithm is the same except that the z-update involves the proximity oper-
ator of the Huber function rather than that of the
1 norm:

ρ  k+1  1
z k+1 := Ax − b + uk + S (Axk+1 − b + uk ).
1+ρ 1 + ρ 1+1/ρ

When the least squares fit xls = (AT A)−1 b satisfies |xls

i | ≤ 1 for all i, it
is also the Huber fit. In this case, ADMM terminates in two steps.
 6.2 Basis Pursuit 41

6.2 Basis Pursuit

Basis pursuit is the equality-constrained
1 minimization problem
minimize x1
subject to Ax = b,
with variable x ∈ Rn , data A ∈ Rm×n , b ∈ Rm , with m < n. Basis pur-
suit is often used as a heuristic for finding a sparse solution to an
underdetermined system of linear equations. It plays a central role in
modern statistical signal processing, particularly the theory of com-
pressed sensing; see [24] for a recent survey.
In ADMM form, basis pursuit can be written as
minimize f (x) + z1
subject to x − z = 0,
where f is the indicator function of {x ∈ Rn | Ax = b}. The ADMM
algorithm is then
xk+1 := Π(z k − uk )
z k+1 := S1/ρ (xk+1 + uk )
uk+1 := uk + xk+1 − z k+1 ,
where Π is projection onto {x ∈ Rn | Ax = b}. The x-update, which
involves solving a linearly-constrained minimum Euclidean norm prob-
lem, can be written explicitly as
xk+1 := (I − AT (AAT )−1 A)(z k − uk ) + AT (AAT )−1 b.
Again, the comments on efficient computation from §4.2 apply; by
caching a factorization of AAT , subsequent projections are much
cheaper than the first one. We can interpret ADMM for basis pur-
suit as reducing the solution of a least
1 norm problem to solving a
sequence of minimum Euclidean norm problems. For a discussion of
similar algorithms for related problems in image processing, see [2].
A recent class of algorithms called Bregman iterative methods have
attracted considerable interest for solving
1 problems like basis pursuit.
For basis pursuit and related problems, Bregman iterative regularization
[176] is equivalent to the method of multipliers, and the split Bregman
method [88] is equivalent to ADMM [68].
42
1 -Norm Problems

6.3 General
1 Regularized Loss Minimization

Consider the generic problem

minimize l(x) + λx1 , (6.1)

where l is any convex loss function.

In ADMM form, this problem can be written as

minimize l(x) + g(z)

subject to x − z = 0,

where g(z) = λz1 . The algorithm is

 
xk+1 := argmin l(x) + (ρ/2)x − z k + uk 22
x
k+1
z := Sλ/ρ (xk+1 + uk )
uk+1 := uk + xk+1 − z k+1 .

The x-update is a proximal operator evaluation. If l is smooth, this can

be done by any standard method, such as Newton’s method, a quasi-
Newton method such as L-BFGS, or the conjugate gradient method.
If l is quadratic, the x-minimization can be carried out by solving lin-
ear equations, as in §4.2. In general, we can interpret ADMM for
1
regularized loss minimization as reducing it to solving a sequence of
2
(squared) regularized loss minimization problems.
A very wide variety of models can be represented with the loss
function l, including generalized linear models [122] and generalized
additive models [94]. In particular, generalized linear models subsume
linear regression, logistic regression, softmax regression, and Poisson
regression, since they allow for any exponential family distribution. For
general background on models like
1 regularized logistic regression, see,
e.g., [95, §4.4.4].
In order to use a regularizer g(z) other than z1 , we simply replace
the soft thresholding operator in the z-update with the proximity oper-
ator of g, as in §4.1.
 6.4 Lasso 43

6.4 Lasso
An important special case of (6.1) is
1 regularized linear regression,
also called the lasso [156]. This involves solving

minimize (1/2)Ax − b22 + λx1 , (6.2)

where λ > 0 is a scalar regularization parameter that is usually cho-

sen by cross-validation. In typical applications, there are many more
features than training examples, and the goal is to find a parsimo-
nious model for the data. For general background on the lasso, see [95,
§3.4.2]. The lasso has been widely applied, particularly in the analy-
sis of biological data, where only a small fraction of a huge number of
possible factors are actually predictive of some outcome of interest; see
[95, §18.4] for a representative case study.
In ADMM form, the lasso problem can be written as
minimize f (x) + g(z)
subject to x − z = 0,

where f (x) = (1/2)Ax − b22 and g(z) = λz1 . By §4.2 and §4.4,
ADMM becomes

xk+1 := (AT A + ρI)−1 (AT b + ρ(z k − uk ))

z k+1 := Sλ/ρ (xk+1 + uk )
uk+1 := uk + xk+1 − z k+1 .

Note that AT A + ρI is always invertible, since ρ > 0. The x-update

is essentially a ridge regression (i.e., quadratically regularized least
squares) computation, so ADMM can be interpreted as a method for
solving the lasso problem by iteratively carrying out ridge regression.
When using a direct method, we can cache an initial factorization to
make subsequent iterations much cheaper. See [1] for an example of an
application in image processing.

6.4.1 Generalized Lasso

The lasso problem can be generalized to

minimize (1/2)Ax − b22 + λF x1 , (6.3)

44
1 -Norm Problems

where F is an arbitrary linear transformation. An important special

case is when F ∈ R(n−1)×n is the difference matrix,

 1 j=i+1
Fij = −1 j = i

0 otherwise,

and A = I, in which case the generalization reduces to


minimize (1/2)x − b22 + λ n−1 i=1 |xi+1 − xi |. (6.4)

The second term is the total variation of x. This problem is often called
total variation denoising [145], and has applications in signal process-
ing. When A = I and F is a second difference matrix, the problem (6.3)
is called
1 trend filtering [101].
In ADMM form, the problem (6.3) can be written as

minimize (1/2)Ax − b22 + λz1

subject to F x − z = 0,

which yields the ADMM algorithm

xk+1 := (AT A + ρF T F )−1 (AT b + ρF T (z k − uk ))

z k+1 := Sλ/ρ (F xk+1 + uk )
uk+1 := uk + F xk+1 − z k+1 .

For the special case of total variation denoising (6.4), AT A + ρF T F

is tridiagonal, so the x-update can be carried out in O(n) flops [90, §4.3].
For
1 trend filtering, the matrix is pentadiagonal, so the x-update is
still O(n) flops.

6.4.2 Group Lasso

As another example, consider replacing the regularizer x1 with
N
i=1 xi 2 , where x = (x1 , . . . , xN ), with xi ∈ R . When ni = 1 and
ni

N = n, this reduces to the
1 regularized problem (6.1). Here the reg-

ularizer is separable with respect to the partition x1 , . . . , xN but not
fully separable. This extension of
1 norm regularization is called the
group lasso [177] or, more generally, sum-of-norms regularization [136].
 6.5 Sparse Inverse Covariance Selection 45

ADMM for this problem is the same as above with the z-update
replaced with block soft thresholding

zik+1 = Sλ/ρ (xk+1

i + uk ), i = 1, . . . , N,

where the vector soft thresholding operator Sκ : Rm → Rm is

Sκ (a) = (1 − κ/a2 )+ a,

with Sκ (0) = 0. This formula reduces to the scalar soft threshold-

ing operator when a is a scalar, and generalizes the expression given
in (4.2).
This can be extended further to handle overlapping groups, which
is often useful in bioinformatics and other applications [181, 118]. In
this case, we have N potentially overlapping groups Gi ⊆ {1, . . . , n} of
variables, and the objective is


N
(1/2)Ax − b22 + λ xGi 2 ,
i=1

where xGi is the subvector of x with entries in Gi . Because the groups

can overlap, this kind of objective is difficult to optimize with many
standard methods, but it is straightforward with ADMM. To use
ADMM, introduce N new variables xi ∈ R|Gi | and consider the problem

minimize (1/2)Az − b22 + λ N i=1 xi 2
subject to xi − z̃i = 0, i = 1, . . . , N,
with local variables xi and global variable z. Here, z̃i is the global
variable z’s idea of what the local variable xi should be, and is given
by a linear function of z. This follows the notation for general form
consensus optimization outlined in full detail in §7.2; the overlapping
group lasso problem above is a special case.

6.5 Sparse Inverse Covariance Selection

Given a dataset consisting of samples from a zero mean Gaussian dis-
tribution in Rn ,

ai ∼ N (0, Σ), i = 1, . . . , N,
46
1 -Norm Problems

consider the task of estimating the covariance matrix Σ under the prior
assumption that Σ−1 is sparse. Since (Σ−1 )ij is zero if and only if
the ith and jth components of the random variable are conditionally
independent, given the other variables, this problem is equivalent to the
structure learning problem of estimating the topology of the undirected
graphical model representation of the Gaussian [104]. Determining the
sparsity pattern of the inverse covariance matrix Σ−1 is also called the
covariance selection problem.
For n very small, it is feasible to search over all sparsity patterns
in Σ−1 since for a fixed sparsity pattern, determining the maximum
likelihood estimate of Σ is a tractable (convex optimization) problem.
A good heuristic that scales to much larger values of n is to minimize
the negative log-likelihood (with respect to the parameter X = Σ−1 )
with an
1 regularization term to promote sparsity of the estimated
inverse covariance matrix [7]. If S is the empirical covariance matrix

(1/N ) N T
i=1 ai ai , then the estimation problem can be written as

minimize Tr(SX) − log det X + λX1 ,

with variable X ∈ Sn+ , where  · 1 is defined elementwise, i.e., as the
sum of the absolute values of the entries, and the domain of log det is
Sn++ , the set of symmetric positive definite n × n matrices. This is a
special case of the general
1 regularized problem (6.1) with (convex)
loss function l(X) = Tr(SX) − log det X.
The idea of covariance selection is originally due to Dempster [48]
and was first studied in the sparse, high-dimensional regime by Mein-
shausen and Bühlmann [123]. The form of the problem above is due to
Banerjee et al. [7]. Some other recent papers on this problem include
Friedman et al.’s graphical lasso [79], Duchi et al. [55], Lu [115], Yuan
[178], and Scheinberg et al. [148], the last of which shows that ADMM
outperforms state-of-the-art methods for this problem.
The ADMM algorithm for sparse inverse covariance selection is
 
X k+1 := argmin Tr(SX) − log det X + (ρ/2)X − Z k + U k 2F
X
 
Z k+1
:= argmin λZ1 + (ρ/2)X k+1 − Z + U k 2F
Z
U k+1 := U k + X k+1 − Z k+1 ,
 6.5 Sparse Inverse Covariance Selection 47

where  · F is the Frobenius norm, i.e., the square root of the sum of
the squares of the entries.
This algorithm can be simplified much further. The Z-minimization
step is elementwise soft thresholding,
k+1 k+1
Zij := Sλ/ρ (Xij + Uijk ),

and the X-minimization also turns out to have an analytical solution.

The first-order optimality condition is that the gradient should vanish,

S − X −1 + ρ(X − Z k + U k ) = 0,

together with the implicit constraint X  0. Rewriting, this is

ρX − X −1 = ρ(Z k − U k ) − S. (6.5)

We will construct a matrix X that satisfies this condition and thus min-
imizes the X-minimization objective. First, take the orthogonal eigen-
value decomposition of the righthand side,

ρ(Z k − U k ) − S = QΛQT ,

where Λ = diag(λ1 , . . . , λn ), and QT Q = QQT = I. Multiplying (6.5)

by QT on the left and by Q on the right gives

ρX̃ − X̃ −1 = Λ,

where X̃ = QT XQ. We can now construct a diagonal solution of this

equation, i.e., find positive numbers X̃ii that satisfy ρX̃ii − 1/X̃ii = λi .
By the quadratic formula,

λi + λ2i + 4ρ
X̃ii = ,
2ρ

which are always positive since ρ > 0. It follows that X = QX̃QT sat-
isfies the optimality condition (6.5), so this is the solution to the X-
minimization. The computational effort of the X-update is that of an
eigenvalue decomposition of a symmetric matrix.
 7
Consensus and Sharing

Here we describe two generic optimization problems, consensus and

sharing, and ADMM-based methods for solving them using distributed
optimization. Consensus problems have a long history, especially in
conjunction with ADMM; see, e.g., Bertsekas and Tsitsiklis [17] for a
discussion of distributed consensus problems in the context of ADMM
from the 1980s. Some more recent examples include a survey by Nedić
and Ozdaglar [131] and several application papers by Giannakis and
co-authors in the context of signal processing and wireless communica-
tions, such as [150, 182, 121].

7.1 Global Variable Consensus Optimization

We first consider the case with a single global variable, with the objec-
tive and constraint terms split into N parts:
N
minimize f (x) = i=1 fi (x),

where x ∈ Rn , and fi : Rn → R ∪ {+∞} are convex. We refer to fi as

the ith term in the objective. Each term can also encode constraints
by assigning fi (x) = +∞ when a constraint is violated. The goal is to

48
 7.1 Global Variable Consensus Optimization 49

solve the problem above in such a way that each term can be handled
by its own processing element, such as a thread or processor.
This problem arises in many contexts. In model fitting, for exam-
ple, x represents the parameters in a model and fi represents the loss
function associated with the ith block of data or measurements. In this
case, we would say that x is found by collaborative filtering, since the
data sources are ‘collaborating’ to develop a global model.
This problem can be rewritten with local variables xi ∈ Rn and a
common global variable z:
N
minimize i=1 fi (xi ) (7.1)
subject to xi − z = 0, i = 1, . . . , N.

This is called the global consensus problem, since the constraint is that
all the local variables should agree, i.e., be equal. Consensus can be

viewed as a simple technique for turning additive objectives N i=1 fi (x),
which show up frequently but do not split due to the variable being

shared across terms, into separable objectives N i=1 fi (xi ), which split
easily. For a useful recent discussion of consensus algorithms, see [131]
and the references therein.
ADMM for the problem (7.1) can be derived either directly from
the augmented Lagrangian

N
 
Lρ (x1 , . . . , xN , z, y) = fi (xi ) + yiT (xi − z) + (ρ/2)xi − z22 ,
i=1

or simply as a special case of the constrained optimization problem (5.1)

with variable (x1 , . . . , xN ) ∈ RnN and constraint set

C = {(x1 , . . . , xN ) | x1 = x2 = · · · = xN }.

The resulting ADMM algorithm is the following:

 
xk+1
i := argmin fi (xi ) + yikT (xi − z k ) + (ρ/2)xi − z k 22
xi

1
 k+1 
N
z k+1 := xi + (1/ρ)yik
N
i=1
yik+1 := yik + ρ(xk+1
i − z k+1 ).
50 Consensus and Sharing

Here, we write y kT instead of (y k )T to lighten the notation. The first

and last steps are carried out independently for each i = 1, . . . , N . In
the literature, the processing element that handles the global variable
z is sometimes called the central collector or the fusion center. Note
that the z-update is simply the projection of xk+1 + (1/ρ)y k onto the
constraint set C of ‘block-constant’ vectors.
This algorithm can be simplified further. With the average (over
i = 1, . . . , N ) of a vector denoted with an overline, the z-update can be
written

z k+1 = xk+1 + (1/ρ)y k .

Similarly, averaging the y-update gives

y k+1 = y k + ρ(xk+1 − z k+1 ).

Substituting the first equation into the second shows that y k+1 = 0,
i.e., the dual variables have average value zero after the first iteration.
Using z k = xk , ADMM can be written as
 
xk+1
i := argmin f (x
i i ) + y kT
i (xi − xk
) + (ρ/2)xi − x 
k 2
2
xi

yik+1 := yik + ρ(xk+1

i − xk+1 ).

We have already seen a special case of this in parallel projections (see

§5.1.2), which is consensus ADMM for the case when fi are all indicator
functions of convex sets.
This is a very intuitive algorithm. The dual variables are separately
updated to drive the variables into consensus, and quadratic regular-
ization helps pull the variables toward their average value while still
attempting to minimize each local fi .
We can interpret consensus ADMM as a method for solving prob-
lems in which the objective and constraints are distributed across mul-
tiple processors. Each processor only has to handle its own objective
and constraint term, plus a quadratic term which is updated each iter-
ation. The quadratic terms (or more accurately, the linear parts of the
quadratic terms) are updated in such a way that the variables converge
to a common value, which is the solution of the full problem.
 7.1 Global Variable Consensus Optimization 51

For consensus ADMM, the primal and dual residuals are

rk = (xk1 − xk , . . . , xkN − xk ), sk = −ρ(xk − xk−1 , . . . , xk − xk−1 ),

so their (squared) norms are

N
rk 22 = xki − xk 22 , sk 22 = N ρ2 xk − xk−1 22 .
i=1

The first term is N times the standard deviation of the points

x1 , . . . , xN , a natural measure of (lack of) consensus.
When the original consensus problem is a parameter fitting problem,
the x-update step has an intuitive statistical interpretation. Suppose
fi is the negative log-likelihood function for the parameter x, given the
measurements or data available to the ith processing element. Then
xk+1
i is precisely the maximum a posteriori (MAP) estimate of the
parameter, given the Gaussian prior distribution N (xk + (1/ρ)yik , ρI).
The expression for the prior mean is also intuitive: It is the average
value xk of the local parameter estimates in the previous iteration,
translated slightly by yik , the ‘price’ of the ith processor disagree-
ing with the consensus in the previous iteration. Note also that the
use of different forms of penalty in the augmented term, as discussed
in §3.4, will lead to corresponding changes in this prior distribution;
for example, using a matrix penalty P rather than a scalar ρ will
mean that the Gaussian prior distribution has covariance P rather
than ρI.

7.1.1 Global Variable Consensus with Regularization

In a simple variation on the global variable consensus problem, an
objective term g, often representing a simple constraint or regulariza-
tion, is handled by the central collector:
N
minimize i=1 fi (xi ) + g(z)
(7.2)
subject to xi − z = 0, i = 1, . . . , N.
52 Consensus and Sharing

The resulting ADMM algorithm is

 
xk+1
i := argmin fi (xi ) + y kT
i (xi − z k
) + (ρ/2)xi − z 
k 2
2 (7.3)
xi
 


N
z k+1 := argmin g(z) + (−yikT z + (ρ/2)xk+1
i − z22 ) (7.4)
z
i=1
yik+1 := yik + ρ(xk+1
i −z k+1
). (7.5)

By collecting the linear and quadratic terms, we can express the z-

update as an averaging step, as in consensus ADMM, followed by a
proximal step involving g:
 
z k+1 := argmin g(z) + (N ρ/2)z − xk+1 − (1/ρ)y k 22 .
z

In the case with nonzero g, we do not in general have y k = 0, so we

cannot drop the yi terms from z-update as in consensus ADMM.
As an example, for g(z) = λz1 , with λ > 0, the second step of the
z-update is a soft threshold operation:

z k+1 := Sλ/N ρ (xk+1 − (1/ρ)y k ).

As another simple example, suppose g is the indicator function of Rn+ ,

which means that the g term enforces nonnegativity of the variable. In
this case, the update is

z k+1 := (xk+1 − (1/ρ)y k )+ .

The scaled form of ADMM for this problem also has an appealing
form, which we record here for convenience:
 
xk+1
i := argmin fi (xi ) + (ρ/2)xi − z k
+ u 
k 2
i 2 (7.6)
xi
 
z k+1 := argmin g(z) + (N ρ/2)z − xk+1 − uk 22 (7.7)
z
uk+1
i := uki + xk+1
i − z k+1 . (7.8)

In many cases, this version is simpler and easier to work with than the
unscaled form.
 7.2 General Form Consensus Optimization 53

7.2 General Form Consensus Optimization

We now consider a more general form of the consensus minimization
problem, in which we have local variables xi ∈ Rni , i = 1, . . . , N , with
the objective f1 (x1 ) + · · · + fN (xN ) separable in the xi . Each of these
local variables consists of a selection of the components of the global
variable z ∈ Rn ; that is, each component of each local variable corre-
sponds to some global variable component zg . The mapping from local
variable indices into global variable index can be written as g = G(i, j),
which means that local variable component (xi )j corresponds to global
variable component zg .
Achieving consensus between the local variables and the global vari-
able means that

(xi )j = zG(i,j) , i = 1, . . . , N, j = 1, . . . , ni .

If G(i, j) = j for all i, then each local variable is just a copy of

the global variable, and consensus reduces to global variable consen-
sus, xi = z. General consensus is of interest in cases where ni  n,
so each local vector only contains a small number of the global
variables.
In the context of model fitting, the following is one way that general
form consensus naturally arises. The global variable z is the full fea-
ture vector (i.e., vector of model parameters or independent variables
in the data), and different subsets of the data are spread out among N
processors. Then xi can be viewed as the subvector of z corresponding
to (nonzero) features that appear in the ith block of data. In other
words, each processor handles only its block of data and only the sub-
set of model coefficients that are relevant for that block of data. If in
each block of data all regressors appear with nonzero values, then this
reduces to global consensus.
For example, if each training example is a document, then the fea-
tures may include words or combinations of words in the document; it
will often be the case that some words are only used in a small sub-
set of the documents, in which case each processor can just deal with
the words that appear in its local corpus. In general, datasets that are
high-dimensional but sparse will benefit from this approach.
54 Consensus and Sharing

Fig. 7.1. General form consensus optimization. Local objective terms are on the left; global
variable components are on the right. Each edge in the bipartite graph is a consistency
constraint, linking a local variable and a global variable component.

For ease of notation, let z̃i ∈ Rni be defined by (z̃i )j = zG(i,j) .

Intuitively, z̃i is the global variable’s idea of what the local variable
xi should be; the consensus constraint can then be written very simply
as xi − z̃i = 0, i = 1, . . . , N .
The general form consensus problem is
N
minimize i=1 fi (xi ) (7.9)
subject to xi − z̃i = 0, i = 1, . . . , N,

with variables x1 , . . . , xN and z (z̃i are linear functions of z).

A simple example is shown in Figure 7.1. In this example, we have
N = 3 subsystems, global variable dimension n = 4, and local variable
dimensions n1 = 4, n2 = 2, and n3 = 3. The objective terms and global
variables form a bipartite graph, with each edge representing a con-
sensus constraint between a local variable component and a global
variable.
The augmented Lagrangian for (7.9) is

N
 
Lρ (x, z, y) = fi (xi ) + yiT (xi − z̃i ) + (ρ/2)xi − z̃i 22 ,
i=1
 7.2 General Form Consensus Optimization 55

with dual variable yi ∈ Rni . Then ADMM consists of the iterations

 
xk+1
i := argmin f (x
i i ) + y kT
i xi + (ρ/2)x i − z̃ 
k 2
i 2
xi
m 

 
z k+1 := argmin −yikT z̃i + (ρ/2)xk+1
i − z̃i 22
z
i=1
yik+1 := yik + ρ(xi −
k+1
z̃ik+1 ),
where the xi - and yi -updates can be carried out independently in par-
allel for each i.
The z-update step decouples across the components of z, since Lρ
is fully separable in its components:
  
k+1 k)
G(i,j)=g (xi )j + (1/ρ)(y i j
zgk+1 :=  ,
G(i,j)=g 1

so zg is found by averaging all entries of xk+1

i + (1/ρ)yik that correspond
to the global index g. Applying the same type of argument as in the
global variable consensus case, we can show that after the first iteration,


(yik )j = 0,
G(i,j)=g

i.e., the sum of the dual variable entries that correspond to any given
global index g is zero. The z-update step can thus be written in the
simpler form


zgk+1 := (1/kg ) (xk+1
i )j ,
G(i,j)=g

where kg is the number of local variable entries that correspond to

global variable entry zg . In other words, the z-update is local averaging
for each component zg rather than global averaging; in the language of
collaborative filtering, we could say that only the processing elements
that have an opinion on a feature zg will vote on zg .

7.2.1 General Form Consensus with Regularization

As in the global consensus case, the general form consensus problem
can be generalized by allowing the global variable nodes to handle an
56 Consensus and Sharing

objective term. Consider the problem

N
minimize i=1 fi (xi ) + g(z) (7.10)
subject to xi − z̃i = 0, i = 1, . . . , N,

where g is a regularization function. The z-update involves the local

averaging step from the unregularized setting, followed by an applica-
tion of the proximity operator proxg,kg ρ to the results of this averaging,
just as in the global variable consensus case.

7.3 Sharing
Another canonical problem that will prove useful in the sequel is the
sharing problem
N N
minimize i=1 fi (xi ) + g( i=1 xi ) (7.11)

with variables xi ∈ Rn , i = 1, . . . , N , where fi is a local cost function for

subsystem i, and g is the shared objective, which takes as argument the
sum of the variables. We can think of the variable xi as being the choice
of agent i; the sharing problem involves each agent adjusting its variable
to minimize its individual cost fi (xi ), as well as the shared objective

term g( N i=1 xi ). The sharing problem is important both because many
useful problems can be put into this form and because it enjoys a dual
relationship with the consensus problem, as discussed below.
Sharing can be written in ADMM form by copying all the variables:
N N
minimize i=1 fi (xi ) + g( i=1 zi ) (7.12)
subject to xi − zi = 0, i = 1, . . . , N,

with variables xi , zi ∈ Rn , i = 1, . . . , N . The scaled form of ADMM is

 
xk+1
i := argmin f (x
i i ) + (ρ/2)x i − z k
i + u 
k 2
i 2
xi
  N 
z k+1 := argmin g( N i=1 z i ) + (ρ/2) i=1 z i − u k − xk+1 2
i i 2
z
uk+1
i := uki + xk+1
i − zik+1 .

The first and last steps can be carried out independently in parallel for
each i = 1, . . . , N . As written, the z-update requires solving a problem
 7.3 Sharing 57

in N n variables, but we will show that it is possible to carry it out by

solving a problem in only n variables.
For simplicity of notation, let ai = uki + xk+1
i . Then the z-update
can be rewritten as

minimize g(N z) + (ρ/2) N z − ai 22
N i=1 i
subject to z = (1/N ) i=1 zi ,
with additional variable z ∈ Rn . Minimizing over z1 , . . . , zN with z fixed
has the solution

zi = ai + z − a, (7.13)
so the z-update can be computed by solving the unconstrained problem

minimize g(N z) + (ρ/2) N i=1 z − a2
2

for z ∈ Rn and then applying (7.13). Substituting (7.13) for zik+1 in

the u-update gives
uk+1
i = uk + xk+1 − z k+1 , (7.14)
which shows that the dual variables uki are all equal (i.e., in consensus)
and can be replaced with a single dual variable u ∈ Rm . Substituting
in the expression for zik in the x-update, the final algorithm becomes
 
xk+1
i := argmin f i (xi ) + (ρ/2)xi − xk
i + xk
− z k
+ u 
k 2
2
xi
 
z k+1 := argmin g(N z) + (N ρ/2)z − uk − xk+1 22
z
u k+1
:= u + xk+1 − z k+1 .
k

The x-update can be carried out in parallel, for i = 1, . . . , N . The z-

update step requires gathering xk+1
i to form the averages, and then
solving a problem with n variables. After the u-update, the new value
of xk+1 − z k+1 + uk+1 is scattered to the subsystems.

7.3.1 Duality
Attaching Lagrange multipliers νi to the constraints xi − zi = 0, the
dual function Γ of the ADMM sharing problem (7.12) is given by
 
−g ∗ (ν1 ) − i fi∗ (−νi ) if ν1 = ν2 = · · · = νN
Γ(ν1 , . . . , νN ) =
−∞ otherwise.
58 Consensus and Sharing

Letting ψ = g ∗ and hi (ν) = fi∗ (−ν), the dual sharing problem can be
written as
N
minimize i=1 hi (νi ) + ψ(ν) (7.15)
subject to νi − ν = 0,

with variables ν ∈ Rn , νi ∈ Rn , i = 1, . . . , N . This is identical to the

regularized global variable consensus problem (7.2). Assuming strong
duality holds, this implies that y k = ρuk → ν
in ADMM, where ν
is
an optimal point of (7.15).
Consider the reverse direction. Attaching Lagrange multipliers
di ∈ Rn to the constraints νi − ν = 0, the dual of the regularized global
consensus problem is
N N
minimize i=1 fi (di ) + g( i=1 di )

with variables di ∈ Rn , which is exactly the sharing problem (7.11).

(This follows because f and g are assumed to be convex and closed, so
f ∗∗ = f and g ∗∗ = g.) Assuming strong duality holds, running ADMM
on the consensus problem (7.15) gives that dki → x
i , where x
i is an
optimal point of the sharing problem (7.11).
Thus, there is a close dual relationship between the consensus prob-
lem (7.15) and the sharing problem (7.11). In fact, the global consensus
problem can be solved by running ADMM on its dual sharing problem,
and vice versa. This is related to work by Fukushima [80] on ‘dual
ADMM’ methods.

7.3.2 Optimal Exchange

Here, we highlight an important special case of the sharing problem
with an appealing economic interpretation. The exchange problem is
N
minimize i=1 fi (xi )
N (7.16)
subject to i=1 xi = 0,

with variables xi ∈ Rn , i = 1, . . . , N , where fi represents the cost func-

tion for subsystem i. This is a sharing problem where the shared objec-
tive g is the indicator function of the set {0}. The components of
the vectors xi represent quantities of commodities that are exchanged
 7.3 Sharing 59

among N agents or subsystems. When (xi )j is nonnegative, it can be

viewed as the amount of commodity j received by subsystem i from the
exchange. When (xi )j is negative, its magnitude |(xi )j | can be viewed as
the amount of commodity j contributed by subsystem i to the exchange.
The equilibrium constraint that each commodity clears, or balances, is

simply N i=1 xi = 0. As this interpretation suggests, this and related
problems have a long history in economics, particularly in the theories
of market exchange, resource allocation, and general equilibrium; see,
for example, the classic works by Walras [168], Arrow and Debreu [4],
and Uzawa [162, 163].
The exchange problem can be solved via ADMM either by applying
the generic sharing algorithm above and simplifying, or by treating it
as a generic constrained convex problem (5.1), with

C = x ∈ RnN | x1 + · · · + xN = 0} .
This gives the exchange ADMM algorithm
 
xk+1
i := argmin f (x
i i ) + (ρ/2)xi − xk
i + xk
+ u 
k 2
2
xi
k+1
u := u + xk+1 .
k

It is also instructive to consider the unscaled form of ADMM for this

problem:
 
xk+1
i := argmin f (x
i i ) + y kT
xi + (ρ/2)xi − (xk
i − xk 2
) 2
xi
k+1
y := y + ρxk+1 .
k

The variable y k converges to an optimal dual variable, which is readily

interpreted as a set of optimal or clearing prices for the exchange. The
proximal term in the x-update is a penalty for xk+1 deviating from xk ,
projected onto the feasible set. The x-update in exchange ADMM can
be carried out independently in parallel, for i = 1, . . . , N . The u-update
requires gathering the xk+1
i (or otherwise averaging), and broadcasting
x k+1 +u k+1 back to the processors handling the xi updates.
Exchange ADMM can be viewed as a form of tâtonnement or price
adjustment process [168, 163] from Walras’ theory of general equilib-
rium. Tâtonnement represents the mechanism of the competitive mar-
ket working towards market equilibrium; the idea is that the market
60 Consensus and Sharing

acts via price adjustment, i.e., increasing or decreasing the price of each
good depending on whether there is an excess demand or excess supply
of the good, respectively.
Dual decomposition is the simplest algorithmic expression of
tâtonnement. In this setting, each agent adjusts his consumption xi
to minimize his individual cost fi (xi ) adjusted by the cost y T xi , where
y is the price vector. The central collector (called the ‘secretary of mar-
ket’ in [163]) works toward equilibrium by adjusting the prices y up or
down depending on whether each commodity or good is overproduced
or underproduced. ADMM differs only in the inclusion of the proximal
regularization term in the updates for each agent. As y k converges to
an optimal price vector y
, the effect of the proximal regularization
term vanishes. The proximal regularization term can be interpreted as
each agent’s commitment to help clear the market.
 8
Distributed Model Fitting

A general convex model fitting problem can be written in the form

minimize l(Ax − b) + r(x), (8.1)

with parameters x ∈ Rn , where A ∈ Rm×n is the feature matrix, b ∈

Rm is the output vector, l : Rm → R is a convex loss function, and r
is a convex regularization function. We assume that l is additive, so


m
l(Ax − b) = li (aTi x − bi ),
i=1

where li : R → R is the loss for the ith training example, ai ∈ Rn is

the feature vector for example i, and bi is the output or response for
example i. Each li can be different, though in practice they are usually
all the same.
We also assume that the regularization function r is separable. The
most common examples are r(x) = λx22 (called Tikhonov regulariza-
tion, or a ridge penalty in statistical settings) and r(x) = λx1 (some-
times generically called a lasso penalty in statistical settings), where
λ is a positive regularization parameter, though more elaborate regu-
larizers can be used just as easily. In some cases, one or more model

61
62 Distributed Model Fitting

parameters are not regularized, such as the offset parameter in a clas-

sification model. This corresponds to, for example, r(x) = λx1:n−1 1 ,
where x1:n−1 is the subvector of x consisting of all but the last compo-
nent of x; with this choice of r, the last component of x is not regular-
ized.
The next section discusses some examples that have the general
form above. We then consider two ways to solve (8.1) in a distributed
manner, namely, by splitting across training examples and by splitting
across features. While we work with the assumption that l and r are
separable at the component level, we will see that the methods we
describe work with appropriate block separability as well.

8.1 Examples
8.1.1 Regression
Consider a linear modeling problem with measurements of the form

bi = aTi x + vi ,

where ai is the ith feature vector and the measurement noises vi

are independent with log-concave densities pi ; see, e.g., [20, §7.1.1].
Then the negative log-likelihood function is l(Ax − b), with li (ω) =
− log pi (−ω). If r = 0, then the general fitting problem (8.1) can be
interpreted as maximum likelihood estimation of x under noise model
pi . If ri is taken to be the negative log prior density of xi , then the
problem can be interpreted as MAP estimation.
For example, the lasso follows the form above with quadratic loss
l(u) = (1/2)u22 and
1 regularization r(x) = λx1 , which is equiva-
lent to MAP estimation of a linear model with Gaussian noise and a
Laplacian prior on the parameters [156, §5].

8.1.2 Classification
Many classification problems can also be put in the form of the general
model fitting problem (8.1), with A, b, l, and r appropriately chosen. We
follow the standard setup from statistical learning theory, as described
in, e.g., [8]. Let pi ∈ Rn−1 denote the feature vector of the ith example
 8.1 Examples 63

and let qi ∈ {−1, 1} denote the binary outcome or class label, for i =
1, . . . , m. The goal is to find a weight vector w ∈ Rn−1 and offset v ∈ R
such that

sign(pTi w + v) = qi

holds for many examples. Viewed as a function of pi , the expression

pTi w + v is called a discriminant function. The condition that the sign
of the discriminant function and the response should agree can also be
written as µi > 0, where µi = qi (pTi w + v) is called the margin of the
ith training example.
In the context of classification, loss functions are generally written
as a function of the margin, so the loss for the ith example is

li (µi ) = li (qi (pTi w + v)).

A classification error is made if and only if the margin is negative, so

li should be positive and decreasing for negative arguments and zero
or small for positive arguments. To find the parameters w and v, we
minimize the average loss plus a regularization term on the weights:

1

m
li (qi (pTi w + v)) + rwt (w). (8.2)
m
i=1

This has the generic model fitting form (8.1), with x = (w, v), ai =
(qi pi , −qi ), bi = 0, and regularizer r(x) = rwt (w). (We also need to scale
li by 1/m.) In the sequel, we will address such problems using the form
(8.1) without comment, assuming that this transformation has been
carried out.
In statistical learning theory, the problem (8.2) is referred to as
penalized empirical risk minimization or structural risk minimization.
When the loss function is convex, this is sometimes termed convex
risk minimization. In general, fitting a classifier by minimizing a sur-
rogate loss function, i.e., a convex upper bound to 0-1 loss, is a
well studied and widely used approach in machine learning; see, e.g.,
[165, 180, 8].
Many classification models in machine learning correspond to dif-
ferent choices of loss function li and regularization or penalty rwt .
64 Distributed Model Fitting

Some common loss functions are hinge loss (1 − µi )+ , exponential

loss exp(−µi ), and logistic loss log(1 + exp(−µi )); the most com-
mon regularizers are
1 and
2 (squared). The support vector machine
(SVM) [151] corresponds to hinge loss with a quadratic penalty, while
exponential loss yields boosting [78] and logistic loss yields logistic
regression.

8.2 Splitting across Examples

Here we discuss how to solve the model fitting problem (8.1) with
a modest number of features but a very large number of training
examples. Most classical statistical estimation problems belong to this
regime, with large volumes of relatively low-dimensional data. The goal
is to solve the problem in a distributed way, with each processor han-
dling a subset of the training data. This is useful either when there are
so many training examples that it is inconvenient or impossible to pro-
cess them on a single machine or when the data is naturally collected
or stored in a distributed fashion. This includes, for example, online
social network data, webserver access logs, wireless sensor networks,
and many cloud computing applications more generally.
We partition A and b by rows,
   
A1 b1
   
A =  ...  , b =  ...  ,
AN bN

with Ai ∈ Rmi ×n and bi ∈ Rmi , where N i=1 mi = m. Thus, Ai and bi
represent the ith block of data and will be handled by the ith processor.
We first put the model fitting problem in the consensus form
N
minimize i=1 li (Ai xi − bi ) + r(z)
(8.3)
subject to xi − z = 0, i = 1, . . . , N,

with variables xi ∈ Rn and z ∈ Rn . Here, li refers (with some abuse of

notation) to the loss function for the ith block of data. The problem can
now be solved by applying the generic global variable consensus ADMM
 8.2 Splitting across Examples 65

algorithm described in §7.1, given here with scaled dual variable:

 
xk+1
i := argmin l i (A i xi − b i ) + (ρ/2)xi − z k
+ u 
k 2
i 2
xi
 
z k+1 := argmin r(z) + (N ρ/2)z − xk+1 − uk 22
z
uk+1
i := uki + xk+1
i − z k+1 .
The first step, which consists of an
2 -regularized model fitting prob-
lem, can be carried out in parallel for each data block. The second
step requires gathering variables to form the average. The minimiza-
tion in the second step can be carried out componentwise (and usually
analytically) when r is assumed to be fully separable.
The algorithm described above only requires that the loss function
l be separable across the blocks of data; the regularizer r does not need
to be separable at all. (However, when r is not separable, the z-update
may require the solution of a nontrivial optimization problem.)

8.2.1 Lasso
For the lasso, this yields the distributed algorithm
 
xk+1
i := argmin (1/2)A x
i i − b 
i 2
2
+ (ρ/2)xi − z k
+ u 
k 2
i 2
xi
k+1
z := Sλ/ρN (xk+1 + uk )
uk+1
i := uki + xk+1
i − z k+1 .
Each xi -update takes the form of a Tikhonov-regularized least squares
(i.e., ridge regression) problem, with analytical solution
xk+1
i := (ATi Ai + ρI)−1 (ATi bi + ρ(z k − uki )).
The techniques from §4.2 apply: If a direct method is used, then the fac-
torization of ATi Ai + ρI can be cached to speed up subsequent updates,
and if mi < n, then the matrix inversion lemma can be applied to let
us factor the smaller matrix Ai ATi + ρI instead.
Comparing this distributed-data lasso algorithm with the serial ver-
sion in §6.4, we see that the only difference is the collection and aver-
aging steps, which couple the computations for the data blocks.
An ADMM-based distributed lasso algorithm is described in [121],
with applications in signal processing and wireless communications.
66 Distributed Model Fitting

8.2.2 Sparse Logistic Regression

Consider solving (8.1) with logistic loss functions li and
1 regulariza-
tion. We ignore the intercept term for notational simplicity; the algo-
rithm can be easily modified to incorporate an intercept. The ADMM
algorithm is
 
xk+1
i := argmin l i (A x
i i ) + (ρ/2)xi − z k
+ u 
k 2
i 2
xi
k+1
z := Sλ/ρN (xk+1 + uk )
uk+1
i := uki + xk+1
i − z k+1 .

This is identical to the distributed lasso algorithm, except for the xi

update, which here involves an
2 regularized logistic regression prob-
lem that can be efficiently solved by algorithms like L-BFGS.

8.2.3 Support Vector Machine

Using the notation of (8.1), the algorithm is
 
xk+1
i := argmin 1 T
(A x
i i + 1)+ + (ρ/2)xi − z k
+ u 
k 2
i 2
xi
ρ
z k+1 := (xk+1 + uk )
(1/λ) + N ρ
uk+1
i := uki + xk+1
i − z k+1 .

Each xi -update essentially involves fitting a support vector machine to

the local data Ai (with an offset in the quadratic regularization term),
so this can be carried out efficiently using an existing SVM solver for
serial problems.
The use of ADMM to train support vector machines in a distributed
fashion was described in [74].

8.3 Splitting across Features

Now we consider the model fitting problem (8.1) with a modest num-
ber of examples and a large number of features. Statistical problems
of this kind frequently arise in areas like natural language processing
and bioinformatics, where there are often a large number of potential
 8.3 Splitting across Features 67

explanatory variables for any given outcome. For example, the obser-
vations may be a corpus of documents, and the features could include
all words and pairs of adjacent words (bigrams) that appear in each
document. In bioinformatics, there are usually relatively few people
in a given association study, but there can be a very large number of
potential features relating to factors like observed DNA mutations in
each individual. There are many examples in other areas as well, and
the goal is to solve such problems in a distributed fashion with each
processor handling a subset of the features. In this section, we show
how this can be done by formulating it as a sharing problem from §7.3.
We partition the parameter vector x as x = (x1 , . . . , xN ), with xi ∈

Rni , where N i=1 ni = n. Conformably partition the data matrix A as
A = [A1 · · · AN ], with Ai ∈ Rm×ni , and the regularization function as
 N
r(x) = N i=1 ri (xi ). This implies that Ax = i=1 Ai xi , i.e., Ai xi can
be thought of as a ‘partial’ prediction of b using only the features ref-
erenced in xi . The model fitting problem (8.1) becomes
  
N
minimize l i=1 Ai xi −b + N
i=1 ri (xi ).

Following the approach used for the sharing problem (7.12), we express
the problem as
  
N
minimize l i=1 zi −b + N
i=1 ri (xi )
subject to Ai xi − zi = 0, i = 1, . . . , N,

with new variables zi ∈ Rm . The derivation and simplification of

ADMM also follows that for the sharing problem. The scaled form
of ADMM is
 
xk+1
i := argmin ri (xi ) + (ρ/2)Ai xi − zik + uki 22
xi
 

N

N
z k+1
:= argmin l( zi − b) + (ρ/2)Ai xk+1
i − zik + uki 22
z
i=1 i=1
uk+1
i := uki + Ai xk+1
i − zik+1 .
68 Distributed Model Fitting

As in the discussion for the sharing problem, we carry out the z-update
by first solving for the average z k+1 :

 
k+1
z k+1 := argmin l(N z − b) + (N ρ/2)z − Ax − uk 22
z
k+1
zik+1 := z k+1 + Ai xk+1
i + uki − Ax − uk ,

k+1 
where Ax = (1/N ) N k+1
i=1 Ai xi . Substituting the last expression
into the update for ui , we find that

k+1
uk+1
i = Ax + uk − z k+1 ,

which shows that, as in the sharing problem, all the dual variables are
equal. Using a single dual variable uk ∈ Rm , and eliminating zi , we
arrive at the algorithm

 
k
xk+1
i := argmin r i (xi ) + (ρ/2)A i xi − A i xk
i − z k
+ Ax + u 
k 2
2
xi
 
k+1
z k+1 := argmin l(N z − b) + (N ρ/2)z − Ax − uk 22
z
k+1
u k+1 k
:= u + Ax − z k+1 .

The first step involves solving N parallel regularized least squares

problems in ni variables each. Between the first and second steps, we
k+1
collect and sum the partial predictors Ai xk+1
i to form Ax . The sec-
ond step is a single minimization in m variables, a quadratically regu-
larized loss minimization problem; the third step is a simple update in
m variables.
This algorithm does not require l to be separable in the training
examples, as assumed earlier. If l is separable, then the z-update fully
splits into m separate scalar optimization problems. Similarly, the regu-
larizer r only needs to be separable at the level of the blocks of features.
For example, if r is a sum-of-norms, as in §6.4.2, then it would be nat-
ural to have each subsystem handle a separate group.
 8.3 Splitting across Features 69

8.3.1 Lasso
In this case, the algorithm above becomes
 
k
xk+1
i := argmin (ρ/2)A x
i i − A xk
i i − z k
+ Ax + u 
k 2
2 + λx 
i 1
xi
1  k+1

z k+1 := b + ρAx + ρuk
N +ρ
k+1
uk+1 := uk + Ax − z k+1 .
Each xi -update is a lasso problem with ni variables, which can be solved
using any single processor lasso method.
In the xi -updates, we have xk+1
i = 0 (meaning that none of the
features in the ith block are used) if and only if
 
 T k k 
Ai (Ai xi + z − Ax − u ) ≤ λ/ρ.
k k
2
When this occurs, the xi -update is fast (compared to the case when
xk+1
i = 0). In a parallel implementation, there is no benefit to speeding
up only some of the tasks being executed in parallel, but in a serial
setting we do benefit.

8.3.2 Group Lasso

Consider the group lasso problem with the feature groups coinciding
with the blocks of features, and
2 norm (not squared) regularization:

minimize (1/2)Ax − b22 + λ N i=1 xi 2 .

The z-update and u-update are the same as for the lasso, but the xi
update becomes
 
k
xk+1
i := argmin (ρ/2)A i xi − A i xk
i − z k
+ Ax + u 
k 2
2 + λxi  2 .
xi

(Only the subscript on the last norm differs from the lasso case.) This
involves minimizing a function of the form
(ρ/2)Ai xi − v22 + λxi 2 ,
which can be carried out as follows. The solution is xi = 0 if and only
if ATi v2 ≤ λ/ρ. Otherwise, the solution has the form
xi = (ATi Ai + νI)−1 ATi v,
70 Distributed Model Fitting

for the value of ν > 0 that gives νxi 2 = λ/ρ. This value can be found
using a one-parameter search (e.g., via bisection) over ν. We can speed
up the computation of xi for several values of ν (as needed for the
parameter search) by computing and caching an eigendecomposition of
ATi Ai . Assuming Ai is tall, i.e., m ≥ ni (a similar method works when
m < ni ), we compute an orthogonal Q for which ATi Ai = Q diag(λ)QT ,
where λ is the vector of eigenvalues of ATi Ai (i.e., the squares of the
singular values of Ai ). The cost is O(mn2i ) flops, dominated (in order)
by forming ATi Ai . We subsequently compute xi 2 using
 
xi 2 = diag(λ + ν1)−1 QT ATi v 2 .
This can be computed in O(ni ) flops, once QT ATi v is computed, so
the search over ν is costless (in order). The cost per iteration is thus
O(mni ) (to compute QT ATi v), a factor of ni better than carrying out
the xi -update without caching.

8.3.3 Sparse Logistic Regression

The algorithm is identical to the lasso problem above, except that the
z-update becomes
 
k+1
z k+1 := argmin l(N z) + (ρ/2)z − Ax − uk 22 ,
z
where l is the logistic loss function. This splits to the component level,
and involves the proximity operator for l. This can be very efficiently
computed by a lookup table that gives the approximate value, followed
by one or two Newton steps (for a scalar problem).
It is interesting to see that in distributed sparse logistic regression,
the dominant computation is the solution of N parallel lasso problems.

8.3.4 Support Vector Machine

The algorithm is
 
k
xk+1
i := argmin (ρ/2)A i xi − A i xk
i − z k
+ Ax + u 
k 2
2 + λxi  2
2
xi
 
k+1
z k+1 := argmin 1T (N z + 1)+ + (ρ/2)z − Ax − uk 22
z
k+1
u k+1 k
:= u + Ax − z k+1 .
 8.3 Splitting across Features 71

The xi -updates involve quadratic functions, and require solving ridge

regression problems. The z-update splits to the component level, and
can be expressed as the shifted soft thresholding operation

 vi − N/ρ vi > −1/N + N/ρ
k+1
zi := −1/N vi ∈ [−1/N, −1/N + N/ρ]

vi vi < −1/N,
k+1
where v = Ax + uk (and here, the subscript denotes the entry in
the vector z k+1 ).

8.3.5 Generalized Additive Models

A generalized additive model has the form


n
b≈ fj (aj ),
j=1

where aj is the jth element of the feature vector a, and fj : R → R are

the feature functions. When the feature functions fj are linear, i.e., of
the form fj (aj ) = wj aj , this reduces to standard linear regression.
We choose the feature functions by solving the optimization problem
m n n
minimize i=1 li ( j=1 fj (aij ) − bi ) + j=1 rj (fj ),

where aij is the jth component of the feature vector of the ith example,
and bi is the associated outcome. Here the optimization variables are
the functions fj ∈ Fj , where Fj is a subspace of functions; rj is now
a regularization functional. Usually fj is linearly parametrized by a
finite number of coefficients, which are the underlying optimization
variables, but this formulation can also handle the case when Fj is
infinite-dimensional. In either case, it is clearer to think of the feature
functions fj as the variables to be determined.
72 Distributed Model Fitting

We split the features down to individual functions so N = n. The

algorithm is

fjk+1 :=
 m 
k
argmin rj (fj ) + (ρ/2) i=1 (fj (aij ) − fj (aij ) − z i + f i + ui )
k k k 2
fj ∈Fj
 N 
m k+1
z k+1 := argmin i=1 li (N z i − bi ) + (ρ/2) i=1 z − f − uk 22
z
k+1
u k+1 k
:= u + f − z k+1 ,
k 
where f i = (1/n) nj=1 fjk (aij ), the average value of the predicted

response nj=1 fjk (aij ) for the ith feature.
The fj -update is an
2 (squared) regularized function fit. The z-
update can be carried out componentwise.
 9
Nonconvex Problems

We now explore the use of ADMM for nonconvex problems, focusing

on cases in which the individual steps in ADMM, i.e., the x- and z-
updates, can be carried out exactly. Even in this case, ADMM need not
converge, and when it does converge, it need not converge to an optimal
point; it must be considered just another local optimization method.
The hope is that it will possibly have better convergence properties
than other local optimization methods, where ‘better convergence’ can
mean faster convergence or convergence to a point with better objective
value. For nonconvex problems, ADMM can converge to different (and
in particular, nonoptimal) points, depending on the initial values x0
and y 0 and the parameter ρ.

9.1 Nonconvex Constraints

Consider the constrained optimization problem

minimize f (x)
subject to x ∈ S,

73
74 Nonconvex Problems

with f convex, but S nonconvex. Here, ADMM has the form

 
xk+1 := argmin f (x) + (ρ/2)x − z k + uk 22
x
k+1
z := ΠS (xk+1 + uk )
uk+1 := uk + xk+1 − z k+1 ,

where ΠS is projection onto S. The x-minimization step (which is evalu-

ating a proximal operator) is convex since f is convex, but the z-update
is projection onto a nonconvex set. In general, this is hard to compute,
but it can be carried out exactly in some important special cases we
list below.

• Cardinality. If S = {x | card(x) ≤ c}, where card gives the

number of nonzero elements, then ΠS (v) keeps the c largest
magnitude elements and zeroes out the rest.
• Rank. If S is the set of matrices with rank c, then ΠS (v)
is determined by carrying out a singular value decomposi-

tion, v = i σi ui viT , and keeping the top c dyads, i.e., form

ΠS (v) = ci=1 σi ui viT .
• Boolean constraints. If S = {x | xi ∈ {0, 1}}, then ΠS (v) sim-
ply rounds each entry to 0 or 1, whichever is closer. Integer
constraints can be handled in the same way.

9.1.1 Regressor Selection

As an example, consider the least squares regressor selection or feature
selection problem,

minimize Ax − b22

subject to card(x) ≤ c,

which is to find the best fit to b as a linear combination of no more than

c columns of A. For this problem, ADMM takes the form above, where
the x-update involves a regularized least squares problem, and the z-
update involves keeping the c largest magnitude elements of xk+1 + uk .
This is just like ADMM for the lasso, except that soft thresholding is
 9.1 Nonconvex Constraints 75

replaced with ‘hard’ thresholding. This close connection is hardly sur-

prising, since lasso can be thought of as a heuristic for solving the
regressor selection problem. From this viewpoint, the lasso controls the
trade-off between least squares error and sparsity through the param-
eter λ, whereas in ADMM for regressor selection, the same trade-off is
controlled by the parameter c, the exact cardinality desired.

9.1.2 Factor Model Fitting

The goal is to approximate a symmetric matrix Σ (say, an empiri-
cal covariance matrix) as a sum of a rank-k and a diagonal positive
semidefinite matrix. Using the Frobenius norm to measure approxima-
tion error, we have the problem
minimize (1/2)X + diag(d) − Σ2F
subject to X ≥ 0, Rank(X) = k, d ≥ 0,
with variables X ∈ Sn , d ∈ Rn . (Any convex loss function could be used
in lieu of the Frobenius norm.)
We take
f (X) = inf (1/2)X + diag(d) − Σ2F
d≥0



n
= (1/2) (Xij − Σij ) + (1/2)
2
(Xii − Σii )2+ ,
i=j i=1

with the optimizing d having the form di = (Σii − Xii )+ , i = 1, . . . , n.

We take S to be the set of positive semidefinite rank-k matrices.
ADMM for the factor model fitting problem is then
 
X k+1 := argmin f (X) + (ρ/2)X − Z k + U k 2F
X
Z k+1 := ΠS (X k+1 + U k )
U k+1 := U k + X k+1 − Z k+1 ,
where Z, U ∈ Sn . The X-update is separable to the component level,
and can be expressed as
 
(X k+1 )ij := (1/(1 + ρ)) Σij + ρ(Zij k − Uk ) i = j
ij
 
(X k+1 )ii := (1/(1 + ρ)) Σii + ρ(Ziik − Uiik ) Σii ≤ Zii − Uii
(X k+1 )ii := Ziik − Uiik Σii > Zii − Uii .
76 Nonconvex Problems

The Z-update is carried out by an eigenvalue decomposition, keeping

only the dyads associated with the largest k positive eigenvalues.

9.2 Bi-convex Problems

Another problem that admits exact ADMM updates is the general bi-
convex problem,
minimize F (x, z)
subject to G(x, z) = 0,

where F : Rn × Rm → R is bi-convex, i.e., convex in x for each z and

convex in z for each x, and G : Rn × Rm → Rp is bi-affine, i.e., affine
in x for each fixed z, and affine in z for each fixed x. When F is
separable in x and z, and G is jointly affine in x and z, this reduces
to the standard ADMM problem form (3.1). For this problem ADMM
has the form
 
xk+1 := argmin F (x, z k ) + (ρ/2)G(x, z k ) + uk 22
x
 
z k+1
:= argmin F (xk+1 , z) + (ρ/2)G(xk+1 , z) + uk 22
z
uk+1 := uk + G(xk+1 , z k+1 ).

Both the x- and z-updates involve convex optimization problems, and

so are tractable.
When G = 0 (or is simply absent), ADMM reduces to simple alter-
nating minimization, a standard method for bi-convex minimization.

9.2.1 Nonnegative Matrix Factorization

As an example, consider nonnegative matrix factorization [110]:

minimize (1/2)V W − C2F

subject to Vij ≥ 0, Wij ≥ 0,

with variables V ∈ Rp×r and W ∈ Rr×q , and data C ∈ Rp×q . In this

form of the problem, the objective (which includes the constraints) is
bi-affine, and there are no equality constraints, so ADMM becomes
the standard method for nonnegative matrix factorization, which is
 9.2 Bi-convex Problems 77

alternately minimizing over V , with W fixed, and then minimizing over

W , with V fixed.
We can also introduce a new variable, moving the bi-linear term
from the objective into the constraints:

minimize (1/2)X − C2F + I+ (V ) + I+ (W )

subject to X − V W = 0,

with variables X, V, W , where I+ is the indicator function for elemen-

twise nonnegative matrices. With (X, V ) serving the role of x, and W
serving the role of z above, ADMM becomes
 
(X k+1 , V k+1 ) := argmin X − C2F + (ρ/2)X − V W k + U k 2F
X,V ≥0

W k+1
:= argmin X k+1 − V k+1 W + U k 2F
W ≥0

U k+1
:= U + X k+1 − V k+1 W k+1 .
k

The first step splits across the rows of X and V , so can be performed
by solving a set of quadratic programs, in parallel, to find each row of
X and V separately; the second splits in the columns of W , so can be
performed by solving parallel quadratic programs to find each column.
 10
Implementation

This section addresses the implementation of ADMM in a distributed

computing environment. For simplicity, we focus on the global consen-
sus problem with regularization,
N
minimize i=1 fi (xi ) + g(z)
subject to xi − z = 0,
where fi is the ith objective function term and g is the global regular-
izer. Extensions to the more general consensus case are mostly straight-
forward. We first describe an abstract implementation and then show
how this maps onto a variety of software frameworks.

10.1 Abstract Implementation

We refer to xi and ui as local variables stored in subsystem i, and to z
as the global variable. For a distributed implementation, it is often more
natural to group the local computations (i.e., the xi - and ui -updates),
so we write ADMM as
ui := ui + xi − z
 
xi := argmin fi (xi ) + (ρ/2)xi − z + ui 22
z := proxg,N ρ (x + u) .

78
 10.1 Abstract Implementation 79

Here, the iteration indices are omitted because in an actual implemen-

tation, we can simply overwrite previous values of these variables. Note
that the u-update must be done before the x-update in order to match
(7.6-7.8). If g = 0, then the z-update simply involves computing x, and
the ui are not part of the aggregation, as discussed in §7.1.
This suggests that the main features required to implement ADMM
are the following:
• Mutable state. Each subsystem i must store the current values
of xi and ui .
• Local computation. Each subsystem must be able to solve a
small convex problem, where ‘small’ means that the problem
is solvable using a serial algorithm. In addition, each local
process must have local access to whatever data are required
to specify fi .
• Global aggregation. There must be a mechanism for averag-
ing local variables and broadcasting the result back to each
subsystem, either by explicitly using a central collector or via
some other approach like distributed averaging [160, 172]. If
computing z involves a proximal step (i.e., if g is nonzero),
this can either be performed centrally or at each local node;
the latter is easier to implement in some frameworks.
• Synchronization. All the local variables must be updated
before performing global aggregation, and the local updates
must all use the latest global variable. One way to implement
this synchronization is via a barrier, a system checkpoint at
which all subsystems must stop and wait until all other sub-
systems reach it.
When actually implementing ADMM, it helps to consider whether to
take the ‘local perspective’ of a subsystem performing local processing
and communicating with a central collector, or the ‘global perspective’
of a central collector coordinating the work of a set of subsystems.
Which is more natural depends on the software framework used.
From the local perspective, each node i receives z, updates ui and
then xi , sends them to the central collector, waits, and then receives the
80 Implementation

updated z. From the global perspective, the central collector broadcasts

z to the subsystems, waits for them to finish local processing, gathers
all the xi and ui , and updates z. (Of course, if ρ varies across itera-
tions, then ρ must also be updated and broadcast when z is updated.)
The nodes must also evaluate the stopping criteria and decide when to
terminate; see below for examples.
In the general form consensus case, which we do not discuss here,
a decentralized implementation is possible that does not require z to
be centrally stored; each set of subsystems that share a variable can
communicate among themselves directly. In this setting, it can be con-
venient to think of ADMM as a message-passing algorithm on a graph,
where each node corresponds to a subsystem and the edges correspond
to shared variables.

10.2 MPI
Message Passing Interface (MPI) [77] is a language-independent
message-passing specification used for parallel algorithms, and is the
most widely used model for high-performance parallel computing today.
There are numerous implementations of MPI on a variety of distributed
platforms, and interfaces to MPI are available from a wide variety of
languages, including C, C++, and Python.
There are multiple ways to implement consensus ADMM in MPI,
but perhaps the simplest is given in Algorithm 1. This pseudocode
uses a single program, multiple data (SPMD) programming style, in
which each processor or subsystem runs the same program code but
has its own set of local variables and can read in a separate subset
of the data. We assume there are N processors, with each processor
i storing local variables xi and ui , a (redundant) copy of the global
variable z, and handling only the local data implicit in the objective
component fi .
In step 4, Allreduce denotes using the MPI Allreduce operation to
compute the global sum over all processors of the contents of the vector
w, and store the result in w on every processor; the same applies to

the scalar t. After step 4, then, w = ni=1 (xi + ui ) = N (x + u) and

t = r22 = ni=1 ri 22 on all processors. We use Allreduce because
 10.3 Graph Computing Frameworks 81

Algorithm 1 Global consensus ADMM in MPI.

initialize N processes, along with xi , ui , ri , z.

repeat
1. Update ui := ui + xi −
z. 
2. Update xi := argminx fi (x) + (ρ/2)
x − z + ui
22 .
3. Let w := xi + ui and t :=
ri
22 .
4. Allreduce w and t.
5. Let z prev√
:= z and update z := proxg,N√ρ (w/N ).
6. exit if ρ N
z − z prev
2 ≤ conv and t ≤ feas .
7. Update ri := xi − z.

its implementation is in general much more scalable than simply

having each subsystem send its results directly to an explicit central
collector.
Next, in steps 5 and 6, all processors (redundantly) compute the
z-update and perform the termination test. It is possible to have the
z-update and termination test performed on just one processor and
broadcast the results to the other processors, but doing so complicates
the code and is generally no faster.

10.3 Graph Computing Frameworks

Since ADMM can be interpreted as performing message-passing on a
graph, it is natural to implement it in a graph processing framework.
Conceptually, the implementation will be similar to the MPI case dis-
cussed above, except that the role of the central collector will often be
handled abstractly by the system, rather than having an explicit central
collector process. In addition, higher-level graph processing frameworks
provide a number of built in services that one would otherwise have to
manually implement, such as fault tolerance.
Many modern graph frameworks are based on or inspired by
Valiant’s bulk-synchronous parallel (BSP) model [164] for parallel
computation. A BSP computer consists of a set of processors net-
worked together, and a BSP computation consists of a series of
global supersteps. Each superstep consists of three stages: parallel
82 Implementation

computation, in which the processors, in parallel, perform local

computations; communication, in which the processors communicate
among themselves; and barrier synchronization, in which the processes
wait until all processes are finished communicating.
The first step in each ADMM superstep consists of performing local
ui - and xi -updates. The communication step would broadcast the new
xi and ui values to a central collector node, or globally to each indi-
vidual processor. Barrier synchronization is then used to ensure that
all the processors have updated their primal variable before the central
collector averages and rebroadcasts the results.
Specific frameworks directly based on or inspired by the BSP model
include the Parallel BGL [91], GraphLab [114], and Pregel [119], among
others. Since all three follow the general outline above, we refer the
reader to the individual papers for details.

10.4 MapReduce
MapReduce [46] is a popular programming model for distributed batch
processing of very large datasets. It has been widely used in indus-
try and academia, and its adoption has been bolstered by the open
source project Hadoop, inexpensive cloud computing services avail-
able through Amazon, and enterprise products and services offered
by Cloudera. MapReduce libraries are available in many languages,
including Java, C++, and Python, among many others, though Java
is the primary language for Hadoop. Though it is awkward to express
ADMM in MapReduce, the amount of cloud infrastructure available
for MapReduce computing can make it convenient to use in practice,
especially for large problems. We briefly review some key features of
Hadoop below; see [170] for general background.
A MapReduce computation consists of a set of Map tasks, which
process subsets of the input data in parallel, followed by a Reduce task,
which combines the results of the Map tasks. Both the Map and Reduce
functions are specified by the user and operate on key-value pairs. The
Map function performs the transformation

(k, v) → [(k1 , v1 ), . . . , (km

 
, vm )],
 10.4 MapReduce 83

that is, it takes a key-value pair and emits a list of intermediate

key-value pairs. The engine then collects all the values v1 , . . . , vr that
correspond to the same output key k  (across all Mappers) and passes
them to the Reduce functions, which performs the transformation

(k  , [v1 , . . . , vr ]) → (k  , R(v1 , . . . , vr )),

where R is a commutative and associative function. For example, R

could simply sum vi . In Hadoop, Reducers can emit lists of key-value
pairs rather than just a single pair.
Each iteration of ADMM can easily be represented as a MapRe-
duce task: The parallel local computations are performed by Maps,
and the global aggregation is performed by a Reduce. We will describe
a simple global consensus implementation to give the general flavor
and discuss the details below. Here, we have the Reducer compute

Algorithm 2 An iteration of global consensus ADMM in Hadoop/ MapReduce.

function map(key i, dataset Di )

1. Read (xi , ui , ẑ) from HBase table.
2. Compute z := proxg,N ρ ((1/N )ẑ).
3. Update ui := ui + xi −
z. 
4. Update xi := argminx fi (x) + (ρ/2)
x − z + ui
22 .
5. Emit (key central, record (xi , ui )).
function reduce(key central, records (x1 , u1 ), . . . , (xN , uN ))

1. Update ẑ := N i=1 xi + ui .
2. Emit (key j, record (xj , uj , ẑ)) to HBase for j = 1, . . . , N .


ẑ = N i=1 (xi + ui ) rather than z or z̃ because summation is associa-
tive while averaging is not. We assume N is known (or, alternatively,

the Reducer can compute the sum N i=1 1). We have N Mappers, one
for each subsystem, and each Mapper updates ui and xi using the
ẑ from the previous iteration. Each Mapper independently executes
the proximal step to compute z, but this is usually a cheap opera-
tion like soft thresholding. It emits an intermediate key-value pair that
essentially serves as a message to the central collector. There is a sin-
gle Reducer, playing the role of a central collector, and its incoming
values are the messages from the Mappers. The updated records are
84 Implementation

then written out directly to HBase by the Reducer, and a wrapper

program restarts a new MapReduce iteration if√the algorithm has not
converged. The wrapper will check whether ρ N z − z prev 2 ≤ conv

and ( N i=1 xi − z2 )
2 1/2 ≤ feas to determine convergence, as in the

MPI case. (The wrapper checks the termination criteria instead of the
Reducer because they are not associative to check.)
The main difficulty is that MapReduce tasks are not designed to
be iterative and do not preserve state in the Mappers across iter-
ations, so implementing an iterative algorithm like ADMM requires
some understanding of the underlying infrastructure. Hadoop con-
tains a number of components supporting large-scale, fault-tolerant
distributed computing applications. The relevant components here are
HDFS, a distributed file system based on Google’s GFS [85], and
HBase, a distributed database based on Google’s BigTable [32].
HDFS is a distributed filesystem, meaning that it manages the
storage of data across an entire cluster of machines. It is designed for
situations where a typical file may be gigabytes or terabytes in size and
high-speed streaming read access is required. The base units of storage
in HDFS are blocks, which are 64 MB to 128 MB in size in a typi-
cal configuration. Files stored on HDFS are comprised of blocks; each
block is stored on a particular machine (though for redundancy, there
are replicas of each block on multiple machines), but different blocks in
the same file need not be stored on the same machine or even nearby.
For this reason, any task that processes data stored on HDFS (e.g., the
local datasets Di ) should process a single block of data at a time, since
a block is guaranteed to reside wholly on one machine; otherwise, one
may cause unnecessary network transfer of data.
In general, the input to each Map task is data stored on HDFS, and
Mappers cannot access local disk directly or perform any stateful com-
putation. The scheduler runs each Mapper as close to its input data
as possible, ideally on the same node, in order to minimize network
transfer of data. To help preserve data locality, each Map task should
also be assigned around a block’s worth of data. Note that this is very
different from the implementation presented for MPI, where each pro-
cess can be told to pick up the local data on whatever machine it is
running on.
 10.4 MapReduce 85

Since each Mapper only handles a single block of data, there will
usually be a number of Mappers running on the same machine. To
reduce the amount of data transferred over the network, Hadoop sup-
ports the use of combiners, which essentially Reduce the results of all
the Map tasks on a given node so only one set of intermediate key-
value pairs need to be transferred across machines for the final Reduce
task. In other words, the Reduce step should be viewed as a two-step
process: First, the results of all the Mappers on each individual node
are reduced with Combiners, and then the records across each machine
are Reduced. This is a major reason why the Reduce function must be
commutative and associative.
Since the input value to a Mapper is a block of data, we also need a
mechanism for a Mapper to read in local variables, and for the Reducer
to store the updated variables for the next iteration. Here, we use
HBase, a distributed database built on top of HDFS that provides
fast random read-write access. HBase, like BigTable, provides a dis-
tributed multi-dimensional sorted map. The map is indexed by a row
key, a column key, and a timestamp. Each cell in an HBase table can
contain multiple versions of the same data indexed by timestamp; in
our case, we can use the iteration counts as the timestamps to store
and access data from previous iterations; this is useful for checking ter-
mination criteria, for example. The row keys in a table are strings, and
HBase maintains data in lexicographic order by row key. This means
that rows with lexicographically adjacent keys will be stored on the
same machine or nearby. In our case, variables should be stored with
the subsystem identifier at the beginning of row key, so information for
the same subsystem is stored together and is efficient to access. For
more details, see [32, 170].
The discussion and pseudocode above omits and glosses over many
details for simplicity of exposition. MapReduce frameworks like Hadoop
also support much more sophisticated implementations, which may
be necessary for very large scale problems. For example, if there
are too many values for a single Reducer to handle, we can use an
approach analogous to the one suggested for MPI: Mappers emit pairs
to ‘regional’ reduce jobs, and then an additional MapReduce step is car-
ried out that uses an identity mapper and aggregates regional results
86 Implementation

into a global result. In this section, our goal is merely to give a gen-
eral flavor of some of the issues involved in implementing ADMM in
a MapReduce framework, and we refer to [46, 170, 111] for further
details. There has also been some recent work on alternative MapRe-
duce systems that are specifically designed for iterative computation,
which are likely better suited for ADMM [25, 179], though the imple-
mentations are less mature and less widely available. See [37, 93] for
examples of recent papers discussing machine learning and optimiza-
tion in MapReduce frameworks.
 11
Numerical Examples

In this section we report numerical results for several examples. The

examples are chosen to illustrate a variety of the ideas discussed above,
including caching matrix factorizations, using iterative solvers for the
updates, and using consensus and sharing ADMM to solve distributed
problems. The implementations of ADMM are written to be as simple
as possible, with no implementation-level optimization or tuning.
The first section discusses a small instance of the lasso problem
with a dense coefficient matrix. This helps illustrate some of the basic
behavior of the algorithm, and the impact of some of the linear algebra-
based optimizations suggested in §4. We find, for example, that we can
compute the entire regularization path for the lasso in not much more
time than it takes to solve a single problem instance, which in turn takes
not much more time than solving a single ridge regression problem of
the same size.
We then discuss a serial implementation of the consensus ADMM
algorithm applied to
1 regularized logistic regression, where we split
the problem across training examples. Here, we focus on details of
implementing consensus ADMM for this problem, rather than on actual

87
88 Numerical Examples

distributed solutions. The following section has a similar discussion of

the group lasso problem, but split across features, with each regular-
ization group corresponding to a distinct subsystem.
We then turn to a real large-scale distributed implementation using
an MPI-based solver written in C. We report on the results of solving
some large lasso problems on clusters hosted in Amazon EC2, and find
that a fairly basic implementation is able to solve a lasso problem with
30 GB of data in a few minutes.
Our last example is regressor selection, a nonconvex problem.
We compare the sparsity-fit trade-off curve obtained using noncon-
vex ADMM, directly controlling the number of regressors, with the
same curve obtained using the lasso regularization path (with poste-
rior least squares fit). We will see that the curves are not the same,
but give very similar results. This suggests that the regressor selection
method may be preferable to the lasso when the desired sparsity level
is known in advance: It is much easier to explicitly set the desired spar-
sity level than tuning the regularization parameter λ to obtain this
level.
All examples except the large-scale lasso are implemented in Mat-
lab, and run on an Intel Core i3 processor running at 3.2 GHz.
The large lasso example is implemented in C using MPI for inter-
process communication and the GNU Scientific Library for linear alge-
bra. Source code and data for these examples (and others) can be
found at www.stanford.edu/˜boyd/papers/admm_distr_stats.html
and most are extensively commented.

11.1 Small Dense Lasso

We consider a small, dense instance of the lasso problem (6.2), where
the feature matrix A has m = 1500 examples and n = 5000 features.
We generate the data as follows. We first choose Aij ∼ N (0, 1)
and then normalize the columns to have unit
2 norm. A ‘true’ value
xtrue ∈ Rn is generated with 100 nonzero entries, each sampled from an
N (0, 1) distribution. The labels b are then computed as b = Axtrue + v,
where v ∼ N (0, 10−3 I), which corresponds to a signal-to-noise ratio
Axtrue 22 /v22 of around 60.
 11.1 Small Dense Lasso 89

Fig. 11.1. Norms of primal residual (top) and dual residual (bottom) versus iteration, for
a lasso problem. The dashed lines show pri (top) and dual (bottom).

We set the penalty parameter ρ = 1 and set termination tolerances

abs = 10−4 and rel = 10−2 . The variables u0 and z 0 were initialized to
be zero.

11.1.1 Single Problem

We first solve the lasso problem with regularization parameter λ =
0.1λmax , where λmax = AT b∞ is the critical value of λ above which
the solution of the lasso problem is x = 0. (Although not relevant, this
choice correctly identifies about 80% of the nonzero entries in xtrue .)
Figure 11.1 shows the primal and dual residual norms by iteration,
as well as the associated stopping criterion limits pri and dual (which
vary slightly in each iteration since they depend on xk , z k , and y k
through the relative tolerance terms). The stopping criterion was sat-
isfied after 15 iterations, but we ran ADMM for 35 iterations to show
the continued progress. Figure 11.2 shows the objective suboptimality
p̃k − p
, where p̃k = (1/2)Az k − b22 + λz k 1 is the objective value
at z k . The optimal objective value p
= 17.4547 was independently ver-
ified using l1 ls [102].
90 Numerical Examples

Fig. 11.2. Objective suboptimality versus iteration for a lasso problem. The stopping crite-
rion is satisfied at iteration 15, indicated by the vertical dashed line.

Since A is fat (i.e., m < n), we apply the matrix inversion lemma
to (AT A + ρI)−1 and instead compute the factorization of the smaller
matrix I + (1/ρ)AAT , which is then cached for subsequent x-updates.
The factor step itself takes about nm2 + (1/3)m3 flops, which is the
cost of forming AAT and computing the Cholesky factorization. Subse-
quent updates require two matrix-vector multiplications and forward-
backward solves, which require approximately 4mn + 2m2 flops. (The
cost of the soft thresholding step in the z-update is negligible.) For these
problem dimensions, the flop count analysis suggests a factor/solve
ratio of around 350, which means that 350 subsequent ADMM itera-
tions can be carried out for the cost of the initial factorization.
In our basic implementation, the factorization step takes about 1
second, and subsequent x-updates take around 30 ms. (This gives a fac-
tor/solve ratio of only 33, less than predicted, due to a particularly effi-
cient matrix-matrix multiplication routine used in Matlab.) Thus the
total cost of solving an entire lasso problem is around 1.5 seconds—only
50% more than the initial factorization. In terms of parameter estima-
tion, we can say that computing the lasso estimate requires only 50%
 11.1 Small Dense Lasso 91

Fig. 11.3. Iterations needed versus λ for warm start (solid line) and cold start (dashed line).

more time than a ridge regression estimate. (Moreover, in an imple-

mentation with a higher factor/solve ratio, the additional effort for the
lasso would have been even smaller.)
Finally, we report the effect of varying the parameter ρ on conver-
gence time. Varying ρ over the 100:1 range from 0.1 to 10 yields a solve
time ranging between 1.45 seconds to around 4 seconds. (In an imple-
mentation with a larger factor/solve ratio, the effect of varying ρ would
have been even smaller.) Over-relaxation with α = 1.5 does not signif-
icantly change the convergence time with ρ = 1, but it does reduce the
worst convergence time over the range ρ ∈ [0.1, 10] to only 2.8 seconds.

11.1.2 Regularization Path

To illustrate computing the regularization path, we solve the lasso prob-
lem for 100 values of λ, spaced logarithmically from 0.01λmax (where
x
has around 800 nonzeros) to 0.95λmax (where x
has two nonzero
entries). We first solve the lasso problem as above for λ = 0.01λmax ,
and for each subsequent value of λ, we then initialize (warm start) z
and u at their optimal values for the previous λ. This requires only one
factorization for all the computations; warm starting ADMM at the
92 Numerical Examples

Task Time (s)

Factorization 1.1
x-Update 0.03
Single lasso (λ = 0.1λmax ) 1.5
Cold start regularization path (100 values of λ) 160
Warm start regularization path (100 values of λ) 13

Table 11.1. Summary of timings for lasso example.

previous value significantly reduces the number of ADMM iterations

required to solve each lasso problem after the first one.
Figure 11.3 shows the number of iterations required to solve each
lasso problem using this warm start initialization, compared to the
number of iterations required using a cold start of z 0 = u0 = 0 for
each λ. For the 100 values of λ, the total number of ADMM itera-
tions required is 428, which takes 13 seconds in all. By contrast, with
cold starts, we need 2166 total ADMM iterations and 100 factorizations
to compute the regularization path, or around 160 seconds total. This
timing information is summarized in Table 11.1.

11.2 Distributed
1 Regularized Logistic Regression

In this example, we use consensus ADMM to fit an
1 regularized logis-
tic regression model. Following §8, the problem is


m
 
minimize log 1 + exp(−bi (aTi w + v)) + λw1 , (11.1)
i=1

with optimization variables w ∈ Rn and v ∈ R. The training set con-

sists of m pairs (ai , bi ), where ai ∈ Rn is a feature vector and bi ∈
{−1, 1} is the corresponding label.
We generated a problem instance with m = 106 training examples
and n = 104 features. The m examples are distributed among N = 100
subsystems, so each subsystem has 104 training examples. Each feature
vector ai was generated to have approximately 10 nonzero features,
each sampled independently from a standard normal distribution. We
chose a ‘true’ weight vector wtrue ∈ Rn to have 100 nonzero values,
and these entries, along with the true intercept v true , were sampled
 11.2 Distributed
1 Regularized Logistic Regression 93

independently from a standard normal distribution. The labels bi were

then generated using
bi = sign(aTi wtrue + v true + vi ),
where vi ∼ N (0, 0.1).
The regularization parameter is set to λ = 0.1λmax , where λmax is
the critical value above which the solution of the problem is w
= 0.
Here λmax is more complicated to describe than in the simple lasso case
described above. Let θneg be the fraction of examples with bi = −1 and
θpos the fraction with bi = 1, and let b̃ ∈ Rm be a vector with entries
θneg where bi = 1 and −θpos where bi = −1. Then λmax = AT b̃∞ (see
[103, §2.1]). (While not relevant here, the final fitted model with λ =
0.1λmax classified the training examples with around 90% accuracy.)
Fitting the model involves solving the global consensus problem
(8.3) in §8.2 with local variables xi = (vi , wi ) and consensus variable
z = (v, w). As in the lasso example, we used abs = 10−4 and rel =
10−2 as tolerances and used the initialization u0i = 0, z 0 = 0. We use
the penalty parameter value ρ = 1 for the iterations.
We used L-BFGS to carry out the xi -updates. We used Nocedal’s
Fortran 77 implementation of L-BFGS with no tuning: We used default
parameters, a memory of 5, and a constant termination tolerance across
ADMM iterations (for a more efficient implementation, these toler-
ances would start large and decrease with ADMM iterations). We warm
started the xi -updates.
We used a serial implementation that performs the xi -updates
sequentially; in a distributed implementation, of course, the xi -updates
would be performed in parallel. To report an approximation of the tim-
ing that would have been achieved in a parallel implementation, we
report the maximum time required to update xi among the K subsys-
tems. This corresponds roughly to the maximum number of L-BFGS
iterations required for the xi -update.
Figure 11.4 shows the progress of the primal and dual residual
norm by iteration. The dashed line shows when the stopping criterion
has been satisfied (after 19 iterations), resulting in a primal residual
norm of about 1. Since the RMS consensus error can be expressed as
√
(1/ m)rk 2 where m = 106 , a primal residual norm of about 1 means
that on average, the elements of xi agree with z up to the third digit.
94 Numerical Examples

Fig. 11.4. Progress of primal and dual residual norm for distributed 1 regularized logistic
regression problem. The dashed lines show pri (top) and dual (bottom).

Fig. 11.5. Left. Objective suboptimality of distributed 1 regularized logistic regression

versus iteration. Right. Progress versus elapsed time. The stopping criterion is satisfied at
iteration 19, indicated by the vertical dashed line.

Figure 11.5 shows the suboptimality p̃k − p
for the consensus vari-
able, where

m  
k
p̃ = log 1 + exp(−bi (aTi wk + v k )) + λwk 1 .
i=1
 11.3 Group Lasso with Feature Splitting 95

The optimal value p
= 0.3302 × 106 was verified using l1 logreg

[103]. The lefthand plot shows ADMM progress by iteration, while the
righthand plot shows the cumulative time in a parallel implementa-
tion. It took 19 iterations to satisfy the stopping criterion. The first 4
iterations of ADMM took 2 seconds, while the last 4 iterations (before
the stopping criterion is satisfied) took less than 0.5 seconds. This is
because as the iterates approach consensus, L-BFGS requires fewer
iterations due to warm starting.

11.3 Group Lasso with Feature Splitting

We consider the group lasso example, described in §6.4.2,

minimize (1/2)Ax − b22 + λ N i=1 xi 2 ,

where x = (x1 , . . . , xN ), with xi ∈ Rni . We will solve the problem

by splitting across feature groups x1 , . . . , xN using the formulation
in §8.3.
We generated a problem instance with N = 200 groups of fea-
tures, with ni = 100 features per group, for i = 1, . . . , 200, for a total
of n = 20000 features and m = 200 examples. A ‘true’ value xtrue ∈ Rn
was generated, with 9 nonzero groups, resulting in 900 nonzero fea-
ture values. The feature matrix A is dense, with entries drawn from
an N (0, 1) distribution, and its columns then normalized to have unit

2 norm (as in the lasso example of §11.1). The outcomes b are gener-
ated by b = Axtrue + v, where v ∼ N (0, 0.1I), which corresponds to a
signal-to-noise ratio Axtrue 22 /v22 of around 60.
We used the penalty parameter ρ = 10 and set termination toler-
ances as abs = 10−4 and rel = 10−2 . The variables u0 and z 0 are ini-
tialized to be zero.
We used the regularization parameter value λ = 0.5λmax , where

λmax = max{AT1 b2 , . . . , ATN b2 }

is the critical value of λ above which the solution is x = 0. (Although

not relevant, this choice of λ correctly identifies 6 of the 9 nonzero
groups in xtrue and produces an estimate with 17 nonzero groups.) The
stopping criterion was satisfied after 47 iterations.
96 Numerical Examples

Fig. 11.6. Norms of primal residual (top) and dual residual (bottom) versus iteration, for
the distributed group lasso problem. The dashed lines show pri (top) and dual (bottom).

The xi -update is computed using the method described in §8.3.2,

which involves computing and caching eigendecompositions of ATi Ai .
The eigenvalue decompositions of ATi Ai took around 7 milliseconds;
subsequent xi -updates took around 350 microseconds, around a fac-
tor of 20 faster. For 47 ADMM iterations, these numbers predict a
total runtime in a serial implementation of about 5 seconds; the actual
runtime was around 7 seconds. For a parallel implementation, we can
estimate the runtime (neglecting interprocess communication and data
distribution) as being about 200 times faster, around 35 milliseconds.
Figure 11.6 shows the progress of the primal and dual residual norm
by iteration. The dashed line shows when the stopping criterion is sat-
isfied (after 47 iterations).
Figure 11.7 shows the suboptimality p̃k − p
for the problem versus
iteration, where


K
p̃ = (1/2)Ax −
k k
b22 +λ xki 2 .
i=1
 11.4 Distributed Large-Scale Lasso with MPI 97

Fig. 11.7. Suboptimality of distributed group lasso versus iteration. The stopping criterion
is satisfied at iteration 47, indicated by the vertical dashed line.

The optimal objective value p
= 430.8390 was found by running

ADMM for 1000 iterations.

11.4 Distributed Large-Scale Lasso with MPI

In previous sections, we discussed an idealized version of a distributed
implementation that was actually carried out serially for simplicity.
We now turn to a much more realistic distributed example, in which
we solve a very large instance of the lasso problem (6.2) using a dis-
tributed solver implemented in C using MPI for inter-process communi-
cation and the GNU Scientific Library (GSL) for linear algebra. In this
example, we split the problem across training examples rather than fea-
tures. We carried out the experiments in a cluster of virtual machines
running on Amazon’s Elastic Compute Cloud (EC2). Here, we focus
entirely on scaling and implementation details.
The data was generated as in §11.1, except that we now solve a prob-
lem with m = 400000 examples and n = 8000 features across N = 80
subsystems, so each subsystem handles 5000 training examples. Note
98 Numerical Examples

that the overall problem has a skinny coefficient matrix but each of
the subproblems has a fat coefficient matrix. We emphasize that the
coefficient matrix is dense, so the full dataset requires over 30 GB to
store and has 3.2 billion nonzero entries in the total coefficient matrix
A. This is far too large to be solved efficiently, or at all, using standard
serial methods on commonly available hardware.
We solved the problem using a cluster of 10 machines. We used
Cluster Compute instances, which have 23 GB of RAM, two quad-core
Intel Xeon X5570 ‘Nehalem’ chips, and are connected to each other
with 10 Gigabit Ethernet. We used hardware virtual machine images
running CentOS 5.4. Since each node had 8 cores, we ran the code
with 80 processes, so each subsystem ran on its own core. In MPI,
communication between processes on the same machine is performed
locally via the shared-memory Byte Transfer Layer (BTL), which pro-
vides low latency and high bandwidth communication, while commu-
nication across machines goes over the network. The data was sized
so all the processes on a single machine could work entirely in RAM.
Each node had its own attached Elastic Block Storage (EBS) volume
that contained only the local data relevant to that machine, so disk
throughput was shared among processes on the same machine but not
across machines. This is to emulate a scenario where each machine is
only processing the data on its local disk, and none of the dataset is
transferred over the network. We emphasize that usage of a cluster set
up in this fashion costs under $20 per hour.
We solved the problem with a deliberately naive implementation of
the algorithm, based directly on the discussion of §6.4, §8.2, and §10.2.
The implementation consists of a single file of C code, under 400 lines
despite extensive comments. The linear algebra (BLAS operations and
the Cholesky factorization) were performed using a stock installation
of the GNU Scientific Library.
We now report the breakdown of the wall-clock runtime. It took
roughly 30 seconds to load all the data into memory. It then took
4-5 minutes to form and then compute the Cholesky factorizations of
I + (1/ρ)Ai ATi . After caching these factorizations, it then took 0.5-2
seconds for each subsequent ADMM iteration. This includes the back-
solves in the xi -updates and all the message passing. For this problem,
 11.4 Distributed Large-Scale Lasso with MPI 99

Total dataset size 30 GB

Number of subsystems 80
Total dataset dimensions 400000 × 8000
Subsystem dimensions 5000 × 8000
Data loading time 30 seconds
Factorization time 5 minutes
Single iteration time 1 second
Total runtime 6 minutes

Table 11.2. Rough summary of a large dense distributed lasso example.

ADMM converged in 13 iterations, yielding a start-to-finish runtime of

under 6 minutes to solve the whole problem. Approximate times are
summarized in Table 11.2.
Though we did not compute it as part of this example, the extremely
low cost of each iteration means that it would be straightforward
to compute the entire regularization path for this problem using the
method described in §11.1.2. In that example, it required 428 iterations
to compute the regularization path for 100 settings of λ, while it took
around 15 iterations for a single instance to converge, roughly the same
as in this example. Extrapolating for this case, it is plausible that the
entire regularization path, even for this very large problem, could easily
be obtained in another five to ten minutes.
It is clear that by far the dominant computation is forming
and computing the Cholesky factorization, locally and in parallel, of
each ATi Ai + ρI (or I + (1/ρ)Ai ATi , if the matrix inversion lemma
is applied). As a result, it is worth keeping in mind that the perfor-
mance of the linear algebra operations in our basic implementation can
be significantly improved by using LAPACK instead of GSL for the
Cholesky factorization, and by replacing GSL’s BLAS implementation
with a hardware-optimized BLAS library produced by ATLAS, a ven-
dor library like Intel MKL, or a GPU-based linear algebra package. This
could easily lead to several orders of magnitude faster performance.
In this example, we used a dense coefficient matrix so the code
could be written using a single simple math library. Many real-world
examples of the lasso have larger numbers of training examples or fea-
tures, but are sparse and do not have billions of nonzero entries, as we
do here. The code we provide could be modified in the usual manner
100 Numerical Examples

to handle sparse or structured matrices (e.g., use CHOLMOD [35]

for sparse Cholesky factorization), and would also scale to very large
problems. More broadly, it could also be adapted with minimal work
to add constraints or otherwise modify the lasso problem, or even
solve completely different problems, like training logistic regression
or SVMs.
It is worth observing that ADMM scales well both horizontally and
vertically. We could easily have solved much larger problem instances in
roughly the same amount of time than the one described here by hav-
ing each subsystem solve a larger subproblem (up to the point where
each machine’s RAM is saturated, which it was not here); by running
more subsystems on each machine (though this can lead to perfor-
mance degradation in key areas like the factorization step); or simply
by adding more machines to the cluster, which is mostly straightfor-
ward and relatively inexpensive on Amazon EC2.
Up to a certain problem size, the solver can be implemented by users
who are not expert in distributed systems, distributed linear algebra,
or advanced implementation-level performance enhancements. This is
in sharp contrast to what is required in many other cases. Solving
extremely large problem instances requiring hundreds or thousands of
machines would require a more sophisticated implementation from a
systems perspective, but it is interesting to observe that a basic ver-
sion can solve rather large problems quickly on standard software and
hardware. To the best of our knowledge, the example above is one of
the largest lasso problems ever solved.

11.5 Regressor Selection

In our last example, we apply ADMM to an instance of the (nonconvex)
least squares regressor selection problem described in §9.1, which seeks
the best quadratic fit to a set of labels b from a combination of no more
than c columns of A (regressors). We use the same A and b generated for
the dense lasso example in §11.1, with m = 1500 examples and n = 5000
features, but instead of relying on the
1 regularization heuristic to
achieve a sparse solution, we explicitly constrain its cardinality to be
below c = 100.
 11.5 Regressor Selection 101

Fig. 11.8. Fit versus cardinality for the lasso (dotted line), lasso with posterior least squares
fit (dashed line), and regressor selection (solid line).

The x-update step has exactly the same expression as in the lasso
example, so we use the same method, based on the matrix inversion
lemma and caching, described in that example. The z-update step con-
sists of keeping the c largest magnitude components of x + u and zero-
ing the rest. For the sake of clarity, we performed an intermediate
sorting of the components, but more efficient schemes are possible. In
any case, the cost of the z-update is negligible compared with that of
the x-update.
Convergence of ADMM for a nonconvex problem such as this one
is not guaranteed; and even when it does converge, the final result can
depend on the choice of ρ and the initial values for z and u. To explore
this, we ran 100 ADMM simulations with randomly chosen initial val-
ues and ρ ranging between 0.1 and 100. Indeed, some of them did not
converge, or at least, were converging slowly. But most of them con-
verged, though not to exactly the same points. However, the objective
values obtained by those that converged were reasonably close to each
other, typically within 5%. The different values of x found had small
102 Numerical Examples

variations in support (choice of regressors) and value (weights), but the

largest weights were consistently assigned to the same regressors.
We now compare the use of nonconvex regressor selection with the
lasso, in terms of obtaining the sparsity-fit trade-off. We obtain this
curve for regressor selection by running ADMM for each value of c
between c = 1 and c = 120. For the lasso, we compute the regularization
path for 300 values of λ; for each xlasso found, we then perform a least
squares fit using the sparsity pattern in xlasso to get our final x. For
each cardinality, we plot the best fit found among all such x. Figure 11.8
shows the trade-off curves obtained by regressor selection and the lasso,
with and without posterior least squares fit. We see that while the
results are not exactly the same, they are quite similar, and for all
practical purposes, equivalent. This suggests that regressor selection via
ADMM can be used as well as lasso for obtaining a good cardinality-fit
trade-off; it might have an advantage when the desired cardinality is
known ahead of time.
 12
Conclusions

We have discussed ADMM and illustrated its applicability to dis-

tributed convex optimization in general and many problems in statis-
tical machine learning in particular. We argue that ADMM can serve
as a good general-purpose tool for optimization problems arising in the
analysis and processing of modern massive datasets. Much like gradient
descent and the conjugate gradient method are standard tools of great
use when optimizing smooth functions on a single machine, ADMM
should be viewed as an analogous tool in the distributed regime.
ADMM sits at a higher level of abstraction than classical optimiza-
tion algorithms like Newton’s method. In such algorithms, the base
operations are low-level, consisting of linear algebra operations and the
computation of gradients and Hessians. In the case of ADMM, the base
operations include solving small convex optimization problems (which
in some cases can be done via a simple analytical formula). For exam-
ple, when applying ADMM to a very large model fitting problem, each
update reduces to a (regularized) model fitting problem on a smaller
dataset. These subproblems can be solved using any standard serial
algorithm suitable for small to medium sized problems. In this sense,
ADMM builds on existing algorithms for single machines, and so can be

103
104 Conclusions

viewed as a modular coordination algorithm that ‘incentivizes’ a set of

simpler algorithms to collaborate to solve much larger global problems
together than they could on their own. Alternatively, it can be viewed
as a simple way of ‘bootstrapping’ specialized algorithms for small to
medium sized problems to work on much larger problems than would
otherwise be possible.
We emphasize that for any particular problem, it is likely that
another method will perform better than ADMM, or that some vari-
ation on ADMM will substantially improve performance. However, a
simple algorithm derived from basic ADMM will often offer perfor-
mance that is at least comparable to very specialized algorithms (even
in the serial setting), and in most cases, the simple ADMM algorithm
will be efficient enough to be useful. In a few cases, ADMM-based meth-
ods actually turn out to be state-of-the-art even in the serial regime.
Moreover, ADMM has the benefit of being extremely simple to imple-
ment, and it maps onto several standard distributed programming mod-
els reasonably well.
ADMM was developed over a generation ago, with its roots stretch-
ing far in advance of the Internet, distributed and cloud computing
systems, massive high-dimensional datasets, and the associated large-
scale applied statistical problems. Despite this, it appears to be well
suited to the modern regime, and has the important benefit of being
quite general in its scope and applicability.
 Acknowledgments

We are very grateful to Rob Tibshirani and Trevor Hastie for encour-
aging us to write this review. Thanks also to Alexis Battle, Dimitri
Bertsekas, Danny Bickson, Tom Goldstein, Dimitri Gorinevsky, Daphne
Koller, Vicente Malave, Stephen Oakley, and Alex Teichman for help-
ful comments and discussions. Yang Wang and Matt Kraning helped in
developing ADMM for the sharing and exchange problems, and Arezou
Keshavarz helped work out ADMM for generalized additive models. We
thank Georgios Giannakis and Alejandro Ribeiro for pointing out some
very relevant references that we had missed in an earlier version. We
thank John Duchi for a very careful reading of the manuscript and for
suggestions that greatly improved it.
Support for this work was provided in part by AFOSR grant
FA9550-09-0130 and NASA grant NNX07AEIIA. Neal Parikh was sup-
ported by the Cortlandt and Jean E. Van Rensselaer Engineering
Fellowship from Stanford University and by the National Science Foun-
dation Graduate Research Fellowship under Grant No. DGE-0645962.
Eric Chu was supported by the Pan Wen-Yuan Foundation Scholarship.

105
 A
Convergence Proof

The basic convergence result given in §3.2 can be found in several ref-
erences, such as [81, 63]. Many of these give more sophisticated results,
with more general penalties or inexact minimization. For completeness,
we give a proof here.
We will show that if f and g are closed, proper, and convex, and
the Lagrangian L0 has a saddle point, then we have primal residual
convergence, meaning that rk → 0, and objective convergence, meaning
that pk → p
, where pk = f (xk ) + g(z k ). We will also see that the dual
residual sk = ρAT B(z k − z k−1 ) converges to zero.
Let (x
, z
, y
) be a saddle point for L0 , and define

V k = (1/ρ)y k − y
22 + ρB(z k − z
)22 ,

We will see that V k is a Lyapunov function for the algorithm, i.e., a

nonnegative quantity that decreases in each iteration. (Note that V k is
unknown while the algorithm runs, since it depends on the unknown
values z
and y
.)
We first outline the main idea. The proof relies on three key inequal-
ities, which we will prove below using basic results from convex analysis

106
 107

along with simple algebra. The first inequality is

V k+1 ≤ V k − ρrk+1 22 − ρB(z k+1 − z k )22 . (A.1)
k
This states that V decreases in each iteration by an amount that
depends on the norm of the residual and on the change in z over one
iteration. Because V k ≤ V 0 , it follows that y k and Bz k are bounded.
Iterating the inequality above gives that
∞ 


ρ rk+1 22 + B(z k+1 − z k )22 ≤ V 0 ,
k=0

which implies that rk → 0 and B(z k+1 − z k ) → 0 as k → ∞. Multi-

plying the second expression by ρAT shows that the dual residual
sk = ρAT B(z k+1 − z k ) converges to zero. (This shows that the stop-
ping criterion (3.12), which requires the primal and dual residuals to
be small, will eventually hold.)
The second key inequality is
pk+1 − p

≤ −(y k+1 )T rk+1 − ρ(B(z k+1 − z k ))T (−rk+1 + B(z k+1 − z
)),
(A.2)
and the third inequality is
p
− pk+1 ≤ y
T rk+1 . (A.3)
The righthand side in (A.2) goes to zero as k → ∞, because B(z k+1 −
z
) is bounded and both rk+1 and B(z k+1 − z k ) go to zero. The right-
hand side in (A.3) goes to zero as k → ∞, since rk goes to zero. Thus
we have limk→∞ pk = p
, i.e., objective convergence.
Before giving the proofs of the three key inequalities, we derive the
inequality (3.11) mentioned in our discussion of stopping criterion from
the inequality (A.2). We simply observe that −rk+1 + B(z k+1 − z k ) =
−A(xk+1 − x
); substituting this into (A.2) yields (3.11),
pk+1 − p
≤ −(y k+1 )T rk+1 + (xk+1 − x
)T sk+1 .

Proof of inequality (A.3)

Since (x
, z
, y
) is a saddle point for L0 , we have
L0 (x
, z
, y
) ≤ L0 (xk+1 , z k+1 , y
).
108 Convergence Proof

Using Ax
+ Bz
= c, the lefthand side is p
. With pk+1 = f (xk+1 ) +

g(z k+1 ), this can be written as

p
≤ pk+1 + y
T rk+1 ,

which gives (A.3).

Proof of inequality (A.2)

By definition, xk+1 minimizes Lρ (x, z k , y k ). Since f is closed, proper,
and convex it is subdifferentiable, and so is Lρ . The (necessary and
sufficient) optimality condition is

0 ∈ ∂Lρ (xk+1 , z k , y k ) = ∂f (xk+1 ) + AT y k + ρAT (Axk+1 + Bz k − c).

(Here we use the basic fact that the subdifferential of the sum of a
subdifferentiable function and a differentiable function with domain
Rn is the sum of the subdifferential and the gradient; see, e.g., [140,
§23].)
Since y k+1 = y k + ρrk+1 , we can plug in y k = y k+1 − ρrk+1 and
rearrange to obtain

0 ∈ ∂f (xk+1 ) + AT (y k+1 − ρB(z k+1 − z k )).

This implies that xk+1 minimizes

f (x) + (y k+1 − ρB(z k+1 − z k ))T Ax.

A similar argument shows that z k+1 minimizes g(z) + y (k+1)T Bz. It

follows that
f (xk+1 ) + (y k+1 − ρB(z k+1 − z k ))T Axk+1
≤ f (x
) + (y k+1 − ρB(z k+1 − z k ))T Ax

and that

g(z k+1 ) + y (k+1)T Bz k+1 ≤ g(z
) + y (k+1)T Bz
.

Adding the two inequalities above, using Ax
+ Bz
= c, and rearrang-

ing, we obtain (A.2).
 109

Proof of inequality (A.1)

Adding (A.2) and (A.3), regrouping terms, and multiplying through by
2 gives
2(y k+1 − y
)T rk+1 − 2ρ(B(z k+1 − z k ))T rk+1
(A.4)
+ 2ρ(B(z k+1 − z k ))T (B(z k+1 − z
)) ≤ 0.
The result (A.1) will follow from this inequality after some manipula-
tion and rewriting.
We begin by rewriting the first term. Substituting y k+1 = y k +
ρr k+1 gives

2(y k − y
)T rk+1 + ρrk+1 22 + ρrk+1 22 ,

and substituting rk+1 = (1/ρ)(y k+1 − y k ) in the first two terms gives

(2/ρ)(y k − y
)T (y k+1 − y k ) + (1/ρ)y k+1 − y k 22 + ρrk+1 22 .

Since y k+1 − y k = (y k+1 − y
) − (y k − y
), this can be written as

 
(1/ρ) y k+1 − y
22 − y k − y
22 + ρrk+1 22 . (A.5)

We now rewrite the remaining terms, i.e.,

ρrk+1 22 − 2ρ(B(z k+1 − z k ))T rk+1 + 2ρ(B(z k+1 − z k ))T (B(z k+1 − z
)),

where ρrk+1 22 is taken from (A.5). Substituting

z k+1 − z
= (z k+1 − z k ) + (z k − z
)

in the last term gives

ρrk+1 − B(z k+1 − z k )22 + ρB(z k+1 − z k )22
+2ρ(B(z k+1 − z k ))T (B(z k − z
)),
and substituting

z k+1 − z k = (z k+1 − z
) − (z k − z
)

in the last two terms, we get

 
ρrk+1 − B(z k+1 − z k )22 + ρ B(z k+1 − z
)22 − B(z k − z
)22 .
110 Convergence Proof

With the previous step, this implies that (A.4) can be written as

V k − V k+1 ≥ ρrk+1 − B(z k+1 − z k )22 . (A.6)

To show (A.1), it now suffices to show that the middle term

−2ρr(k+1)T (B(z k+1 − z k )) of the expanded right hand side of (A.6)
is positive. To see this, recall that z k+1 minimizes g(z) + y (k+1)T Bz
and z k minimizes g(z) + y kT Bz, so we can add

g(z k+1 ) + y (k+1)T Bz k+1 ≤ g(z k ) + y (k+1)T Bz k

and

g(z k ) + y kT Bz k ≤ g(z k+1 ) + y kT Bz k+1

to get that

(y k+1 − y k )T (B(z k+1 − z k )) ≤ 0.

Substituting y k+1 − y k = ρrk+1 gives the result, since ρ > 0.

 References

[1] M. V. Afonso, J. M. Bioucas-Dias, and M. A. T. Figueiredo, “Fast image recov-

ery using variable splitting and constrained optimization,” IEEE Transactions
on Image Processing, vol. 19, no. 9, pp. 2345–2356, 2010.
[2] M. V. Afonso, J. M. Bioucas-Dias, and M. A. T. Figueiredo, “An Aug-
mented Lagrangian Approach to the Constrained Optimization Formulation of
Imaging Inverse Problems,” IEEE Transactions on Image Processing, vol. 20,
pp. 681–695, 2011.
[3] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. D. Croz, A. Green-
baum, S. Hammarling, A. McKenney, and D. Sorenson, LAPACK: A portable
linear algebra library for high-performance computers. IEEE Computing Soci-
ety Press, 1990.
[4] K. J. Arrow and G. Debreu, “Existence of an equilibrium for a competitive
economy,” Econometrica: Journal of the Econometric Society, vol. 22, no. 3,
pp. 265–290, 1954.
[5] K. J. Arrow, L. Hurwicz, and H. Uzawa, Studies in Linear and Nonlinear
Programming. Stanford University Press: Stanford, 1958.
[6] K. J. Arrow and R. M. Solow, “Gradient methods for constrained maxima,
with weakened assumptions,” in Studies in Linear and Nonlinear Program-
ming, (K. J. Arrow, L. Hurwicz, and H. Uzawa, eds.), Stanford University
Press: Stanford, 1958.
[7] O. Banerjee, L. E. Ghaoui, and A. d’Aspremont, “Model selection through
sparse maximum likelihood estimation for multivariate Gaussian or binary
data,” Journal of Machine Learning Research, vol. 9, pp. 485–516, 2008.

111
112 References

[8] P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe, “Convexity, classification,

and risk bounds,” Journal of the American Statistical Association, vol. 101,
no. 473, pp. 138–156, 2006.
[9] H. H. Bauschke and J. M. Borwein, “Dykstra’s alternating projection algo-
rithm for two sets,” Journal of Approximation Theory, vol. 79, no. 3, pp. 418–
443, 1994.
[10] H. H. Bauschke and J. M. Borwein, “On projection algorithms for solving
convex feasibility problems,” SIAM Review, vol. 38, no. 3, pp. 367–426, 1996.
[11] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm
for linear inverse problems,” SIAM Journal on Imaging Sciences, vol. 2, no. 1,
pp. 183–202, 2009.
[12] S. Becker, J. Bobin, and E. J. Candès, “NESTA: A fast and accurate first-
order method for sparse recovery,” Available at http://www.acm.caltech.
edu/˜emmanuel/papers/NESTA.pdf, 2009.
[13] J. F. Benders, “Partitioning procedures for solving mixed-variables program-
ming problems,” Numerische Mathematik, vol. 4, pp. 238–252, 1962.
[14] A. Bensoussan, J.-L. Lions, and R. Temam, “Sur les méthodes de
décomposition, de décentralisation et de coordination et applications,” Meth-
odes Mathematiques de l’Informatique, pp. 133–257, 1976.
[15] D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods.
Academic Press, 1982.
[16] D. P. Bertsekas, Nonlinear Programming. Athena Scientific, second ed., 1999.
[17] D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation:
Numerical Methods. Prentice Hall, 1989.
[18] J. M. Bioucas-Dias and M. A. T. Figueiredo, “Alternating Direction Algo-
rithms for Constrained Sparse Regression: Application to Hyperspectral
Unmixing,” arXiv:1002.4527, 2010.
[19] J. Borwein and A. Lewis, Convex Analysis and Nonlinear Optimization: The-
ory and Examples. Canadian Mathematical Society, 2000.
[20] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University
Press, 2004.
[21] L. M. Bregman, “Finding the common point of convex sets by the method
of successive projections,” Proceedings of the USSR Academy of Sciences,
vol. 162, no. 3, pp. 487–490, 1965.
[22] L. M. Bregman, “The relaxation method of finding the common point of con-
vex sets and its application to the solution of problems in convex program-
ming,” USSR Computational Mathematics and Mathematical Physics, vol. 7,
no. 3, pp. 200–217, 1967.
[23] H. Brézis, Opérateurs Maximaux Monotones et Semi-Groupes de Contractions
dans les Espaces de Hilbert. North-Holland: Amsterdam, 1973.
[24] A. M. Bruckstein, D. L. Donoho, and M. Elad, “From sparse solutions of
systems of equations to sparse modeling of signals and images,” SIAM Review,
vol. 51, no. 1, pp. 34–81, 2009.
[25] Y. Bu, B. Howe, M. Balazinska, and M. D. Ernst, “HaLoop: Efficient Iterative
Data Processing on Large Clusters,” Proceedings of the 36th International
Conference on Very Large Databases, 2010.
 References 113

[26] R. H. Byrd, P. Lu, and J. Nocedal, “A Limited Memory Algorithm for Bound
Constrained Optimization,” SIAM Journal on Scientific and Statistical Com-
puting, vol. 16, no. 5, pp. 1190–1208, 1995.
[27] E. J. Candès and Y. Plan, “Near-ideal model selection by
1 minimization,”
Annals of Statistics, vol. 37, no. 5A, pp. 2145–2177, 2009.
[28] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact
signal reconstruction from highly incomplete frequency information,” IEEE
Transactions on Information Theory, vol. 52, no. 2, p. 489, 2006.
[29] E. J. Candès and T. Tao, “Near-optimal signal recovery from random pro-
jections: Universal encoding strategies?,” IEEE Transactions on Information
Theory, vol. 52, no. 12, pp. 5406–5425, 2006.
[30] Y. Censor and S. A. Zenios, “Proximal minimization algorithm with D-
functions,” Journal of Optimization Theory and Applications, vol. 73, no. 3,
pp. 451–464, 1992.
[31] Y. Censor and S. A. Zenios, Parallel Optimization: Theory, Algorithms, and
Applications. Oxford University Press, 1997.
[32] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows,
T. Chandra, A. Fikes, and R. E. Gruber, “BigTable: A distributed storage
system for structured data,” ACM Transactions on Computer Systems, vol. 26,
no. 2, pp. 1–26, 2008.
[33] G. Chen and M. Teboulle, “A proximal-based decomposition method for con-
vex minimization problems,” Mathematical Programming, vol. 64, pp. 81–101,
1994.
[34] S. S. Chen, D. L. Donoho, and M. A. Saunders, “Atomic decomposition by
basis pursuit,” SIAM Review, vol. 43, pp. 129–159, 2001.
[35] Y. Chen, T. A. Davis, W. W. Hager, and S. Rajamanickam, “Algo-
rithm 887: CHOLMOD, supernodal sparse Cholesky factorization and
update/downdate,” ACM Transactions on Mathematical Software, vol. 35,
no. 3, p. 22, 2008.
[36] W. Cheney and A. A. Goldstein, “Proximity maps for convex sets,” Proceed-
ings of the American Mathematical Society, vol. 10, no. 3, pp. 448–450, 1959.
[37] C. T. Chu, S. K. Kim, Y. A. Lin, Y. Y. Yu, G. Bradski, A. Y. Ng, and
K. Olukotun, “MapReduce for machine learning on multicore,” in Advances
in Neural Information Processing Systems, 2007.
[38] J. F. Claerbout and F. Muir, “Robust modeling with erratic data,” Geophysics,
vol. 38, p. 826, 1973.
[39] P. L. Combettes, “The convex feasibility problem in image recovery,” Advances
in Imaging and Electron Physics, vol. 95, pp. 155–270, 1996.
[40] P. L. Combettes and J. C. Pesquet, “A Douglas-Rachford splitting approach
to nonsmooth convex variational signal recovery,” IEEE Journal on Selected
Topics in Signal Processing, vol. 1, no. 4, pp. 564–574, 2007.
[41] P. L. Combettes and J. C. Pesquet, “Proximal Splitting Methods in Signal
Processing,” arXiv:0912.3522, 2009.
[42] P. L. Combettes and V. R. Wajs, “Signal recovery by proximal forward-
backward splitting,” Multiscale Modeling and Simulation, vol. 4, no. 4,
pp. 1168–1200, 2006.
114 References

[43] G. B. Dantzig, Linear Programming and Extensions. RAND Corporation,

1963.
[44] G. B. Dantzig and P. Wolfe, “Decomposition principle for linear programs,”
Operations Research, vol. 8, pp. 101–111, 1960.
[45] I. Daubechies, M. Defrise, and C. D. Mol, “An iterative thresholding algorithm
for linear inverse problems with a sparsity constraint,” Communications on
Pure and Applied Mathematics, vol. 57, pp. 1413–1457, 2004.
[46] J. Dean and S. Ghemawat, “MapReduce: Simplified data processing on large
clusters,” Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008.
[47] J. W. Demmel, Applied Numerical Linear Algebra. SIAM: Philadelphia, PA,
1997.
[48] A. P. Dempster, “Covariance selection,” Biometrics, vol. 28, no. 1, pp. 157–
175, 1972.
[49] D. L. Donoho, “De-noising by soft-thresholding,” IEEE Transactions on Infor-
mation Theory, vol. 41, pp. 613–627, 1995.
[50] D. L. Donoho, “Compressed sensing,” IEEE Transactions on Information
Theory, vol. 52, no. 4, pp. 1289–1306, 2006.
[51] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms
for compressed sensing,” Proceedings of the National Academy of Sciences,
vol. 106, no. 45, p. 18914, 2009.
[52] D. L. Donoho and Y. Tsaig, “Fast solution of
1 -norm minimization problems
when the solution may be sparse,” Tech. Rep., Stanford University, 2006.
[53] J. Douglas and H. H. Rachford, “On the numerical solution of heat conduction
problems in two and three space variables,” Transactions of the American
Mathematical Society, vol. 82, pp. 421–439, 1956.
[54] J. C. Duchi, A. Agarwal, and M. J. Wainwright, “Distributed Dual Averaging
in Networks,” in Advances in Neural Information Processing Systems, 2010.
[55] J. C. Duchi, S. Gould, and D. Koller, “Projected subgradient methods for
learning sparse Gaussians,” in Proceedings of the Conference on Uncertainty
in Artificial Intelligence, 2008.
[56] R. L. Dykstra, “An algorithm for restricted least squares regression,” Journal
of the American Statistical Association, vol. 78, pp. 837–842, 1983.
[57] J. Eckstein, Splitting methods for monotone operators with applications to
parallel optimization. PhD thesis, MIT, 1989.
[58] J. Eckstein, “Nonlinear proximal point algorithms using Bregman func-
tions, with applications to convex programming,” Mathematics of Operations
Research, pp. 202–226, 1993.
[59] J. Eckstein, “Parallel alternating direction multiplier decomposition of convex
programs,” Journal of Optimization Theory and Applications, vol. 80, no. 1,
pp. 39–62, 1994.
[60] J. Eckstein, “Some saddle-function splitting methods for convex program-
ming,” Optimization Methods and Software, vol. 4, no. 1, pp. 75–83, 1994.
[61] J. Eckstein, “A practical general approximation criterion for methods of mul-
tipliers based on Bregman distances,” Mathematical Programming, vol. 96,
no. 1, pp. 61–86, 2003.
[62] J. Eckstein and D. P. Bertsekas, “An alternating direction method for linear
programming,” Tech. Rep., MIT, 1990.
 References 115

[63] J. Eckstein and D. P. Bertsekas, “On the Douglas-Rachford splitting method

and the proximal point algorithm for maximal monotone operators,” Mathe-
matical Programming, vol. 55, pp. 293–318, 1992.
[64] J. Eckstein and M. C. Ferris, “Operator-splitting methods for monotone
affine variational inequalities, with a parallel application to optimal control,”
INFORMS Journal on Computing, vol. 10, pp. 218–235, 1998.
[65] J. Eckstein and M. Fukushima, “Some reformulations and applications of the
alternating direction method of multipliers,” Large Scale Optimization: State
of the Art, pp. 119–138, 1993.
[66] J. Eckstein and B. F. Svaiter, “A family of projective splitting methods for
the sum of two maximal monotone operators,” Mathematical Programming,
vol. 111, no. 1-2, p. 173, 2008.
[67] J. Eckstein and B. F. Svaiter, “General projective splitting methods for sums
of maximal monotone operators,” SIAM Journal on Control and Optimization,
vol. 48, pp. 787–811, 2009.
[68] E. Esser, “Applications of Lagrangian-based alternating direction methods
and connections to split Bregman,” CAM report, vol. 9, p. 31, 2009.
[69] H. Everett, “Generalized Lagrange multiplier method for solving problems of
optimum allocation of resources,” Operations Research, vol. 11, no. 3, pp. 399–
417, 1963.
[70] M. J. Fadili and J. L. Starck, “Monotone operator splitting for optimization
problems in sparse recovery,” IEEE ICIP, 2009.
[71] A. V. Fiacco and G. P. McCormick, Nonlinear Programming: Sequential
Unconstrained Minimization Techniques. Society for Industrial and Applied
Mathematics, 1990. First published in 1968 by Research Analysis Corporation.
[72] M. A. T. Figueiredo and J. M. Bioucas-Dias, “Restoration of Poissonian
Images Using Alternating Direction Optimization,” IEEE Transactions on
Image Processing, vol. 19, pp. 3133–3145, 2010.
[73] M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, “Gradient projection
for sparse reconstruction: Application to compressed sensing and other inverse
problems,” IEEE Journal on Selected Topics in Signal Processing, vol. 1, no. 4,
pp. 586–597, 2007.
[74] P. A. Forero, A. Cano, and G. B. Giannakis, “Consensus-based distributed
support vector machines,” Journal of Machine Learning Research, vol. 11,
pp. 1663–1707, 2010.
[75] M. Fortin and R. Glowinski, Augmented Lagrangian Methods: Applications to
the Numerical Solution of Boundary-Value Problems. North-Holland: Ams-
terdam, 1983.
[76] M. Fortin and R. Glowinski, “On decomposition-coordination methods using
an augmented Lagrangian,” in Augmented Lagrangian Methods: Applications
to the Solution of Boundary-Value Problems, (M. Fortin and R. Glowinski,
eds.), North-Holland: Amsterdam, 1983.
[77] M. Forum, MPI: A Message-Passing Interface Standard, version 2.2. High-
Performance Computing Center: Stuttgart, 2009.
[78] Y. Freund and R. Schapire, “A decision-theoretic generalization of on-line
learning and an application to boosting,” in Computational Learning Theory,
pp. 23–37, Springer, 1995.
116 References

[79] J. Friedman, T. Hastie, and R. Tibshirani, “Sparse inverse covariance estima-

tion with the graphical lasso,” Biostatistics, vol. 9, no. 3, p. 432, 2008.
[80] M. Fukushima, “Application of the alternating direction method of multipli-
ers to separable convex programming problems,” Computational Optimization
and Applications, vol. 1, pp. 93–111, 1992.
[81] D. Gabay, “Applications of the method of multipliers to variational inequal-
ities,” in Augmented Lagrangian Methods: Applications to the Solution of
Boundary-Value Problems, (M. Fortin and R. Glowinski, eds.), North-Holland:
Amsterdam, 1983.
[82] D. Gabay and B. Mercier, “A dual algorithm for the solution of nonlinear vari-
ational problems via finite element approximations,” Computers and Mathe-
matics with Applications, vol. 2, pp. 17–40, 1976.
[83] M. Galassi, J. Davies, J. Theiler, B. Gough, G. Jungman, M. Booth, and
F. Rossi, GNU Scientific Library Reference Manual. Network Theory Ltd.,
third ed., 2002.
[84] A. M. Geoffrion, “Generalized Benders decomposition,” Journal of Optimiza-
tion Theory and Applications, vol. 10, no. 4, pp. 237–260, 1972.
[85] S. Ghemawat, H. Gobioff, and S. T. Leung, “The Google file system,” ACM
SIGOPS Operating Systems Review, vol. 37, no. 5, pp. 29–43, 2003.
[86] R. Glowinski and A. Marrocco, “Sur l’approximation, par elements finis
d’ordre un, et la resolution, par penalisation-dualité, d’une classe de problems
de Dirichlet non lineares,” Revue Française d’Automatique, Informatique, et
Recherche Opérationelle, vol. 9, pp. 41–76, 1975.
[87] R. Glowinski and P. L. Tallec, “Augmented Lagrangian methods for the
solution of variational problems,” Tech. Rep. 2965, University of Wisconsin-
Madison, 1987.
[88] T. Goldstein and S. Osher, “The split Bregman method for
1 regularized
problems,” SIAM Journal on Imaging Sciences, vol. 2, no. 2, pp. 323–343,
2009.
[89] E. G. Gol’shtein and N. V. Tret’yakov, “Modified Lagrangians in convex pro-
gramming and their generalizations,” Point-to-Set Maps and Mathematical
Programming, pp. 86–97, 1979.
[90] G. H. Golub and C. F. van Loan, Matrix Computations. Johns Hopkins Uni-
versity Press, third ed., 1996.
[91] D. Gregor and A. Lumsdaine, “The Parallel BGL: A generic library for dis-
tributed graph computations,” Parallel Object-Oriented Scientific Computing,
2005.
[92] A. Halevy, P. Norvig, and F. Pereira, “The Unreasonable Effectiveness of
Data,” IEEE Intelligent Systems, vol. 24, no. 2, 2009.
[93] K. B. Hall, S. Gilpin, and G. Mann, “MapReduce/BigTable for distributed
optimization,” in Neural Information Processing Systems: Workshop on
Learning on Cores, Clusters, and Clouds, 2010.
[94] T. Hastie and R. Tibshirani, Generalized Additive Models. Chapman & Hall,
1990.
[95] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learn-
ing: Data Mining, Inference and Prediction. Springer, second ed., 2009.
 References 117

[96] B. S. He, H. Yang, and S. L. Wang, “Alternating direction method with self-
adaptive penalty parameters for monotone variational inequalities,” Journal
of Optimization Theory and Applications, vol. 106, no. 2, pp. 337–356, 2000.
[97] M. R. Hestenes, “Multiplier and gradient methods,” Journal of Optimization
Theory and Applications, vol. 4, pp. 302–320, 1969.
[98] M. R. Hestenes, “Multiplier and gradient methods,” in Computing Methods
in Optimization Problems, (L. A. Zadeh, L. W. Neustadt, and A. V. Balakr-
ishnan, eds.), Academic Press, 1969.
[99] J.-B. Hiriart-Urruty and C. Lemaréchal, Fundamentals of Convex Analysis.
Springer, 2001.
[100] P. J. Huber, “Robust estimation of a location parameter,” Annals of Mathe-
matical Statistics, vol. 35, pp. 73–101, 1964.
[101] S.-J. Kim, K. Koh, S. Boyd, and D. Gorinevsky, “
1 Trend filtering,” SIAM
Review, vol. 51, no. 2, pp. 339–360, 2009.
[102] S.-J. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, “An interior-point
method for large-scale
1 -regularized least squares,” IEEE Journal of Selected
Topics in Signal Processing, vol. 1, no. 4, pp. 606–617, 2007.
[103] K. Koh, S.-J. Kim, and S. Boyd, “An interior-point method for large-scale
1 -
regularized logistic regression,” Journal of Machine Learning Research, vol. 1,
no. 8, pp. 1519–1555, 2007.
[104] D. Koller and N. Friedman, Probabilistic Graphical Models: Principles and
Techniques. MIT Press, 2009.
[105] S. A. Kontogiorgis, Alternating directions methods for the parallel solution of
large-scale block-structured optimization problems. PhD thesis, University of
Wisconsin-Madison, 1994.
[106] S. A. Kontogiorgis and R. R. Meyer, “A variable-penalty alternating direc-
tions method for convex optimization,” Mathematical Programming, vol. 83,
pp. 29–53, 1998.
[107] L. S. Lasdon, Optimization Theory for Large Systems. MacMillan, 1970.
[108] J. Lawrence and J. E. Spingarn, “On fixed points of non-expansive piecewise
isometric mappings,” Proceedings of the London Mathematical Society, vol. 3,
no. 3, p. 605, 1987.
[109] C. L. Lawson, R. J. Hanson, D. R. Kincaid, and F. T. Krogh, “Basic linear
algebra subprograms for Fortran usage,” ACM Transactions on Mathematical
Software, vol. 5, no. 3, pp. 308–323, 1979.
[110] D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factoriza-
tion,” Advances in Neural Information Processing Systems, vol. 13, 2001.
[111] J. Lin and M. Schatz, “Design Patterns for Efficient Graph Algorithms in
MapReduce,” in Proceedings of the Eighth Workshop on Mining and Learning
with Graphs, pp. 78–85, 2010.
[112] P. L. Lions and B. Mercier, “Splitting algorithms for the sum of two nonlinear
operators,” SIAM Journal on Numerical Analysis, vol. 16, pp. 964–979, 1979.
[113] D. C. Liu and J. Nocedal, “On the Limited Memory Method for Large Scale
Optimization,” Mathematical Programming B, vol. 45, no. 3, pp. 503–528,
1989.
118 References

[114] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein,

“GraphLab: A New Parallel Framework for Machine Learning,” in Conference
on Uncertainty in Artificial Intelligence, 2010.
[115] Z. Lu, “Smooth optimization approach for sparse covariance selection,” SIAM
Journal on Optimization, vol. 19, no. 4, pp. 1807–1827, 2009.
[116] Z. Lu, T. K. Pong, and Y. Zhang, “An Alternating Direction Method for
Finding Dantzig Selectors,” arXiv:1011.4604, 2010.
[117] D. G. Luenberger, Introduction to Linear and Nonlinear Programming.
Addison-Wesley: Reading, MA, 1973.
[118] J. Mairal, R. Jenatton, G. Obozinski, and F. Bach, “Network flow algorithms
for structured sparsity,” Advances in Neural Information Processing Systems,
vol. 24, 2010.
[119] G. Malewicz, M. H. Austern, A. J. C. Bik, J. C. Dehnert, I. Horn, N. Leiser,
and G. Czajkowski, “Pregel: A system for large-scale graph processing,” in
Proceedings of the 2010 International Conference on Management of Data,
pp. 135–146, 2010.
[120] A. F. T. Martins, M. A. T. Figueiredo, P. M. Q. Aguiar, N. A. Smith, and E. P.
Xing, “An Augmented Lagrangian Approach to Constrained MAP Inference,”
in International Conference on Machine Learning, 2011.
[121] G. Mateos, J.-A. Bazerque, and G. B. Giannakis, “Distributed sparse linear
regression,” IEEE Transactions on Signal Processing, vol. 58, pp. 5262–5276,
Oct. 2010.
[122] P. J. McCullagh and J. A. Nelder, Generalized Linear Models. Chapman &
Hall, 1991.
[123] N. Meinshausen and P. Bühlmann, “High-dimensional graphs and variable
selection with the lasso,” Annals of Statistics, vol. 34, no. 3, pp. 1436–1462,
2006.
[124] A. Miele, E. E. Cragg, R. R. Iver, and A. V. Levy, “Use of the augmented
penalty function in mathematical programming problems, part 1,” Journal of
Optimization Theory and Applications, vol. 8, pp. 115–130, 1971.
[125] A. Miele, E. E. Cragg, and A. V. Levy, “Use of the augmented penalty function
in mathematical programming problems, part 2,” Journal of Optimization
Theory and Applications, vol. 8, pp. 131–153, 1971.
[126] A. Miele, P. E. Mosely, A. V. Levy, and G. M. Coggins, “On the method of
multipliers for mathematical programming problems,” Journal of Optimiza-
tion Theory and Applications, vol. 10, pp. 1–33, 1972.
[127] J.-J. Moreau, “Fonctions convexes duales et points proximaux dans un espace
Hilbertien,” Reports of the Paris Academy of Sciences, Series A, vol. 255,
pp. 2897–2899, 1962.
[128] D. Mosk-Aoyama, T. Roughgarden, and D. Shah, “Fully distributed algo-
rithms for convex optimization problems,” Available at http://theory.
stanford.edu/˜tim/papers/distrcvxopt.pdf, 2007.
[129] I. Necoara and J. A. K. Suykens, “Application of a smoothing technique
to decomposition in convex optimization,” IEEE Transactions on Automatic
Control, vol. 53, no. 11, pp. 2674–2679, 2008.
 References 119

[130] A. Nedić and A. Ozdaglar, “Distributed subgradient methods for multi-

agent optimization,” IEEE Transactions on Automatic Control, vol. 54, no. 1,
pp. 48–61, 2009.
[131] A. Nedić and A. Ozdaglar, “Cooperative distributed multi-agent optimiza-
tion,” in Convex Optimization in Signal Processing and Communications,
(D. P. Palomar and Y. C. Eldar, eds.), Cambridge University Press, 2010.
[132] Y. Nesterov, “A method of solving a convex programming problem with
convergence rate O(1/k2 ),” Soviet Mathematics Doklady, vol. 27, no. 2,
pp. 372–376, 1983.
[133] Y. Nesterov, “Gradient methods for minimizing composite objective function,”
CORE Discussion Paper, Catholic University of Louvain, vol. 76, p. 2007,
2007.
[134] M. Ng, P. Weiss, and X. Yuang, “Solving Constrained Total-Variation Image
Restoration and Reconstruction Problems via Alternating Direction Meth-
ods,” ICM Research Report, Available at http://www.optimization-online.
org/DB_FILE/2009/10/2434.pdf, 2009.
[135] J. Nocedal and S. J. Wright, Numerical Optimization. Springer-Verlag, 1999.
[136] H. Ohlsson, L. Ljung, and S. Boyd, “Segmentation of ARX-models using sum-
of-norms regularization,” Automatica, vol. 46, pp. 1107–1111, 2010.
[137] D. W. Peaceman and H. H. Rachford, “The numerical solution of parabolic
and elliptic differential equations,” Journal of the Society for Industrial and
Applied Mathematics, vol. 3, pp. 28–41, 1955.
[138] M. J. D. Powell, “A method for nonlinear constraints in minimization prob-
lems,” in Optimization, (R. Fletcher, ed.), Academic Press, 1969.
[139] A. Ribeiro, I. Schizas, S. Roumeliotis, and G. Giannakis, “Kalman filtering in
wireless sensor networks — Incorporating communication cost in state esti-
mation problems,” IEEE Control Systems Magazine, vol. 30, pp. 66–86, Apr.
2010.
[140] R. T. Rockafellar, Convex Analysis. Princeton University Press, 1970.
[141] R. T. Rockafellar, “Augmented Lagrangians and applications of the prox-
imal point algorithm in convex programming,” Mathematics of Operations
Research, vol. 1, pp. 97–116, 1976.
[142] R. T. Rockafellar, “Monotone operators and the proximal point algorithm,”
SIAM Journal on Control and Optimization, vol. 14, p. 877, 1976.
[143] R. T. Rockafellar and R. J.-B. Wets, “Scenarios and policy aggregation in
optimization under uncertainty,” Mathematics of Operations Research, vol. 16,
no. 1, pp. 119–147, 1991.
[144] R. T. Rockafellar and R. J.-B. Wets, Variational Analysis. Springer-Verlag,
1998.
[145] L. Rudin, S. J. Osher, and E. Fatemi, “Nonlinear total variation based noise
removal algorithms,” Physica D, vol. 60, pp. 259–268, 1992.
[146] A. Ruszczyński, “An augmented Lagrangian decomposition method for block
diagonal linear programming problems,” Operations Research Letters, vol. 8,
no. 5, pp. 287–294, 1989.
[147] A. Ruszczyński, “On convergence of an augmented Lagrangian decomposition
method for sparse convex optimization,” Mathematics of Operations Research,
vol. 20, no. 3, pp. 634–656, 1995.
120 References

[148] K. Scheinberg, S. Ma, and D. Goldfarb, “Sparse inverse covariance selection

via alternating linearization methods,” in Advances in Neural Information
Processing Systems, 2010.
[149] I. D. Schizas, G. Giannakis, S. Roumeliotis, and A. Ribeiro, “Consensus in ad
hoc WSNs with noisy links — part II: Distributed estimation and smoothing of
random signals,” IEEE Transactions on Signal Processing, vol. 56, pp. 1650–
1666, Apr. 2008.
[150] I. D. Schizas, A. Ribeiro, and G. B. Giannakis, “Consensus in ad hoc WSNs
with noisy links — part I: Distributed estimation of deterministic signals,”
IEEE Transactions on Signal Processing, vol. 56, pp. 350–364, Jan. 2008.
[151] B. Schölkopf and A. J. Smola, Learning with Kernels: Support Vector
Machines, Regularization, Optimization, and Beyond. MIT Press, 2002.
[152] N. Z. Shor, Minimization Methods for Non-Differentiable Functions. Springer-
Verlag, 1985.
[153] J. E. Spingarn, “Applications of the method of partial inverses to convex pro-
gramming: decomposition,” Mathematical Programming, vol. 32, pp. 199–223,
1985.
[154] G. Steidl and T. Teuber, “Removing multiplicative noise by Douglas-Rachford
splitting methods,” Journal of Mathematical Imaging and Vision, vol. 36,
no. 2, pp. 168–184, 2010.
[155] C. H. Teo, S. V. N. Vishwanathan, A. J. Smola, and Q. V. Le, “Bundle meth-
ods for regularized risk minimization,” Journal of Machine Learning Research,
vol. 11, pp. 311–365, 2010.
[156] R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of
the Royal Statistical Society, Series B, vol. 58, pp. 267–288, 1996.
[157] P. Tseng, “Applications of a splitting algorithm to decomposition in convex
programming and variational inequalities.,” SIAM Journal on Control and
Optimization, vol. 29, pp. 119–138, 1991.
[158] P. Tseng, “Alternating projection-proximal methods for convex program-
ming and variational inequalities,” SIAM Journal on Optimization, vol. 7,
pp. 951–965, 1997.
[159] P. Tseng, “A modified forward-backward splitting method for maximal mono-
tone mappings,” SIAM Journal on Control and Optimization, vol. 38, p. 431,
2000.
[160] J. N. Tsitsiklis, Problems in decentralized decision making and computation.
PhD thesis, Massachusetts Institute of Technology, 1984.
[161] J. N. Tsitsiklis, D. P. Bertsekas, and M. Athans, “Distributed asynchronous
deterministic and stochastic gradient optimization algorithms,” IEEE Trans-
actions on Automatic Control, vol. 31, no. 9, pp. 803–812, 1986.
[162] H. Uzawa, “Market mechanisms and mathematical programming,” Economet-
rica: Journal of the Econometric Society, vol. 28, no. 4, pp. 872–881, 1960.
[163] H. Uzawa, “Walras’ tâtonnement in the theory of exchange,” The Review of
Economic Studies, vol. 27, no. 3, pp. 182–194, 1960.
[164] L. G. Valiant, “A bridging model for parallel computation,” Communications
of the ACM, vol. 33, no. 8, p. 111, 1990.
 References 121

[165] V. N. Vapnik, The Nature of Statistical Learning Theory. Springer-Verlag,

2000.
[166] J. von Neumann, Functional Operators, Volume 2: The Geometry of Orthogo-
nal Spaces. Princeton University Press: Annals of Mathematics Studies, 1950.
Reprint of 1933 lecture notes.
[167] M. J. Wainwright and M. I. Jordan, “Graphical models, exponential fami-
lies, and variational inference,” Foundations and Trends in Machine Learning,
vol. 1, no. 1-2, pp. 1–305, 2008.
[168] L. Walras, Éléments d’économie politique pure, ou, Théorie de la richesse
sociale. F. Rouge, 1896.
[169] S. L. Wang and L. Z. Liao, “Decomposition method with a variable parameter
for a class of monotone variational inequality problems,” Journal of Optimiza-
tion Theory and Applications, vol. 109, no. 2, pp. 415–429, 2001.
[170] T. White, Hadoop: The Definitive Guide. O’Reilly Press, second ed., 2010.
[171] J. M. Wooldridge, Introductory Econometrics: A Modern Approach. South
Western College Publications, fourth ed., 2009.
[172] L. Xiao and S. Boyd, “Fast linear iterations for distributed averaging,” Systems
& Control Letters, vol. 53, no. 1, pp. 65–78, 2004.
[173] A. Y. Yang, A. Ganesh, Z. Zhou, S. S. Sastry, and Y. Ma, “A Review of Fast

1 -Minimization Algorithms for Robust Face Recognition,” arXiv:1007.3753,
2010.
[174] J. Yang and X. Yuan, “An inexact alternating direction method for
trace norm regularized least squares problem,” Available at http://www.
optimization-online.org, 2010.
[175] J. Yang and Y. Zhang, “Alternating direction algorithms for
1 -problems in
compressive sensing,” Preprint, 2009.
[176] W. Yin, S. Osher, D. Goldfarb, and J. Darbon, “Bregman iterative algorithms
for
1 -minimization with applications to compressed sensing,” SIAM Journal
on Imaging Sciences, vol. 1, no. 1, pp. 143–168, 2008.
[177] M. Yuan and Y. Lin, “Model selection and estimation in regression with
grouped variables,” Journal of the Royal Statistical Society: Series B (Sta-
tistical Methodology), vol. 68, no. 1, pp. 49–67, 2006.
[178] X. M. Yuan, “Alternating direction methods for sparse covariance selec-
tion,” Preprint, Available at http://www.optimization-online.org/DB_
FILE/2009/09/2390.pdf, 2009.
[179] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark:
Cluster computing with working sets,” in Proceedings of the 2nd USENIX
Conference on Hot Topics in Cloud Computing, 2010.
[180] T. Zhang, “Statistical behavior and consistency of classification methods based
on convex risk minimization,” Annals of Statistics, vol. 32, no. 1, pp. 56–85,
2004.
[181] P. Zhao, G. Rocha, and B. Yu, “The composite absolute penalties family
for grouped and hierarchical variable selection,” Annals of Statistics, vol. 37,
no. 6A, pp. 3468–3497, 2009.
122 References

[182] H. Zhu, A. Cano, and G. B. Giannakis, “Distributed consensus-based demod-

ulation: algorithms and error analysis,” IEEE Transactions on Wireless Com-
munications, vol. 9, no. 6, pp. 2044–2054, 2010.
[183] H. Zhu, G. B. Giannakis, and A. Cano, “Distributed in-network channel decod-
ing,” IEEE Transactions on Signal Processing, vol. 57, no. 10, pp. 3970–3983,
2009.