LEO LIBERTI

M AT H E M AT I C A L
PROGRAMMING

E C O L E P O LY T E C H N I Q U E
Copyright © 2018 Leo Liberti

published by ecole polytechnique

typeset using LATEX on template https://tufte- latex.github.io/tufte- latex/

Zeroth printing, February 2018

 Contents

I An overview of Mathematical Programming 11

1 Imperative and declarative programming 13

1.1 Turing machines 13
1.2 Register machines 14
1.3 Universality 14
1.4 Partial recursive functions 14
1.5 The main theorem 15
1.5.1 The Church-Turing thesis and computable functions 16
1.6 Imperative and declarative programming 16
1.7 Mathematical programming 17
1.7.1 The transportation problem 17
1.7.2 Network flow 19
1.7.3 Extremely short summary of complexity theory 20
1.7.4 Definitions 21
1.7.5 The canonical MP formulation 22
1.8 Modelling software 22
1.8.1 Structured and flat formulations 23
1.8.2 AMPL 23
1.8.3 Python 24
1.9 Summary 26

2 Systematics and solution methods 27

2.1 Linear programming 27
2.2 Mixed-integer linear programming 28
2.3 Integer linear programming 30
 4

2.4 Binary linear programming 30

2.4.1 Linear assignment 30
2.4.2 Vertex cover 31
2.4.3 Set covering 31
2.4.4 Set packing 32
2.4.5 Set partitioning 32
2.4.6 Stables and cliques 32
2.4.7 Other combinatorial optimization problems 33
2.5 Convex nonlinear programming 33
2.6 Nonlinear programming 34
2.7 Convex mixed-integer nonlinear programming 36
2.8 Mixed-integer nonlinear programming 37
2.9 Quadratic programming formulations 37
2.9.1 Convex quadratic programming 37
2.9.2 Quadratic programming 38
2.9.3 Binary quadratic programming 39
2.9.4 Quadratically constrained quadratic programming 40
2.10 Semidefinite programming 40
2.10.1 Second-order cone programming 42
2.11 Multi-objective programming 43
2.11.1 Maximum flow with minimum capacity 44
2.12 Bilevel programming 44
2.12.1 Lower-level LP 45
2.12.2 Minimum capacity maximum flow 46
2.13 Semi-infinite programming 46
2.13.1 LP under uncertainty 47
2.14 Summary 48

3 Reformulations 51
3.1 Elementary reformulations 52
3.1.1 Objective function direction and constraint sense 52
3.1.2 Liftings, restrictions and projections 52
3.1.3 Equations to inequalities 52
3.1.4 Inequalities to equations 52
3.1.5 Absolute value terms 52
 5

3.1.6 Product of exponential terms 53

3.1.7 Binary to continuous variables 53
3.1.8 Discrete to binary variables 53
3.1.9 Integer to binary variables 53
3.1.10 Feasibility to optimization problems 54
3.1.11 Minimization of the maximum of finitely many functions 54
3.2 Exact linearizations 54
3.2.1 Product of binary variables 54
3.2.2 Product of a binary variable by a bounded continuous variable 55
3.2.3 Linear complementarity 55
3.2.4 Minimization of absolute values 55
3.2.5 Linear fractional terms 57
3.3 Advanced examples 57
3.3.1 The Hyperplane Clustering Problem 57
3.3.2 Selection of software components 61
3.4 Summary 64

II In-depth topics 65

4 Constraint programming 67
4.1 The dual CP 67
4.2 CP and MP 68
4.3 The CP solution method 69
4.3.1 Domain reduction and consistency 69
4.3.2 Pruning and solving 70
4.4 Objective functions in CP 71
4.5 Some surprising constraints in CP 71
4.6 Sudoku 72
4.6.1 AMPL code 72
4.7 Summary 73

5 Maximum cut 75
5.1 Approximation algorithms 76
5.1.1 Approximation schemes 76
 6

5.2 Randomized algorithms 76

5.2.1 Randomized approximation algorithms 77

5.3 MP formulations 78
5.3.1 A natural formulation 78
5.3.2 A BQP formulation 79
5.3.3 Another quadratic formulation 79
5.3.4 An SDP relaxation 79

5.4 The Goemans-Williamson algorithm 82

5.4.1 Randomized rounding 82
5.4.2 Sampling a uniform random vector from Sn−1 84
5.4.3 What can go wrong in practice? 85

5.5 Python implementation 85

5.5.1 Preamble 85
5.5.2 Functions 86
5.5.3 Main 89
5.5.4 Comparison on a set of random weighted graphs 90

5.6 Summary 91

6 Distance Geometry 93
6.1 The fundamental problem of DG 93
6.2 Some applications of the DGP 94
6.2.1 Clock synchronization 94
6.2.2 Sensor network localization 94
6.2.3 Protein conformation 94
6.2.4 Unmanned underwater vehicles 95

6.3 Formulations of the DGP 95

6.4 The 1-norm and the max norm 95
6.4.1 The 2-norm 96

6.5 Euclidean Distance Matrices 96

6.5.1 The Gram matrix from the square EDM 97
6.5.2 Is a given matrix a square EDM? 98
6.5.3 The rank of a square EDM 98
 7

6.6 The EDM Completion Problem 99

6.6.1 An SDP formulation for the EDMCP 99
6.6.2 Inner LP approximation of SDPs 100
6.6.3 Refining the DD approximation 101
6.6.4 Iterative DD approximation for the EDMCP 102
6.6.5 Working on the dual 103
6.6.6 Can we improve the DD condition? 103
6.7 The Isomap heuristic 104
6.7.1 Isomap and DG 104
6.8 Random projections 104
6.8.1 Usefulness for solving DGPs 105
6.9 How to adjust an approximately feasible solution 105
6.10 Summary 106

Bibliography 107

Index 115
Introduction

These lecture notes are a companion to a revamped INF580

course at DIX, Ecole Polytechnique, academic year 2015/2016. The
course title is actually “Constraint Programming” (CP), but I intend
to shift the focus to Mathematical Programming (MP), by which I
mean a summary of techniques for programming computers by
supplying a mathematical description of the solution properties.
This course is about stating what your solution is like, and then
letting some general method tackle the problem. MP is a paradigm
shift from solving towards modelling. This course is different from
traditional MP courses. I do not explain the simplex method, or
Branch-and-Bound, or Gomory cuts. Heck, I don’t even explain
what the linear programming dual is! (There are other courses at
Ecole Polytechnique for that.) I want to convey an overview about
using MP as a tool: the many possible types of MP formulations,
how to change formulations to fit a solver requirement or improve
its performance, how to use MP within more complex algorithms,
how to scale MP techniques to large sizes, with some comments
about how MP can help you when your client tells you he has a
problem and does not know what the problem is, let alone how to
solve it!
A word about notation: letter symbols (such as x) usually denote
tensors1 of quantities. So xi could be an element of x, and xij an 1
I.e. multi-dimensional arrays.
element of xi . And so on. Usually the correct dimensions are either
specified, or can be understood from the context. If in doubt, ask
(write to [email protected]) — I will clarify and
perhaps amend the notes.
 Part I

An overview of
Mathematical
Programming
1
Imperative and declarative programming

People usually program a computer by writing a “recipe”

which looks like a set of prescriptions: do this, do that if condition
is true, repeat.
But there is another way of programming a computer, which
consists in putting together complex functions using simpler ones,
such that the end function exactly describes the output of the
problem we are trying to solve.
Figure 1.1: A Turing Machine (from
2009.igem.org/Turing_machines).

1.1 Turing machines

You might be familiar with the idea of a Turing Machine (TM):

there is an infinitely long tape divided in cells, each of which can
bear a symbol from some alphabet1 A. 1
An alphabet is an arbitrary set of sym-
A mechanical head, capable of writing symbols on cells or eras- bols, such as e.g. {0, 1} or {,
, } or
{a,b,. . . ,z}. We assume it is finite.
ing them, can move left and right along the tape, or simply rest,
according to some instructions labeled by states2 . 2
The name “state” probably stems
For example, a state q from a finite state set Q corresponds to the from a wish of Turing to equate the
machine states with “states of mind”
following instruction: [Turing, 1937].

q: read symbol s from current cell, depending on s perform an

movement µ ∈ M = {right, left, stay}, write a symbol t ∈ A in the new
cell, and switch to a new state r.

We can formally store such instructions in a dictionary3 π : Q → 3

A dictionary is a data structure
A × M × A × Q mapping states q to quadruplets (s, µ, t, r ). which stores a map key 7→ value in
the form of (key, value) pairs, and is
We assume Q must contain an initial state S0 and a final state optimized for efficiently retrieving
Shalt . Then the TM with program π starts in state S0 , and applies the values corresponding to a given key —
see your favorite basic data structure
rules in π until r = Shalt . course.
Any TM is given the sequence of symbols written in the cells of
the tape prior to starting the computation as input, and obtains as
an output the sequence of symbols written in the cells of the tape
after the computation ends. Note that the computation need not
always end. Consider, e.g., the TM with alphabet A = {0, 1} and
the program π (1) = (0, stay, 0, 2) and π (2) = (0, stay, 0, 1) with
starting tape (0, 0, . . .), S0 = 1 and Shalt = 3: this simply flips states
from 1 to 2 and back to 1 forever. If we take the sequence (1, 1, . . .)
14 mathematical programming

as a starting tape, we find no relevant instruction in π (1) which

starts with 1, and hence the TM simply “gets stuck” and never
terminates.

1.2 Register machines

Computation models which are closer to today’s computers are

register machines (RM) [Minsky, 1967]. RMs consist of a finite set R
of registers R1 , R2 , . . . which can contain a natural number, a finite
set Q of states, and some very basic instructions4 such as: 4
The instruction may vary, according
to different authors; those given below
(a) given j ∈ R, let R j ← R j + 1, then switch to a new state r are very simple, and due to Marvin
Minsky.

(b) given j ∈ R, if R j > 0 then R j ← R j − 1 then switch to a new

state r else switch to a new state s.

Registers are similar to bytes of memory, states are used to enu-

merate instructions in a program (as for TMs), and the instructions
have very basic semantics on the memory similar to what you could
find in a very simple CPU.

1.3 Universality
Figure 1.2: A program in Minsky’s RM
(from [Liberti and Marinelli, 2014]).
TMs and RMs are flexible enough so that they can “simulate” other
TMs and RMs. Specifically, we are able to design a TM U such that

U (h T, x i) = T ( x ), (1.1)

where T is the description of any TM, h T, x i is a pair given to U as

input, and T ( x ) denotes the output of T when fed the input x. The
TM U is called a universal TM (UTM).5 The same can be said for 5
In more recent terminology, the
RMs. program of a UTM is sometimes
described as an interpreter.
How do we “describe a TM”? Since there are only countably
many possible TM (or RM) descriptions, we can count them, and
then match them to natural numbers. We can therefore speak of the
i-th TM (or RM) where i ∈ N. In fact we can do better: through
Gödel numbers6 we can compute i from the TM (or RM) description, 6
Many Gödel number schemes are
and given an i ∈ N computed this way, we can infer the TM (or possible. Gödel actually defined them
for logical propositions, but every
RM) which originated it. sequence s = (s1 , . . . , sk ) of symbols
UTMs can be used to prove several mind-boggling results, such of A can be encoded as an integer,
and any thusly obtained integer
as the existence of problems which cannot be solved by any TM.7 can be decoded into the original
sequence. Gödel assigned the number
s
G(s) = 2s1 3s2 5s3 · · · pkk to the sequence
1.4 Partial recursive functions s. [Gödel, 1930]
7
Just google halting problem, see
Fig. 1.4.

Without any loss of generality, we consider TMs or RMs having

natural numbers as inputs and output. We investigate a class of
functions φ : Nk → N, called partial recursive (p.r.) [Cutland, 1980],
 imperative and declarative programming 15

for all values of k ≥ 0, built recursively using some basic functions

and basic operations on functions.
The basic p.r. functions are:

• for all c ∈ N, the constant functions constc (s1 , . . . , sk ) = c;

• the successor function succ(s) = s + 1 where s ∈ N, defined for

k = 1 only;

• for all k and j ≤ k, the projection functions projkj (s1 , . . . , sk ) = s j .

The basic operations are:

• composition: if f : Nk → N and g : N → N are p.r., then

g ◦ f : Nk → N mapping s 7→ g( f (s)) is p.r.

• primitive recursion:8 if f : Nk → N and g : Nk+2 → N are p.r., Figure 1.3: Alonzo Church, grey
eminence of recursive functions.
then the function φ f ,g : Nk+1 → N defined by: 8
For example, the factorial function
n! is defined by primitive recursion
φ(s, 0) = f (s) (1.2) by setting k = 0, f = const1 = 1,
g(n) = n(n − 1)!.
φ(s, n + 1) = g(s, n, φ(s, n)) (1.3)

is p.r.
• minimalization: if f : Nk+1 → N, then the function ψ : Nk → N
maps s to the least9 natural number y such that, for each z ≤ y, 9
This is a formalization of an opti-
f (s, z) is defined and f (s, y) = 0 is p.r. mization problem in number theory:
finding a minimum natural number
which satisfies a given p.r. property f
Notationally, we denote minimalization by means of the search is p.r.
quantifier µ:10 10
The µ quantifier is typical of com-
putability. Eq. (1.4) is equivalent
ψ(s) ≡ µy (∀z ≤ y ( f (s, z) is defined) ∧ f (s, y) = 0). (1.4) to ψ(s) = min{y ∈ N | ∀z ≤
y ( f (s, y) is defined) ∧ f (s, y) = 0}.

1.5 The main theorem

The main result in computability states that TMs (or RMs) and
p.r. functions are both universal models of computation. More
precisely, let T be the description of a TM taking an input s ∈
Nk and producing an output in N, and let Φ T : Nk → N the
associated function s 7→ T (s). Then Φ T is p.r.; and, conversely, for
any p.r. function f there is a TM T such that Φ T = f .
Here is the sketch of a proof. (⇒) Given a p.r. function, we
must show there is a TM which computes it. It is possible, and not
Figure 1.4: The Halting Problem
too difficult,11 to show that there are TMs which compute basic (vinodwadhawan.blogspot.com).
p.r. functions and basic p.r. operations. Since we can combine TMs 11
See Ch. 3, Lemma 1.1, Thm. 3.1, 4.4,
in the appropriate ways, for any p.r. function we can construct a 5.2 in [Cutland, 1980]
TM which computes it.
(⇐) Conversely, given a TM T with input/output function
τ : Rn → R, we want to show that τ ( x ) is a p.r. function.12 We 12
This is the most interesting direction
assume T’s alphabet is the set N of natural numbers, and that the for the purposes of these lecture notes,
since it involves “modelling” a TM by
output of T (if the TM terminates) is going to be stored in a given means of p.r. functions.
16 mathematical programming

“output” cell of the TM tape. We define two functions:

(
value in output cell after t steps of T on input x
ω ( x, t) =
τ ( x ) if T stopped before t steps on input x
(
state label of next instruction after t steps of T Start here ↓
σ ( x, t) =
0 if T stops after t steps or fewer,

where we assumed that Shalt = 0. Now there are two cases: either T
terminates on x, or it does not. If it terminates, there is a number t∗
of steps after which T terminates, and

t∗ = µt(σ( x, t) = 0) (1.5)
∗
τ (x) = ω ( x, t ). (1.6)

Note that Eq. (1.5) says that t∗ is the least t such that σ ( x, t) = 0 and
σ ( x, u) is defined for each u ≤ t.
If T does not terminate on x, then the two functions µt(σ ( x, t) =
0) and τ ( x ) are undefined on x. In either case, we have

τ ( x ) ≡ ω ( x, µt(σ ( x, t) = 0)).

By minimalization, the fact that τ ( x ) is p.r. follows if we prove that

over to next column →
ω and σ are p.r. This is not hard to do, but it is quite long, and
is beside the scope of these lectures. Informally, one starts from
← finally, this
the Gödel number of T, decodes it into T (this decoding function
is p.r.), and essentially executes T over its input until step t. That
this last task is p.r. follows by decomposing it into two functions:
finding the new state and tape content after an instruction has been Figure 1.5: Details of the proof from
[Cutland, 1980], p. 95-99 — with
executed, and finding the next instruction. These two functions are
slightly different notation.
in fact primitive recursive,13 and hence p.r. (see Fig. 1.5), which 13
A function is primitive recursive if
concludes this proof sketch.14 it can be build from repeated appli-
cation of composition and primitive
recursion to the basic p.r. functions —
1.5.1 The Church-Turing thesis and computable functions i.e. minimalization is not necessary.
14
This proof sketch was adapted from
[Cutland, 1980]
Not only TMs and p.r. functions can be used to express univer-
sal computation. Every “sensible” computational model ever imag-
ined was shown to be equivalent to TMs, RMs and p.r. functions.
The Church-Turing thesis states that this empirical observation is in
fact a truth. Any function which can be computed by a TM is called
a computable function.

1.6 Imperative and declarative programming

Figure 1.6: TMs are all that is the case!

Today, most people program a computer by means of imperative (rationallyspeaking.blogspot.fr).

instructions. We might tell a computer to store a certain value in

a given register, to test a condition and move control to a different
instruction if the condition is verified, or unconditionally move
control to another instruction. If we are using a high-level language,
we use assignments, tests and loops.15 Languages whose programs 15
Assignments, tests and loops are
the essential syntactical elements of
any computer language which makes
the language universal, i.e. capable of
describing a universal TM, see [Harel,
1980]
 imperative and declarative programming 17

consist of finite lists of imperative instructions are called imperative

languages.16
The execution environment of an imperative language is a TM,
or RM, or similar — in short, a “real computer”. As is well known,
E.g. Fortran, Basic, C/C++, Java,
16

CPUs take instructions of the imperative type, such as “move a Python.

value from a register to a location in memory” or “jump to instruc-
tion such-and-such”.
We saw in previous sections that p.r. functions have the same
expressive capabilities as imperative languages. And yet, we do
not prescribe any action when writing a p.r. function. This begs
the question: can one program using p.r. functions? This is indeed
possible, as the many existing declarative languages testify.17 17
E.g. LISP, CAML.
Declarative languages based on p.r. functions usually consist of
a rather rich set of basic functions, and various kinds of operators
in order to put together new functions.18 The complicated functions 18
Although it would be theoretically
obtained using recursive constructions from the basic ones are possible to limit ourselves to the basic
functions and operators above, this
the “programs” of a declarative language. Each such function is would yield unwieldy programs for a
evaluated at the input in order to obtain the output. Since each human to code.

complex function is the result of various operators being applied

recursively to some basic functions, its output can be obtained
accordingly.
The natural execution environment of a declarative language
would be a machine which can natively evaluate basic functions,
and natively combine evaluations using basic operators. In practice,
interpreters are used to turn declarative programs RM descriptions
which can be executed on real computers.

1.7 Mathematical programming

Figure 1.7: Academic journals in MP.

We now come to the object of these notes. By “Mathematical
Programming” we indicate a special type of declarative program-
ming, namely one where the interpreter varies according to the
mathematical properties of the functions involved in the program.
For example, if all the functions in a MP are linear in the vari-
ables, and the variables range over continuous domains,19 one can 19
Optimizing a linear form over
use very efficient algorithms as interpreters, such as the simplex20 a polyhedron is known as linear
programming.
or barrier21 methods [Matoušek and Gärtner, 2007]. If the functions 20
From any vertex of the polyhedron,
are nonlinear or the variables are integer, one must resort to enu- the simplex method finds the adjacent
merative methods such as branch-and-bound. [Liberti, 2006] Working vertex improving the value of the
linear form, until this is no longer
implementations of these solution methods are collectively referred possible.
to as solvers. They read the MP (the program), the input of the prob- 21
From any point inside a polyhedron,
lem, and they execute an imperative program which runs on the the barrier method defines a central
path in the interior of the polyhedron,
physical hardware. which ends at the vertex which
maximizes or minimizes a given linear
form.
1.7.1 The transportation problem
MP is commonly used to formulate and solve optimization prob-
lems. A typical, classroom example is provided by the transportation
18 mathematical programming

problem22 [Munkres, 1957]. 22

The transportation problem is
one of the fundamental problems in
MP. It turns up time and again in
Given a set P of production facilities with production capacities
practical settings. It is at the basis of
ai for i ∈ P, a set Q of customer sites with demands b j for j ∈ Q, the important business concern of
and knowing that the unit transportation cost from facility i ∈ P to assignment: who works on which task?
customer j ∈ Q is cij , find the optimal transportation plan.

If this is the first optimization problem you have ever come a1

c1 c11
across, you are likely to find it meaningless. Since the only costs 2
b1
mentioned by the definition are transportation costs, we understand c 21
that the optimization must affect the total transportation cost. a2
c22 c 31
However, what exactly is meant by a “transportation plan”?
b2
The formalization of a problem description is really an art,
honed with experience and by knowing how to recognize patterns c 32
a3
and structures within an informal text description of the problem.
The first question one needs to ask when confronted with an infor- Figure 1.8: Visualize the setting of
the transportation problem using a
mal description of a problem is what are the decision variables?, bipartite graph, facilities on the left,
or, in other words, what is being decided? There is rarely a unique customers on the right; arcs are the
correct answer,23 but the answer will influence the type of formula- potential transportation possibilities.
23
Sometimes there is none; this
tion obtained, and the efficiency of the solver. Experience will point often happens when consulting with
the most likely correct direction. industry — if they had a formal
In the transportation problem we are given the costs of transport- description of the problem, or one
which is close to being formal, they
ing a unit of material from some entity (facility) to another (cus- would not be needing our services!
tomer). This appears to imply that, in order to minimize the total
cost, we shall need to decide on some quantities. Which quantities?
We know that each facility cannot exceed its production capacity,
and we know that each customer must receive its demanded sup-
ply. We should therefore decide quantities out of each production
facility and into each customer site. One possibility24 is to consider 24
MP beginners usually come up with
decision variables xij for each facility i ∈ P and customer j ∈ Q, xij the following set of decision variables:
∀i ∈ P, yi is the quantity of commodity
which denote the quantity of commodity shipped from i to j. The shipping from i; ∀ j ∈ Q, z j is the
total cost is the sum of the unit costs multiplied by quantities: quantity of commodity shipping to
j. Intuition fails them when trying
to express the transportation costs in
min ∑ cij xij . terms of yi and z j .
i∈ P
j∈ Q

The capacity constraints dictate that the amount of commodity

shipped out of each facility cannot exceed its capacity:

∀i ∈ P ∑ xij ≤ ai .
j∈ Q

The demand constraints make sure that each customer receives at

least its desired demand:
Figure 1.9: Student’s view of the
transportation problem (twiki.esc.
∀j ∈ Q ∑ xij ≥ bj . auckland.ac.nz).
i∈ P

What else? Well, suppose you have two facilities with capacities
a1 = a2 = 2 and one customer with demand b1 = 1. Suppose that
the transportation costs are c11 = 1 and c21 = 2 (see Fig. 1.10). Then
 imperative and declarative programming 19

the problem reads:

min x11 + 2x21

x11 ≤ 2
x21 ≤ 2
2.0
x11 + x21 ≥ 1. 1.0
1.0
The optimal solution is x11 = 2, x21 = −1: it satisfies the 2.0
2.0
constraints and has zero total cost.25 This solution, however, cannot
be realistically implemented: what does it mean for a shipped
Figure 1.10: A very simple transporta-
quantity to be negative (Fig. 1.11)? tion instance.
This example makes two points. One is that we had forgotten a 25
Every solution on the line x11 + x21 =
1 satisfies the constraints; in particular,
constraint, namely: because of the costs, the solution
x11 = 1 and x21 = 0 has unit cost.
∀i ∈ P, j ∈ Q xij ≥ 0.

The other point is that forgetting a constraint might yield a solu-

tion which appears impossible to those who know the problem setting.
If we had been given the problem in its abstract setting, without
specifying that xij are supposed to mean a shipped quantity, we
would have had no way to decide that x21 = −1 is inadmissible.

1.7.2 Network flow

In the optimization of industrial processes, the network flow prob-
lem26 [Ford and Fulkerson, 1956] is as important as the transporta-
tion problem.
Figure 1.11: Negative shipments look
Consider a rail network connecting two cities s, t by way of a number weird (artcritique.wordpress.com).
of intermediate cities, where each link of the network has a number 26
Network flow is another fundamen-
assigned to it representing its capacity. Assuming a steady state tal MP problem. Its constraints can be
condition, find a maximal flow from city s to city t. used as part of larger formulations in
order to ensure network connectivity.
Note that the network flow problem is also about transportation.
• We model the rail network by a directed graph27 G = (V, A), 27
A directed graph, or digraph, is a
and remark that s, t are two distinct nodes in V. pair (V, A) where V is any set and
A ⊆ V × V. We shall assume V is finite.
• For each arc (u, v) ∈ A we let Cuv denote the capacity of the arc. Elements of V are called nodes and
elements of A arcs.
The commodity being transported over the network (people,
freight or whatever else) starts from s; we assume this commodity
can be split, so it is routed towards t along a variety of paths. We
want to find the paths which will maximize the total flow rate.28 28
Assuming a constant flow, this is
We have to decide which paths to use, and what quantities of equivalent to maximizing the total
quantity.
commodity to route along the paths. Since paths are sequences of
arcs, and the material could be split at a node and follow different
paths thereafter, we define decision variables xuv indicating the flow
rate along each arc (u, v) ∈ A.
A solution is feasible if the flow rates are non-negative (Fig. 1.12):

∀(u, v) ∈ A xuv ≥ 0,

if the flow rates do not exceed the arc capacities:

∀(u, v) ∈ A xuv ≤ Cuv , Figure 1.12: A negative flow coupler

from starships of the rebellion era
(starwars.wikia.com).
20 mathematical programming

and if the positive flow rates along the arcs define a set of paths
from s to t.
How do we express this constraint? First, since the flow orig-
inates from the source node s, we require that no flow actually
arrives at s:
∑ xus = 0. (1.7)
(u,s)∈ A Figure 1.13: A Max Flow instance.

Next, no benefit29 would derive from flow leaving the target node t: 29
Eq. (1.8) follows from (1.7) and (1.9)
(prove it), and so it is redundant. If
you use these constraints within a
∑ xtv = 0. (1.8) MILP, nonconvex NLP or MINLP,
(t,v)∈ A however, they might help certain
solvers perform better.
Lastly, the total flow entering intermediate nodes on paths between
s and t must be equal to the total flow exiting them:30 30
These constraints are called flow
conservation equations or material
∀v ∈ V r {s, t} ∑ xuv = ∑ xvu . (1.9) balance constraints.
(u,v)∈ A (v,u)∈ A

The objective function aims at maximizing the amount of flow

from s to t. Since flow is conserved at any node but s, t, it follows
that however much flow exits s must enter t, and hence that it
suffices to maximize the amount of flow exiting s:31

max ∑ xsv . Figure 1.14: An optimal solution for

the instance in Fig. 1.13.
(s,v)∈ A
31
We could equivalently have maxi-
mized the amount of flow entering t
1.7.3 Extremely short summary of complexity theory (why?).

Formally, decision and optimization problems are defined as sets

of infinitely many instances (input data) of increasing size. This
is in view of the asymptotic worst-case complexity analysis of
algorithms, which says that an algorithm A is O( f (n)) (for some
function f ) when there is an n0 ∈ N such that for all n > n0 the
CPU time taken by A to terminate on inputs requiring n bits of
storage is bounded above by f (n).
If a problem had finitely many (say m) instances, we could pre-
compute the solutions to these m instances and store them into a
hash table, then solve any instance of this problem in constant time,
making the problem trivial.
We say a decision problem is in P when there exists a polynomial-
time algorithm that solves it, and in NP if every YES instance has a
solution which can be verified to be YES in polynomial time.
Problems are NP-hard when they are as hard as the hardest prob-
lems in NP: NP-hardness of a problem Q is proved by reducing
(i.e. transforming) a known NP-hard problem P to Q in polynomial
time, such that each YES instance in P maps to a YES instance in Q
and every NO instance in P to a NO instance in Q.
This method works since, if Q were any easier than P, we could
solve P by reducing it to Q, solve Q, then map the solution of Q
back to P; hence P would be “as easy as Q” (under polynomial-time
Figure 1.15: Hardness/completeness
reductions), contradicting the supposition that Q is easier than P. reductions.
These notions are informally applied to optimization problems
by referring to their decision version: given an optimization problem
 imperative and declarative programming 21

Find the minimum of f ( x ) subject to g( x ) ≤ 0,

its decision version is

given a number F, is it the case that there exists a solution x such that
f ( x ) ≤ F and g( x ) ≤ 0?

Barring unboundedness, solving the decision version in polytime

allows the (at least approximate) solution of the optimization
problem in polytime too via the bisection method, which takes a
Figure 1.16: The bisection method
logarithmic number of steps in terms of the range where f varies, at (from codeproject.com).
least if this range is discrete.

1.7.4 Definitions

A mathematical program (MP) consists of parameters, decision

variables, objective functions and constraints.

• The parameters encode the input (or instance): their values are
fixed before the solver is called. The decision variables encode
the output (or solution): their values are fixed by the solver, at
termination.

• Decision variables are usually specified together with simple

constraints on their domain, such as e.g. interval ranges of the
form
x ∈ [ x L , x U ],

where x is a vector of n decision variables and x L , xU ∈ Rn such

that x jL ≤ xU
j for each j ≤ n, or integrality constraints of the form

∀j ∈ Z x j ∈ Z,

where Z is a subset of the decision variable indices {1, . . . , n}.

• Objective functions have one of the following forms:

min f ( p, x )
max f ( p, x ),

where p is a vector of q parameters, x is a vector of n decision

variables, and f is a function mapping Rq+n → R. A MP can
have zero or more objective functions. Usually, it has one.

• Constraints have one of the following forms:

g( p, x ) ≤ 0
g( p, x ) = 0
g( p, x ) ≥ 0,

where, again, g : Rq+n → R. A MP can have zero or more

constraints.
22 mathematical programming

For most MP subclasses, the functions f , g in the objectives and

constraints are explicitly given32 as mathematical expressions recur- 32
When f , g are not given explicitly we
have Black-box optimization.
sively built up of parameter symbols and decision variable symbols,
combined by means of the usual arithmetical operators and a few
transcendental operators such as logarithms, exponentials and (very
rarely) some trigonometric functions.

1.7.5 The canonical MP formulation

Most MP formulations have the following form:


minx f (x) 


∀i ≤ m gi ( x ) ≤ 0 
(1.10)
x ∈ [ x L , xU ] 

∈ Z.

∀j ∈ Z xj


Remarks: Figure 1.17: Parsing a recursively built

mathematical expression yields an
1. we do not explicitly mention the parameters in the function expression tree (from maplesoft.com).
forms;

2. most MPs in the mathematical literature have exactly one objec-

tive;33 33
The study of MPs with multiple
objectives is known as multi-objective
3. objectives in maximization form max f can be transformed to programming (MOP), see Fig. 1.18 and
minimization form by simply considering the problem min − f Sect. 2.11.

then taking the negative of the optimal objective function value,

since:
max f = − min(− f );

4. most MPs in the mathematical literature have a finite number of

constraints;34
5. constraints in the form g( x ) ≥ 0 can be replaced by − g( x ) ≤ 0,
and those in the form g( x ) = 0 by pairs of constraints ( g( x ) ≤
0, g( x ) ≥ 0).
Figure 1.18: MOP yields a Pareto set of
Let P be a MP formulation such as Eq. (1.10). We let: solutions (from www.noesissolutions.
com).

F ( P ) = { x ∈ [ x L , x U ] | ∀ i ≤ m gi ( x ) ≤ 0 ∧ ∀ j ∈ Z x j ∈ Z } The study of MPs with infinitely

34
(1.11)
many constraints is known as semi-
infinite optimization, see Sect. 2.13.
be the feasible set of P. Any x ∈ F ( P) is a feasible solution. A feasible
solution x 0 is a local optimum35 if there is a neighbourhood B of x 0 35
There are other definitions of local
such that B ⊆ F ( P) and f ( x 0 ) is minimal in B. A locally optimal optimality, involving e.g. the deriva-
tives of f at x 0 (but for them to hold, f
solution x ∗ is a global optimum if B = F ( P). We also let G( P) be the must obviously be differentiable).
set of global optima of P, and L( P) be the set of local optima of P.

1.8 Modelling software

If the solvers are the interpreters for MP, the modelling software
provides a mapping between optimization problems, formulated in
abstract terms, and their instances, which is the actual input to the
solver. Figure 1.19: Gradient-based minimality
conditions (from www.sce.carleton.
ca).
 imperative and declarative programming 23

1.8.1 Structured and flat formulations

Compare the transportation problem formulation given above:

∑ cij xij

min 
i∈ P


j∈ Q



∀i ∈ P ∑ xij ≤

ai 
j∈ Q (1.12)
∑ xij

∀j ∈ Q ≥ bj 


i∈ P



≥

x 0,

given in general terms, to its instantiation


min x11 + 2x21 

x11 ≤

2 

x21 ≤ 2 (1.13)

x11 + x21 ≥ 1 


x11 , x21 ≥

0,

which refers to a given instance of the problem (see Sect. 1.7.3). The
two formulations are different: the former is the generalization
which describes all possible instances of the transportation prob-
lem,36 and the latter corresponds to the specific instance where 36
An instance of the transportation
P = {1, 2}, Q = {1}, a = (2, 2), b = (1), c = (1, 2). The former problem is an assignment of values to
the parameter symbols a, b, c indexed
cannot be solved by any software, unless P, Q, a, b, c are initialized over sets P, Q of varying size.
to values; the latter, on the other hand, can be solved numerically
by an appropriate solver.
Formulation (1.12) is as a structured formulation, whereas (1.13)
is a flat formulation (see Fig. 1.20).

Figure 1.20: A flat linear MP can be

1.8.2 AMPL represented by the sparsity pattern of
the problem matrix (objective in top
row, and RHS in rightmost column).
The syntax of the AMPL [Fourer and Gay, 2002] modelling soft- Here is a random transportation
problem instance.
ware is very close to a structured formulation such as (1.12), and
which interfaces with many solvers, some of them are free and/or
open source. AMPL itself, however, is a commercial product. It has
a free “demo” version which can deal with instances of up to 300
decision variables and 300 constraints.
The transportation problem can be formulated in AMPL as
follows.37 37
Note how close AMPL is to a
structured form. On the other hand,
AMPL can also take flat forms: see the
# transportation.mod encoding of Formulation (1.13) below.
param Pmax integer;
var x11 >= 0;
param Qmax integer; var x21 >= 0;
set P := 1..Pmax; minimize cost: x11 + x21;
subject to production1: x11 <= 2;
set Q := 1..Qmax; subject to production2: x21 <= 2;
param a{P}; subject to demand1: x11 + x21 >= 1;

param b{Q};
The point is that AMPL can automat-
param c{P,Q}; ically transform a structured form to a
var x{P,Q} >= 0; flat form.
minimize cost: sum{i in P, j in Q} c[i,j]*x[i,j];
subject to production{i in P}: sum{j in Q} x[i,j] <= a[i];
subject to demand{j in Q}: sum{i in P} x[i,j] >= b[j];
24 mathematical programming

The various entities of a MP (sets, parameters, decision variables,

objective functions, constraints) are introduced by a corresponding
keyword. Every entity is named.38 Every entity can be quantified 38
This is obvious for sets, parameters
over sets, including sets themselves. The cartesian product P × Q and variables, but not for objective
functions and constraints.
is written {P,Q}. AMPL formulations can be saved in a file with
extension .mod.39 Note that comments are introduced by the # 39
E.g. transportation.mod.
symbol at the beginning of the line.
The instance is introduced in a file with extension .dat.40 40
E.g. transportation.dat

# transportation.dat
param Pmax := 2; AMPL .dat files have a slightly differ-
param Qmax := 1; ent syntax with respect to .mod or .run
files. For example, you would define a
param a :=
set using
1 2.0
2 2.0 set := { 1, 2 };

; in .mod and .run files, but using

param b := set := 1 2;
1 1.0 in .dat files. Also, there are ways to
; bi-dimensional arrays in tabular forms
param c := and transposed tabular forms. Multi-
dimensional arrays can be written as
1 1 1.0
many bi-dimensional array slices. For
2 1 2.0 simplicity, I suggest you stick with the
; basic syntax.

Every param line introduces one or more parameters as func-

tions41 from their index sets to the values. 41
The parameter tensor pi1 ,...,ik , where
In order to actually solve the problem numerically, another file P is the set of tuples (i1 , . . . , ik ) is
(bearing the extension .run) is used.42 encoded as the function p : P →
ran( p). Each pair (i1 , . . . , ik ) 7→ pi1 ,...,ik
# transportation.run is given as a line i1 i2 ... ik
p_value.
model transportation.mod; 42
The AMPl .run file functionality can
data transportation.dat;
be much richer than this example, and
option solver cplexamp; implement nontrivial algorithms. The
solve; imperative part of the AMPL language
display x, cost; is however severely limited by the lack
of a function call mechanism.
The transportation.run file is executed in a command line en-
vironment by running43 ampl < transportation.run. The output 43
AMPL actually has an Integrated
should look as follows. Development Environment (IDE). I
prefer using the command line since
CPLEX 12.6.2.0: optimal solution; objective 1 it allows me to pipe each command
1 dual simplex iterations (0 in phase I) into the next. For example, in Unix
(including Linux and MacOSX) you
x :=
can ask for the optimal value by
1 1 1 running
2 1 0
ampl < transportation.run
;
| grep optimal
(all on one line).
cost = 1

which corresponds to x11 = 1, x21 = 0 having objective function

value 1.

1.8.3 Python
Python is currently very popular with programmers because of the
large amounts of available libraries: even complex tasks require
very little coding.

Figure 1.21: The Python language

namesake.
 imperative and declarative programming 25

Python is a high-level,44 general purpose, imperative interpreted

programming language. We assume at least a cursory knowledge of 44
A programming language is high-
Python. level when it is humanly readable (the
The transportation problem can be formulated in Python as opposite, low-level, would be said of
machine code or assembly language).
follows. First, we import the PyOMO [Hart et al., 2012] libraries: The opposite of “interpreted” is
“compiled”: in interpreted languages,
# define concrete transportation model in pyomo and solve it
the translation to machine code occurs
from pyomo.environ import * line by line, and the corresponding
from pyomo.opt import SolverStatus, TerminationCondition machine code chunks are executed
import pyomo.opt immediately after translation. In
compiled languages, the program is
Next, we define a function which creates an instance of the completely translated to an executable
transportation problem, over the sets P, Q and the parameters a, b, c file prior to running.
(see Eq. (1.12)).45 PyOMO also offers an “abstract”
45

modelling framework, through which

def create_transportation_model(P = [], Q = [], a = {}, b = {}, c = {}):
you can actually read the instance
model = ConcreteModel() directly from AMPL .dat files.
model.x = Var(P, Q, within = NonNegativeReals)
def objective_function(model):
return sum(c[i,j] * model.x[i,j] for i in P for j in Q)
model.obj = Objective(rule = objective_function)
def production_constraint(model, i):
return sum(model.x[i,j] for j in Q) <= a[i]
model.production = Constraint(P, rule = production_constraint)
def demand_constraint(model, j):
return sum(model.x[i,j] for i in P) >= b[j]
model.demand = Constraint(Q, rule = demand_constraint)
return model

Finally, we write the “main” procedure,46 i.e. the point of entry

to the execution of the program.
# main
46
What you do with the solution once
you extract it from PyOMO using
P = [1,2]
solutions.load_from is up to you
Q = [1] (and Python). Since Python offers
a = {1:2,2:2} considerably more language flexibility
b = {1:1} and functionality than the imperative
c = {(1,1):1, (2,1):2} part of AMPL, the Python interface is
to be preferred whenever solving MPs
instance = create_transportation_model(P,Q,a,b,c)
is just a step of a more complicated
solver = pyomo.opt.SolverFactory(’cplexamp’, solver_io = ’nl’) algorithm.
if solver is None:
raise RuntimeError(’solver not found’)
solver.options[’timelimit’] = 60
results = solver.solve(instance, keepfiles = False, tee = True)
if ((results.solver.status == SolverStatus.ok) and
(results.solver.termination_condition == TerminationCondition.optimal)):
# feasible and optimal
instance.solutions.load_from(results)
xstar = [instance.x[i,j].value for i in P for j in Q]
print(’solved to optimality’)
print(xstar)
else:
print(’not solved to optimality’)

Note that these three pieces of code have to be saved, in se-

quence, in a file named transportation.py, which can be executed
by running python transportation.py. The output is as follows.

timelimit=60
26 mathematical programming

CPLEX 12.6.2.0: optimal solution; objective 1

1 dual simplex iterations (0 in phase I) Are you wondering how I produced
Fig. 1.20? Take transportation.py and
solved to optimality modify it as follows: (i) include
[1.0, 0.0] import numpy as np
import itertools
import np.random as npr

import matplotlib.pyplot as plt

1.9 Summary at the beginning of the code. Then
replace the definition of a, b, c after #
• Most people are used to imperative programming main to
U=10; p=10; q=6
P=range(1,p+1); Q=range(1,q+1)
• There is also declarative programming, and it is just as powerful avals=npr.random_integers(1,U,size=p)
bvals=npr.random_integers(1,U,size=q)
• Mathematical programming is a kind of declarative program- cvals=npr.random_integers(1,U,size=p*q)
a=dict(zip(P,avals))
ming for optimization problems b=dict(zip(Q,bvals))
c=dict(zip(list(itertools.product(P,Q)),cvals))
• Using parameters, decision variables, objectives and constraints, M=np.zeros((1+p+q, p*q+1))
for i in P:
optimization problems can be “modelled” using MP for j in Q:
M[0,q*(i-1)+(j-1)]=c[(i,j)]
• Efficient solution algorithms exist for different classes of MP for i in P:
for j in Q:
M[i,q*(i-1)+(j-1)]=1
• Algorithms are implemented into solvers M[i,p*q] = a[i]
for j in Q:
• MP interpreted as “modelling, then calling an existing solver” for i in P:
M[p+j,p*(j-1)+(i-1)]=1
shifts focus from creating a solution algorithm for a given problem M[p+j,p*q] = b[j]
plt.matshow(M, cmap = ’bone_r’)
(perceived as more difficult) to modelling the problem using MP
plt.show()
(perceived as easier)
Essentially, I’m sampling random in-
• MP formulations are created by humans using quantifiers, sets, teger dictionaries for a, b, c (note the
use of zip: look it up, it’s an important
and tensors/arrays of variables and constraints (structured form); function in Python), then I’m construct-
solvers require their input to be MP formulation as explicit ing the problem matrix M consisting
in the objective function coefficients
sequences of constants, variables, objectives, constraints (flat (top row), the constraint matrix, and
form) the right hand side (RHS) vector in the
rightmost column.
• Modelling software is used to turn structured form to flat form,
submit the flat form to a solver, then retrieve the solution from
the solver and structure it back to the original form

• We look at two modelling environments: AMPL and Python.

2
Systematics and solution methods

MP formulations can belong to different classes. We list some

of the most important in this chapter, together with a few notes
about solution methodologies, software packages, and applications.
Although all objective function directions have been cast to mini-
mization, this bears no loss of generality. As already remarked in
Sect. 1.7.5, max f can be replaced by − min(− f ). Figure 2.1: This picture, showing
a polyhedron (for LP), a nonlinear
function (for NLP) and an integer
2.1 Linear programming lattice (for MILP), is the logo of
another modeling environment for MP,
namely JuliaOpt (www.juliaopt.org).
Linear Programming (LP) [Dantzig et al., 1955] is the minimiza- You are very welcome to try it!
tion of a linear form on a polyhedron. The standard form is1 1
LP can also be formulated in canonical
 form as min{c> x | Ax ≤ b}. The
min c> x 
 associated feasibility problem Ax ≤ b
was first tackled by Fourier in 1827.
Ax = b (2.1) Kantorovich and Koopmans won the

x ≥ 0,  Nobel prize in Economics in 1975 for
their seminal work on formulating
where c ∈ Rn , b ∈ Rm , A an m × n matrix are the parameters. LP is and solving economics problems
as LPs. Hitchcock formulated the
the most fundamental problem in MP.
transportation problem (see Sect. 1.7.1)
LP is arguably the most fundamental problem in MP. It can be as an LP. Dantzig formulated many
solved in polynomial time by the ellipsoid algorithm [Khachiyan, operation planning problems as LPs,
and in 1947 he proposed the celebrated
1980] and by the interior point algorithm [Karmarkar, 1984], and is simplex method for solving LPs.
therefore in the computational complexity class2 P. By continually 2
Its associated decision problem
improving and tuning the simplex and interior point methods, turns out to be P-complete under
log-space reductions, a useful fact in
modern solver technology3 can efficiently solve (sparse) LPs with the complexity analysis of parallel
millions of nonzeros in their constraint matrix. algorithms.
LP often occurs as a subproblem of higher-level algorithms. In
3
See e.g. IBM-ILOG CPLEX [IBM,
2010] and GLPK [Makhorin, 2003] for
the Branch-and-Bound (BB) algorithm for solving MILPs, for exam- the state of the art in commercial and
ple, every node of the BB search tree either gives rise to subnodes, open-source LP solvers.
or is pruned, according to the feasibility or integrality of the LP
relaxation of the MILP.
Classical applications of LP are: modelling points of equilibrium
in economic behaviour, determining optimal blending recipes for
composite food or chemical products, the already mentioned trans-
portation, assignment, flow problems, planning production over
time, finding sparse solutions to under-determined linear systems
[Candès, 2014] and approximate solutions to over-determined lin-
ear systems, finding upper bounds to sphere packing problems,
and many more. Several references exist on LP; [Vanderbei, 2001,
28 mathematical programming

Matoušek and Gärtner, 2007] are among my favorites; the latter, in

particular, contains a very interesting chapter on the application of
LP to upper bounds for error correcting codes and sphere packing
problems [Delsarte, 1972].

2.2 Mixed-integer linear programming

Mixed-Integer Linear Programming4 (MILP) consists in the 4

The first exact method for solving
minimization of a linear form on a polyhedron, restricted to a non- MILPs was proposed by Ralph Go-
mory [Gomory, 1958]. After joining
negative integer lattice cartesian product the non-negative orthant. IBM Research, he exploited integer pro-
The standard form is gramming to formulate some cutting
problem for a client, and derived much
of the existing theory behind cutting
min c> x



 planes from the empirical observation
Ax = b 
of apparently periodic data in endless
(2.2) tabulations of computational results.
x ≥ 0 

x j ∈ Z+ ,

∀j ∈ Z


where c, b, A are as in the LP case, and Z ⊆ {1, . . . , n}.

This is another fundamental problem in MP. It is NP-hard5 ; 5
A problem P is NP-hard when any
many problems can be cast as MILPs — its combination of contin- other problem in the class NP can
be reduced to it in polynomial time:
uous and integer variables give MILP a considerable expressive from this follows the statement that
power as a modelling language. It includes ILP (integer variables an NP-hard problem must be at least
as hard to solve (asymptotically, in the
only, see below), BLP (binary variables only, see below) and LP as worst case) as the hardest problem of
special cases. MILPs can either be solved heuristically [Fischetti and the class NP (see Sect. 1.7.3).
Lodi, 2005, Fischetti et al., 2005] or exactly, by BB6 6
The state of the art BB implementa-
Here is an example of a problem which can be formulated as a tion for MILP is given by IBM-ILOG
CPLEX [IBM, 2010]; the SCIP solver
MILP. [Berthold et al., 2012], also very ad-
vanced, is free for academics. Other
∗ advanced commercial solvers are
Power Generation Problem. There are Kmax types of power
XPress-MP and GuRoBi.
generation facilities, and, for each k ≤ Kmax , nk facilities of type
k, each capable of producing wk megawatts. Switching on or off a

∗ types of power plants

facility of type k ≤ Kmax has a fixed cost f k and a unit time cost
ck . This set of power generation facilities has to meet demands
dt for each time period t ≤ Tmax , where Tmax is the time horizon.
The unmet demand has to be purchased externally, at a cost C
per megawatt. Formulate a minimum cost plan for operating the
facilities.

Some of the decision variables7 are defined as the number of fa- 7

Try and formulate this problem as
cilities of a certain type used at a given period (these are integer a MILP — use variables xkt ∈ Z+ to
mean the number of facilities of type
variables). And yet other decision variables will be the quantity of k generating at period t, y1kt ∈ Z+ to
power which needs to be purchased externally (these are continu- mean the number of facilities of type
k that are switched on at time t, y0kt
ous variables). Every relation between variables is linear, as well as switched off, and zt ∈ R+ to mean the
the expression for the costs. quantity of power purchased externally
Another example is the following problem, which has its roots in at period t.

[Euler, 1736] (see Fig. 2.2).

Traveling Salesman Problem8 (TSP) [Flood, 1956]. Consider a 8

There is an enormous amount of
traveling salesman who has to visit n + 1 cities 0, 1, . . . , n spending the literature about the TSP — it appears
minimum amount of money on car fuel. Consumption is assumed to be the most studied problem in
combinatorial optimization! Entire
to be proportional to inter-city distances cij for each pair of cities books have been written about it
i < j ≤ n. Determine the optimal order of the cities to be visited. [Applegate et al., 2007, Gutin and
Punnen, 2002].
 systematics and solution methods 29

We define binary decision variables xij ∈ {0, 1} to mean that the

order visits city i immediately before city j if and only if xij = 1.
We also introduce continuous decision variables u0 , . . . , un ∈ R.
Consider the following MILP formulation:
n
min ∑ cij xij (2.3)
x,u
i6= j
n
∀i ≤ n ∑ xij = 1 (2.4)
j =0
n
∀j ≤ n ∑ xij = 1 (2.5)
i =0
∀i 6= j 6= 0 ui − u j + nxij ≤ n−1 (2.6)
∀i, j ≤ n xij ∈ {0, 1}. (2.7)

The objective function Eq. (2.3) aims at minimizing the total cost
Figure 2.2: Euler, and TSP art.
of the selected legs of the traveling salesman tour. By Eq. (2.4)-
(2.5),9 we know that the feasible region consists of permutations of 9
Constraints (2.4)-(2.5) define the
{0, . . . , n}: if, by contradiction, there were two integers j, ` such that assignment constraints, which define
an incredibly important substructure
xij = xi` = 1, then this would violate Eq. (2.4); and, conversely, if in MP formulation — these crop up
there were two integers i, ` such that xij = x` j = 1, then this would in scheduling, logistics, resource
allocation, and many other types of
violate Eq. (2.5). Therefore all ordered pairs (i, j) with xij = 1 define problems. Assignment constraints
a bijection {0, . . . , n} → {0, . . . , n}, in other words a permutation. define a bijection on the set of their
indices.
Permutations can be decomposed in products of disjoint cycles.
This, however, would not yield a tour but many subtours. We have
to show that having more than one tour would violate Eq. (2.6).
Suppose, to get a contradiction, that there are at least two tours.
Then one cannot contain city 0 (since the tours have to be disjoint
by definition of bijection): suppose this is the tour i1 , . . . , ih . Then
from Eq. (2.6), by setting the x variables to one along the relevant
legs, for each ` < h we obtain ui` − ui`+1 + n ≤ n − 1 as well
as uih − ui1 + n ≤ n − 1. Now we sum all these inequalities and
observe that all of the u variables cancel out, since they all occur
with changed sign in exactly two inequalities. Thus we obtain
n ≤ n − 1, a contradiction.10 Therefore the above is a valid MILP 10
At least one tour (namely the one
formulation for the TSP. containing city 0) is safe, since we
quantify Eq. (2.6) over i, j both non-
Eq. (2.6) is not the only possible way to eliminate the subtours. zero.
Consider for example the exponentially large family of inequalities

∀∅ 6= S ( {0, . . . , n} ∑ xij ≥ 1. (2.8)

i ∈S
j6∈S

These subtour elimination inequalities state that for every nontrivial

subset S of the cities, there must be a leg in the tour which exits
S. Since it would be too time-consuming to generate the whole
family of constraints (2.8), one usually proceeds iteratively: solve
the formulation, and hope the resulting solution is a single tour. If
not, identify a constraint which is violated by the current subtour
solution, add it to the formulation, and repeat. This algorithm is
called row generation.11 11
Note, however, that Eq. (2.8) are
only defined on the binary variables x
rather than the continuous ones u; so
the formulation actually belongs to the
BLP class (see Sect. 2.4).
30 mathematical programming

2.3 Integer linear programming

Integer Linear Programming (ILP) consists in the minimization

of a linear form on a polyhedron, restricted to the non-negative
integer lattice. The standard form is

min c> x 

Ax = b (2.9)
∈ Zn+ , 

x

where c, b, A are as in the LP case. ILP is just like MILP but without
continuous variables. Consider the following problem.

Scheduling. There are n tasks to be executed on m processors. Task

Figure 2.3: In scheduling, solutions
i ≤ n takes τi units of processor time, where τ ∈ [0, T ]n and T ∈ N are often represented on a plane, with
is a time horizon. There is a precedence relation digraph G = (V, A) processors on a discretized vertical
where V = {1, . . . , n} and (i, j) ∈ A if the execution of task i must axis, and time on the horizontal axis
end before the execution of task j begins. Determine the execution (from www.femto-st.fr).
plan which ensures that all tasks are executed exactly once, and
which minimizes the total time.

Scheduling problems12 are a vast area of combinatorial optimiza- 12

Scheduling problems usually have
tion, business planning and, in general, of all practical life! We can two intertwined decisions: a temporal
(partial) order, and an assignment.
solve them using ILP by using the following decision variables: In this case the order is encoded by
xi ∈ Z+ indicates the starting time for task i ≤ n, and yik ∈ {0, 1} the < relation on the values of the
xi variables and the assignment is
for each i ≤ n, k ≤ m indicates that task i is executed by processor k encoded by the yik variables. The
if and only if yik = 1. optimal value of these variables must
depend on each other, since if two
tasks i, j are assigned to the same
processor, then necessarily one must
2.4 Binary linear programming end before the other can begin. Can
you write these constraints? You
Binary Linear Programming (BLP) consists in the minimization might need other binary variables to
determine a precedence for (i, j) 6∈ A.
of a linear form on a polyhedron, restricted to the hypercube. The
standard form is

min c> x 

Ax = b (2.10)
n

x ∈ {0, 1} , 

where c, b, A are as in the LP case. BLP is usually employed for

purely combinatorial problems.

2.4.1 Linear assignment

Find the assignment of minimum cost [Burkard et al., 2009]

(Fig. 2.4). Figure 2.4: Image recognition appli-
cation of linear assignment (from
Linear Assignment Problem (LAP)13 Given a set V = {1, . . . , n} pages.cs.wisc.edu/~pachauri).
and a cost matrix cij for 1 ≤ i, j ≤ n, find the permutation π of V
such that ∑i≤n ciπ (i) is minimum.
13
The LAP can be solved in poly-
nomial time: it is the same as the
weighted perfect matching problem
Since we are looking for a permutation, we can exploit some of the in a bipartite graph, which is known
constraints of the MILP formulation for the TSP, namely Eq. (2.4)- to be in P (also see Sect. 4.3.2 and its
sidenotes).
 systematics and solution methods 31

(2.5): 
min ∑ cij xij 

i,j≤n 


∀i ≤ n ∑ xij

= 1 

j≤n (2.11)
∀j ≤ n ∑ xij = 1



i ≤n




∀i, j ≤ n xij ∈ {0, 1}.


2.4.2 Vertex cover

Figure 2.5: Vertex cover: location
Find the smallest set of vertices such that ever edge is adjacent to a of communication towers (from
www.bluetronix.net).
vertex in the set14 (see Fig. 2.5). 14
In other words, the edge is covered by
Vertex Cover.15 Given a simple undirected graph G = (V, E), find the vertex.
15
This is an NP-hard problem [Karp,
the smallest subset S ⊆ V such that, for each {i, j} ∈ E, either i or j is 1972].
in S.

Since we have to decide a subset S ⊆ V, we can represent S by its

indicator vector16 x = ( x1 , . . . , xn ), where n = |V |, xi ∈ {0, 1} for all 16
In combinatorial problems, subsets
i ≤ n, and xi = 1 if and only if i ∈ S. are very often represented by indi-
cator vectors. This allows a natural
 formalization by MP.
min ∑ xi 

i≤n 
∀{i, j} ∈ E xi + x j ≥ 1 (2.12)


∀i ≤ n xi ∈ {0, 1}.


The generalization of Vertex Cover to hypergraphs17 is the 17

In a hypergraph, hyperedges are
arbitrary sets of vertices instead of just
Minimum Hypergraph Cover. Given a set V and a family E = being pairs of vertices.
{ E1 , . . . , Ek } of subsets of V, find the smallest subset S ⊆ V such that
every set in E has at least one element in S.

Generalizing from Vertex Cover, every edge e ∈ E has been

replaced by a set E` in the family E . We now have to generalize the
cover constraint xi + x j ≥ 1 to a set.

min ∑ xi 

i ≤n 

∀` ≤ k ∑ xi ≥ 1 (2.13)
i ∈ E` 


∀i ≤ n xi ∈ {0, 1}.

Figure 2.6: A minimum hypergraph
cover is useful to choose the smallest
numbers of optimally placed sites from
2.4.3 Set covering a set (from support.sas.com).

Given a family E of k subsets of V, we want to find the smallest

subset of E the union of which contains V (see Fig. 2.7 for an appli-
cation).
Set Covering. Given a set V and a family E = { E1 , . . . , Ek } of
subsets of V, find the smallest subset S ⊆ {1, . . . , k} s.t.
S
E` = V.
`∈S

The formulation is deceptively similar to Eq. (2.13):

 Figure 2.7: In airline crew rostering, V
min ∑ x` 
 are flight segments (or legs), E contains
`≤k 

 tours (or rosters), to which are assigned
∀v ∈ V ∑ x` ≥ 1 (covering constraint) (2.14) crews (since they eventually have to
`≤k  get back to where they started from) —
v ∈ E` 

 the optimal crew roster is given by a
∀` ≤ k x` ∈ {0, 1}.

minimum set cover.
32 mathematical programming

The covering constraints state that every v ∈ V must belong to at

least one set E` . Set Covering is NP-hard.

2.4.4 Set packing

In the same setting as Set Covering, we now want to find the
largest disjoint subset of E .

Set Packing. Given a set V and a family E = { E1 , . . . , Ek } of subsets

of V, find the largest subset S ⊆ {1, . . . , k} such that E` ∩ Eh = ∅ for
all ` 6= h in S.
Figure 2.8: What is the largest number
We simply invert objective function direction and inequality sign: of most diverse (E` ∩ Eh = ∅) recipes
 (from your cookbook E ) that you can
max ∑ x`  make with the ingredients V you have
at home?

`≤k 


∀v ∈ V ∑ x` ≤ 1 (packing constraint) (2.15)
`≤k 
v∈ E` 


∀` ≤ k x` ∈ {0, 1}.


The packing constraints state that every v ∈ V must belong to at

most one set E` . In general, Set Packing is NP-hard.
If V is the set of vertices of a graph, and E consists of the set
of its edges, then Set Packing is equal to the Matching problem,
i.e. find a subset M of non-incident edges such that each vertex is
adjacent to at most an edge in M. This particular version of the
problem can be solved in polytime [Edmonds, 1965].

2.4.5 Set partitioning

Putting covering and packing together, we get partitioning.

Set Partitioning. Given a set V and a family E = { E1 , . . . , Ek } of

subsets of V, find a set S ⊆ {1, . . . , k} such that (i)
S
E` = V and (ii)
`∈S Figure 2.9: Set partitioning is some-
∀` 6= h ∈ S ( E` ∩ Eh = ∅). times useful as a modelling tool in
clustering, specifically when one has
an exhaustive list of all the possible
This is a pure feasibility problem which can be formulated as
clusters as part of the input (from
follows. scikit-learn.sourceforge.net).

∀v ∈ V ∑ x` = 1 (partitioning constraint)  
`≤k
v ∈ E` (2.16)
∀` ≤ k ∈ {0, 1}.

x` 

The partitioning constraints state that every v ∈ V must belong

to exactly one set E` . Set Partitioning is NP-hard.
Eq. (2.16) becomes an optimization problem whenever we can
assign a cost c` to each set E` (` ≤ k), for example we could have
c` = | E` |. Other types of objective functions yield applications such
Figure 2.10: Political districting in
as that of Fig. 2.10. NY state (from andrewgelman.com).
Given a set E of potential (overlapping)
districts, choose the partition of the
2.4.6 Stables and cliques state which minimizes the maximum
deviation from the average.
A stable set in a graph G = (V, E) is a subset I ⊆ V such that
{u, v} 6∈ E for any pair u 6= v ∈ I.
 systematics and solution methods 33

Given a set X and a family E of k subsets of V, consider the

graph G having E as its vertex set, and such that { E` , Eh } is an edge
of G if and only if E` ∩ Eh 6= ∅. Then18 S is a set packing of ( X, E ) if 18
Prove it.
and only if S is a stable set of G .
The relevant problem is to find a stable set of maximum size:
Max Stable Set. Given a simple undirected graph G = (V, E), find a Figure 2.11: A clique and a stable set
in a graph.
stable set S ⊆ V of having maximum cardinality.

If we now look at the complement graph,19 a stable set becomes 19

This is the graph obtained by negat-
a clique20 The corresponding Max Clique problem is finding the ing each edge; i.e., if {i, j} ∈ E in
the given graph, {i, j} 6∈ E in the
clique S ⊆ V of G having maximum cardinality.21 Both Max Stable complement, and vice versa.
Set and Max Clique are NP-hard.22 20
A set of vertices pairwise linked by
edges.
21
Write the BLP formulations of Max
2.4.7 Other combinatorial optimization problems Stable Set and Max Clique.
22
Prove this.
There cannot be an exhaustive list of BLP formulations for all
combinatorial problems on graphs and hypergraphs. Those we
mentioned here are supposed to give some indication of the most
basic modelling techniques using BLP.
Figure 2.12: Let A1 , . . . , An ∈ Rm
be vectors representing a small set
2.5 Convex nonlinear programming of high-dimensional images from a
live feed (e.g. a security camera), and
convex Nonlinear Programming (cNLP) consists in the mini- b ∈ Rm be a vector representing a
target image (e.g. a thief). Can we
mization of a convex function on a convex set. The standard form is approximately express b as a linear
) combination of A1 , . . . , Am (in other
min f (x) words, can we detect the thief)? Since
(2.17) m > n, there will of course be an error
g( x ) ≤ 0, k Ax − bk2 , where A is the m × n matrix
consisting of the columns Ai (i ≤ n),
where f : Rn → R and g : Rn → Rm are convex functions.
and x ∈ Rn are the coefficients we
cNLPs can be solved efficiently using local NLP solvers such want to find. Depending on how small
as IPOPT or SNOPT, which are designed to find a local optimum we can make k Ax − bk2 , we might
convict the thief or let her walk. The
on (general, possibly nonconvex) NLPs (see Sect. 2.6 below). Since problem min k Ax − bk2 is a cNLP.
every local optimum of a cNLP is also global (by convexity of f , g),
local solvers find global optima in this setting.
cNLPs can in fact be solved in polynomial time. Strictly speak-
ing, however, we cannot state the cNLP is in P, since solutions
of cNLPs could be irrational and even transcendental — and our
computational model based on TMs does not allow us to write
all23 irrational and transcendental numbers precisely. So we have 23
Here’s an interesting question: how
to speak about ε error tolerances: With this caveat, we can state about being able to write just one
optimum precisely? E.g., given a
that we can find arbitrary close approximations of cNLP optima in problem instance, couldn’t we simply
polynomial time.24 denote the optimum of interest by a
simple symbol, e.g. ω?
Similarly to LP, which is used within other algorithms such as 24
In general, these results are implied
BB, cNLP is also used in the nonlinear version of the BB algorithm. by showing that some algorithm runs
Applicationwise, many problems can be cast as cNLPs (see [Boyd in O( p(| I |, 1ε )) where | I | is the size of
the instance and p(·, ·) is a polynomial.
and Vandenberghe, 2004]). For example, given a point p ∈ Rn and
a convex feasible set F = { x ∈ Rn | ∀i ≤ m gi ( x ) ≤ 0}, finding the
point y ∈ F closest to p is a cNLP:25 25
Linear regression generalizes pro-
) jection and is possibly one of the
min k x − pk` practically most useful (and used)
(2.18) tools in all of mathematics. It can be
∀i ≤ m gi ( x ) ≤ 0. cast as the following cNLP, which is
in fact a cQP (see Sect. 2.9.1 below):
min k Ax − bk2 , where A is an m × n
matrix, b ∈ Rm , and m > n.
34 mathematical programming

Since ` norms are convex functions and the constraints are con-
vex, Eq. (2.18) is a cNLP. It is routinely used with the Euclidean
norm (` = 2) to project points onto convex sets, such as cones and
subspaces.
Surprisingly, the problem of interpolating an arbitrary set of
points in the plane26 by means of a given class of functions can be 26
Generalize this to Rn .
cast as a cNLP. Let P be a finite set of k points ( pi , qi ) in R2 , and
consider the parametrized function:

f u ( x ) = u0 + u1 x + u2 x 2 + · · · + u m x m .

The problem is that of looking for the values u0 , . . . , um minimizing

the norm of the error vector F (u) = ( f u ( pi ) − qi | i ≤ k), e.g.:

min k F (u)k`
u Figure 2.13: Nonlinear function
interpolation [Boyd and Vandenberghe,
for some positive integer `. The point is that f u is nonlinear in x but 2004].
linear in u, which are the actual decision variables.

2.6 Nonlinear programming

Nonlinear Programming (NLP) consists in the minimization

of any nonlinear function on any nonlinearly defined set. The
standard form is )
min f ( x )
(2.19)
g( x ) ≤ 0,
where f : Rn → R and g : Rn → Rm are any nonlinear functions.
Adding nonconvex functions to NLP makes it much more expres-
sive than cNLP. For example, NLP can model circle packing [Costa
et al., 2013] (see Fig. 2.14).

Packing Equal Circles in a Square (PECS). Given an integer n,

find the largest radius r such that n circles of radius r can be drawn
in a unit square in such a way that they intersect pairwise in at most
one point.

The formulation27 is as follows:


max r 
 Figure 2.14: Some configurations
∀i < j ≤ n ( xi − x j )2 )2
+ ( yi − y j ≥ (2r )2

 for circle packing in a square (from
(2.20) www.packomania.com).
∀i ≤ n r ≤ xi ≤ 1−r  
 27
Write a formulation for the variant of
∀i ≤ n r ≤ yi ≤ 1 − r.

this problem where r is fixed and you
maximize n.
Solving circle packing problems is hard in practice (see the www.
packomania.com website). However, no-one ever established
whether the PECS is actually NP-hard.28 This is the case for many 28
It is unlikely that the PECS will
problems having small instance sizes, e.g. for the decision version actually be in NP, since a certificate of
optimality is a solution (( x j , y j ) | i ≤ n)
of the PECS, where r is given as part of the input instead of being which might have irrational compo-
maximized, an instance is the pair (r, n), and the number of bits nents — and it might be impossible
to represent them with a number of
required to store them is |n| + |r |, where bits which is at most polynomial in the
size of the input; which, in this case, is
p
∀y = ∈Q |y| = dlog2 pe + dlog2 qe the memory necessary to store the two
q scalars r, n.
 systematics and solution methods 35

is the number of bits of memory required to store the floating point

number y. By contrast, the size of an LP min{cx | Ax ≤ b} where A
m n m n
is m × n is ∑ ∑ | aij | + ∑ |bi | + ∑ |c j |. Even taking the storage for
i =1 j =1 i =1 j =1
components A, b, c to be as low as possible (1 bit), this amounts to
m + n + mn, against |n| + |r | which is of order log2 n.
The issue with having a problem Q with small instance size29 is 29
NP reductions have to be performed
that NP-hardness is usually established by reduction (see Sect. 1.7.3) in time bounded by a polynomial of
the instance size of P, and have map
from a problem P which is already known to be NP-hard. All YES (respectively, NO) instances of P
NP-hard problems we know have sizes at least proportional to an to YES (respectively, NO) instances of
Q. Inever had any promising intuition
integer n, rather than its logarithm. for encoding O(n) bits of information
A related decision problem is the following. in O(log2 n) in such a way as to
preserve YES and NO answers.
feasibility Kissing Number Problem (fKNP).30 Given two integers 30
This problem in K = 3 was originally
n, K, can we fit n unit balls B1 , . . . , Bn around the unit ball B0 cen- discussed between Isaac Newton and
tered at the origin of RK , so that | B0 ∩ Bi | = 1 for all i ≤ n but the one of his ex students, David Gregory
interiors of all the balls are pairwise disjoint?31 [Liberti, 2012].
31
Formulate the fKNP as a nonconvex
NLP.
Like the PECS, the fKNP is very hard in practice [Kucherenko et al.,
2007] but we do not know whether it is NP-hard or not, for much
the same reasons as the PECS.
Another problem which can be formulated as an NLP is the
following.

Euclidean Distance Geometry Problem (EDGP). Given a positive

integer K and a simple undirected graph G = (V, E) with a distance
function d : E → R+ on the edges, determine whether there is a
realization32 x : V → RK such that:

∀{u, v} ∈ E k xu − xv k2 = duv , (2.21) Figure 2.15: A 12-sphere packing: is

there space for a 13th ball?
where ∀v ∈ V xv ∈ RK . A realization is a function satisfying
32

Eq. (2.21).
If K = 2 the EDGP essentially asks to draw a given graph on the
plane such that edge segments have lengths consistent with the
edge function d. The EDGP has many applications in engineering
and other fields [Liberti et al., 2014].
Nonconvex NLPs may have many local optima which are not
global. There are therefore two categories of solvers which one
can use to solve NLPs: local solvers (for finding local optima) and
global solvers (for finding global optima).
We mentioned local NLP solvers (such as IPOPT or SNOPT)
in connection with solving cNLP globally in Sect. 2.5 above. Un-
Figure 2.16: Two applications of EDGP:
fortunately, any guarantee of global optimality disappears when localization of sensor networks, and
deploying such solvers on nonconvex NLPs. The actual situation is protein conformation from Nuclear
Magnetic Resonance (NMR) data.
even worse: most local NLP solvers are designed to simply finding
the closest local optimum from a given feasible point.
In general, local NLP algorithms are iterative, and identify a
sequence x k ∈ Rn (for k ∈ N) which starts from a given feasible
point x0 and by finding a feasible improving direction vector dk by
means of solving an auxiliary (but easier) optimization subproblem,
they then set
x k +1 = x k + d k .
36 mathematical programming

Termination is achieved whenever k x k+1 − x k k ≤ ε, where ε > 0 is a

given approximation error tolerance. The auxiliary subproblem usu-
ally aims at satisfying first, and possibly second order33 optimality 33
For example, Sequential Quadratic
conditions. Programming (SQP) is a class of
methods which is based on solving the
These conditions, however, are not valid everywhere: further as- following auxiliary subproblem:
sumptions called constraint qualifications must be met. Unfortunately, >

min f ( x k ) + [∇ x f ( x k )] d+ 
d ∈R

whether these assumptions are valid depends on the instance rather n
1 > k k

+ 2 d ∇ xx L( x , λ )d
than the problem. Moreover, from a worst-case complexity point of >
∀i ≤ m gi ( x k ) + [∇ x gi ( x k )] d ≤ 0



view, finding a feasible point in a nonconvex NLP is just as hard as
where L( x, λ) = f ( x ) + λg( x ) is the
finding an optimal point,34 so the assumption that a feasible point Lagrangian of the NLP, and λ is the
is given is also unrealistic.35 On the other hand, if all the assump- vector of Lagrange coefficients. The
solution to this problem yields dk at
tions are verified and a feasible point exists, local NLP methods can
each step k (λk are also updated at
find the closest local optimum rather efficiently. each step).
Nonconvex NLPs can be solved globally to a given ε > 0 toler- 34
Why?
ance (on the optimal objective function value) by means of a BB
35
These methods are often used
to improve existing business or
variant called spatial Branch-and-Bound (sBB) [Liberti, 2006, Belotti production processes, where a feasible
et al., 2009].36 The best-known sBB implementation is BARON solution can be derived from the
real-world setting.
[Sahinidis and Tawarmalani, 2005], which is a commercial solver.
36
BB type algorithms are used to
Recent versions of SCIP [Berthold et al., 2012] can also solve MINLP handle nonconvexity. In MILP, the inte-
(see Sect. 2.8) and hence, in principle, NLPs as well. The only open grality constraints are the only source
source sBB implementation worthy of note is Couenne [Belotti, of nonconvexity, and, accordingly, BB
branches on variables which take a
2011]. fractional value when solving the LP
relaxation at each node. In nonconvex
NLP, objective and constraints are
2.7 Convex mixed-integer nonlinear programming nonconvex. Instead of branching on
variables taking fractional values,
we branch on variables contributing
convex Mixed-Integer Nonlinear Programming (cMINLP)
a discrepancy between an objective
consists in the minimization of a convex function on a convex function value of the original problem
set, restricted to the cartesian product of an integer lattice and a and of the convex relaxation solved at
each sBB node.
Euclidean space. The standard form is

min f (x) 

g( x ) ≤ 0 (2.22)
∈ Z, 

∀j ∈ Z xj
where f : Rn → R and g : Rn → Rm are convex functions, and
Z ⊆ {1, . . . , n}.
A typical cMINLP application is mean-variance portfolio selec-
tion with cardinality constraints:
Figure 2.17: Optimization of water
Cardinality Constrained Mean-Variance Portfolio Selection networks: deciding diameters for all
(CCMVPS).37 Decide how to partition a unit budget into an invest- pipes and water flow given water
ment portfolio which maximizes the expected return (given the at each node over a time horizon
return coefficient vector ρ) subject to a given bound r on the variance [Bragalli et al., 2012]. Although this is
actually a nonconvex cMINLP methods
(given the covariance matrix Q) and to a maximum of k different
are used heuristically to solve it.
investments out of a possible K.38 37
This is a cardinality constrained
variant of the celebrated Markowitz’
Insofar as cMINLPs are a generalization of cNLP to integer portfolio selection problem [Konno
variables, all of the applications mentioned in Sect. 2.5 are relevant and Wijayanayake, 2001]. Markowitz
won the Nobel prize in Economics
to this section, too. For example, the integral point x ∗ ∈ Zn inside
for his contributions on the portfolio
a given convex set { x ∈ Rn | ∀i ≤ m gi ( x ) ≤ 0} which is closest problem.
(using euclidean distances) to a given point p ∈ Rn can be obtained 38
Formulate the CCPS as a cMINLP.
by solving a variant of Eq. (2.18) where x ∈ Zn and ` = 2.
 systematics and solution methods 37

Also, most nonconvex functions are convex in some subdomain.

This means that cMINLP algorithms can also be deployed on non-
convex MINLP (see Sect. 2.8 below) forsaking the global optimality
guarantee (i.e. they will act as heuristics and may find local op-
tima).
Currently, the best solver for cMINLP is the open-source BON-
MIN [Bonami and Lee, 2007].

2.8 Mixed-integer nonlinear programming

Mixed-Integer Nonlinear Programming (MINLP) consists in the

minimization of any nonlinear function on any nonlinear defined
set, restricted to an integer lattice cartesian product a Euclidean
space. The standard form is

min f (x) 

g( x ) ≤ 0 (2.23)
∈ Zn , 

∀j ∈ Z xj

where f : Rn → R and g : Rn → Rm are any nonlinear functions,

and Z ⊆ {1, . . . , n}.
MINLP is possibly the most general class of MPs. It contains
all of the other classes listed in this chapter with the exception
of SDP (see below). The kissing number problem (optimization
version) can be formulated as a MINLP [Maculan et al., 1996]. Most
industrial processes can be formulated as MINLP [Liberti, 2015]. Its Figure 2.18: The maximization for-
expressive power is remarkable: it can encode any TM, including mulation of the KNP [Maculan et al.,
1996].
universal ones, such as Minsky’s RM [Liberti and Marinelli, 2014],
which means that every problem can be formulated as a MINLP
(albeit one with an infinite number of variables and constraints),39 39
In fact we only require a finite
even unsolvable ones such as the halting problem. In other words, number of polynomial functions in
finitely many variables for a MINLP to
MINLP is Turing-complete40 as a programming language. Since MP be able to formulate any problem, see
contains MINLP as a sub-class, MP is also Turing-complete. Hilbert’s 10th problem [Matiyasevich,
1993].
MINLPs can be approximated to a given ε > 0 tolerance us- 40
A programming language is Turing-
ing the sBB algorithm (implemented in e.g. BARON, SCIP and complete when it can describe a UTM.
Couenne). See [D’Ambrosio and Lodi, 2011] for more details.

2.9 Quadratic programming formulations

Quadratic programming is the easiest type of nonlinear program-

ming. With respect to linear programming it allows a lot more
flexibility in expressive power whilst keeping a lot of structure,
which allows us to give stronger theoretical guarantees to formula-
tion properties and solution algorithms.

2.9.1 Convex quadratic programming

convex Quadratic Programming (cQP) consists in the minimiza-
tion of a semi-definite quadratic form subject to a polyhedron. The
38 mathematical programming

standard form is
)
min x > Qx + c> x
(2.24)
Ax ≤ b,

where c ∈ Rn , Q is an n × n semi-definite matrix, b ∈ Rm and A is

an m × n matrix.
One of the best known application in this class Markowitz’
mean-variance portfolio selection problem [Markowitz, 1952],
which we already discussed in its cardinality constrained variant
CCMVPS (see Sect. 2.7). Given a unit budget, decide the fractions Figure 2.19: Harry Markowitz (from
of budget to invest in n possible stakes with returns ρ1 , . . . , ρn , so www.ifa.com).

as to minimize the risk (given the covariance matrix Q of the n

investments) and subject to making at least a given mean return µ.

x > Qx

min 
n


∑ ρi xi

≥ µ 


i =1 (2.25)
n
∑ x1 = 1 


i =1



x ∈ [0, 1]n .


Since covariance matrices are positive semidefinite41 (PSD),42 41

A square symmetric matrix Q is
Eq. (2.25) is a cQP. positive semidefinite (denoted Q  0) if
∀ x x > Qx ≥ 0; or, equivalently, if all of
Another well known application of cQP is: its eigenvalues are non-negative.
Prove that all covariance matrices are
42
restricted Linear Regression (rLR). Given m points p1 , . . . , pm ∈ PSD, but the converse does not hold.
Rn arranged as rows of an m × n matrix P, find the hyperplane
a> y = a0 (where ( a0 , a) = ( a0 , a1 , . . . , an ) is restricted to lie in a set
X ⊆ Rn+1 described by linear inequalities) which minimizes43 the 43
The rLR formulation is
error min{k Pa − a0 k2 | ( a0 , a) ∈ X }.
a0 ,a
k Pa − a0 k2 .

Like all convex programs, the cQP may also arise as a convex
relaxation of a nonconvex NLP or QP, used at every node of a sBB
algorithm.

2.9.2 Quadratic programming

Quadratic Programming (QP) consists in the minimization of any
quadratic form subject to a polyhedron. The standard form is
)
min x > Qx + c> x
(2.26) Figure 2.20: Quadratic programming
Ax ≤ b, (from support.sas.com).

where c ∈ Rn , Q is an n × n matrix, b ∈ Rm and A is an m × n

matrix.
If Q  0 then Q is positive semidefinite the QP is actually a cQP
(see Sect. 2.9.1 above). If Q only has strictly positive eigenvalues, we
write Q  0 and call Q positive definite. If Q  0 (corresponding to
− Q  0), Q is called negative semidefinite, and if Q ≺ 0 then Q is
negative definite. The remaining case, of Q having both positive and
negative eigenvalues is called indefinite. Figure 2.21: Positive/negative definite,
semidefinite, and indefinite matrices
(from J. Kim’s slides Solving Poisson
equation using conjugate gradient
method).
 systematics and solution methods 39

As an application of indefinite QP, we introduce a QP formu-

lation of the Max Clique problem (see Sect. 2.4.6) on a graph
G = (V, E), usually formulated as a BLP as:

max ∑ xi 

i ∈V 
∀{i, j} 6∈ E xi + x j ≤ 1 (2.27)

x ∈ {0, 1}n 


The reformulation below, to a continuous QP, is also valid:44 44

This is known as the Motzkin-Straus
theorem, see [Vavasis, 1991, p. 82-83]
max 21 x > AG x for a proof that a global optimum x ∗


n

 has the form 1/k, where k is the size of
∑ xi = 1 (2.28) a maximum clique. Vavasis proceeds
i =1 
 by induction on n = |V |: start the
x ≥ 0,

induction at |V | = n = 2, then handle
three cases: (i) that ∃i ≤ n xi∗ = 0, and
where AG is the adjacency matrix45 of G. argue you can remove vertex i then use
induction; (ii) that G is not complete
By adding a penalty term to the objective function of Eq. (2.28), a but x ∗ > 0, and argue that from a
bijection was obtained [Bomze et al., 1998, Thm. 2.6] between solu- missing edge {i, j} you can make
xi∗ = 0 at the expense of x ∗j , thereby
tions of max x > ( AG + I ) x and the scaled characteristic vectors46 of
x ∈4n−1 moving back to case (i); (iii) that G is
all the cliques in G. More precisely, the following statements hold:47 complete, in which case x ∗ = 1/n.
45
The adjacency matrix of a graph is a
1. S is a maximal clique of G if and only if xS is a local optimum of binary square symmetric matrix which
has a 1 in entry (i, j) if and only if
Eq. (2.28); {i, j} ∈ E. The feasible set

2. S is a maximum clique of G if and only if xS is a global optimum { x ∈ Rn | 1 > x = 1 ∧ x ≥ 0 }

of Eq. (2.28). is known as the standard unit simplex

and denoted by 4n−1 .
From this it also follows48 that all local (and hence global) optima 46
The characteristic vector of a subset
S ⊆ V is a vector x in {0, 1}|V | with
of Eq. (2.28) are strict and have the form xS for some maximal
xi = 1 if and only if i ∈ S. The scaled
clique S ⊆ V. characteristic vector is |S1 | x.
Negative semidefinite and indefinite QPs can be solved49 to local 47
This theorem, proved in [Bomze,
optimality using any local NLP solver, such as IPOPT or SNOPT, 1997], extends the Motzkin-Straus
result [Motzkin and Straus, 1965].
and to global optimality using an sBB algorithm implementation 48
Why does this follow? Prove it.
such as BARON or Couenne. 49
Whenever solutions can be irrational,
such as in this case, we really mean
“approximated to arbitrary accuracy”.
2.9.3 Binary quadratic programming
Binary Quadratic Programming (BQP) consists in the minimiza-
tion of any quadratic form subject to a polyhedron. The standard
form is 
min x > Qx + c> x 

Ax ≤ b (2.29)
x ∈ {0, 1}n 


where c ∈ Rn , Q is an n × n matrix, b ∈ Rm and A is an m × n

matrix.
The following is a classic application of BQP.

Max Cut. Given a simple undirected graph G = (V, E) find a subset

S ⊆ V (called a cut) yielding the maximum number of edges {u, v}
with u ∈ S and v 6∈ S.

The Max Cut problem arises in electrical circuit layout, as well as

in statistical physics.50 It has the following formulation (also see 50
See Ch. 5 and search for the terms
“VLSI” and “Ising model” in connec-
tion with “Max Cut” on Google.
40 mathematical programming

Sect. 5.3): 
1
max 2 ∑ (1 − xi x j ) 
i< j≤n (2.30)
x ∈ {−1, 1}n , 
where n = |V |. We assign a decision variable xi to each vertex
i ∈ V, and we let xi = 1 if i ∈ S, and xi = −1 if i 6∈ S. If {i, j} is an
edge such that i, j ∈ S, then xi x j = 1 and the corresponding term on
the objective function is zero. Likewise, if i, j 6∈ S, then xi x j = 1 and,
again, the objective term is zero. The only positive contribution to
the objective function arises if xi x j = −1, which happens when
i ∈ S, j 6∈ S or vice versa, in which case the contribution of the
term is 2 (hence we divide the objective by 2 to count the number
of edges in the cut). Formulation (2.30) is not quite a BQP yet, since
the BQP prescribes x ∈ {0, 1}n . However, an affine transformation
takes care of this detail.51
BQPs are usually linearized52 before solving them. Specifically,
they are transformed to (larger) BLPs, and a MILP solver (such as
CPLEX) is then employed [Liberti, 2007].
Figure 2.22: D-Wave’s claimed “quan-
tum computer” (from dwavesys.com),
above. I sat at a few meetings where
2.9.4 Quadratically constrained quadratic programming
IBM researchers de-bunked D-Wave’s
claims, by solving Ising Model BQPs
Quadratically Constrained Quadratic Programming (QCQP) (see Ch. 5) defined on chimera graphs
consists in the minimization of any quadratic form subject to a set (below) on standard laptops faster
of quadratic constraints. The standard form is than on D-Wave’s massive hardware.
Read Scott Aaronson’s blog entry
scottaaronson.com/blog/?p=1400 (it
)
min x > Q0 x + c0 x is written in inverse chronological
(2.31)
∀i ≤ m x > Qi x + ci x ≤ b, order, so start at the bottom). By the
way, I find Prof. Aaronson’s insight,
where c0 , . . . , cm are row vectors in ∈ Rn , Q0 , . . . , Qm are n × n writing and teaching style absolutely
wonderful and highly entertaining, so
matrices, and b ∈ Rm . I encourage you to read anything he
The EDGP (see Sect. 2.6) can be written as a feasibility QCQP writes.
51
Write down the affine transformation
by simply squaring both sides of Eq. (2.21).53 Solving this QCQP,
and the transformed formulation.
however, is empirically very hard:54 there are better formulations 52
This means that each term xi x j is
for the EDGP.55 replaced by an additional variable yij ;
usually, additional linear constraints
Convex QCQPs can be solved to global optimality by local NLP are also added to the formulation to
solvers such as IPOPT and SNOPT. For nonconvex QCQPs, global link y and x variables in a meaningful
solutions can be approximated to desired accuracy using the sBB way.
53
The standard form can be achieved
algorithm (e.g. BARON or Couenne). by replacing each quadratic equation
by two inequalities of opposite sign.
54
Try solving some small EDGP
2.10 Semidefinite programming instances using Couenne. What kind
of size can you achieve?
Semidefinite Programming (SDP) consists in the minimization of
55
Write and test some formulations for
the EDGP.
a linear form subject to a semidefinite cone. The standard form is

min C • X 

i
∀i ≤ m A • X = b i (2.32)

X  0, 

where C and Ai (for i ≤ m) are n × n symmetric matrices, X is an

n × n matrix of decision variables, bi ∈ R for all i ≤ m, and for two
n × n matrices L = (λij ), M = (µij ) we have L • M = ∑ λij µij .
i,j≤n
 systematics and solution methods 41

To make Eq. (2.32) clearer, write out the componentwise product

• of the matrices C = (c jh ), Ai = ( aijh ) and X = ( x jh ):

min ∑ c jh x jh
j,h≤n

∀i ≤ m ∑ aijh x jh = bi .
j,h≤n

This is just an LP subject to a semidefinite constraint X  0.

Figure 2.23: This picture of an SDP
What does X  0 really mean? By the definition of PSD matrices, cone was borrowed from one of
this is the same as requiring the decision variables x jh to take values the most fascinating books I know
[Dattorro, 2015]. This book has been
such that, when they are arranged in a square n × n array, the
growing since the early 2000s (and
resulting matrix has non-negative eigenvalues.56 counting): part book and part encyclo-
A basic result in linear algebra states that PSD matrices have pedia, if you are looking for something
in convex optimization or Euclidean
(real) factorizations,57 namely if X  0 then X = Y > Y for some k × n geometry, this is almost sure to have it!
where k ≤ n. Now suppose we had a quadratic form f (y) = y> Qy 56
See the lecture notes [Laurent and
for some vector y = (y1 , . . . , yn ). Then Vallentin, 2012] for more information
about SDP.

∑
57
Equivalently: any PSD matrix is the
f (y) = q jh y j yh , Gram matrix of a k × n matrix Y (the
j,h≤n converse also holds).

where Q = (q jh ), and suppose f ( x ) appeared in some QP. We could

then linearize the products y j yh by replacing them with additional
variables x jh , and end up with the linear form

f (X) = ∑ q jh x jh = Q • X,
j,h≤n

where X = ( x jh ), and try to solve an easier LP resulting by replac-

ing f (y) with f ( X ).
However, not all solutions X to this LP might have the property
that there exists a vector y such that y> y = X, which we would
Figure 2.24: Another picture of the
need to retrieve a solution y to the original QP. On the other hand, same cone as in Fig. 2.23 (from L. Van-
if X  0 and rk X = 1, then the existence of such a y would surely denberghe’s slides on Convex optimiza-
follow by the existence of the factorization X = Y > Y, since rk X = 1 tion).

also implies rk Y = 1, i.e. Y is a one-dimensional vector y.

In summary, provided we were able to find X having rank one,
then the PSD constraint X  0 would automatically guarantee that
for any solution of f ( X ) we could retrieve y such that f (y) = f ( X ).
This does not really help in practice, since the rank constraint rk X =
1 is hard to handle. But it does suggest that PSD constraints might
be useful in relaxing QPs to LPs subject to X  0. Or, in other
words, that y might be a k × n matrix instead of a vector (i.e. a 1 × n
matrix).
The above discussion suggests we consider MPs with quadratic
forms where the decision variables are naturally arranged in a K × n
array, such as for example the EDGP (see Sect. 2.6 and Sect. 2.9.4): a Figure 2.25: An EDGP instance (named
realization consists on vectors y1 , . . . , yn in RK , which we arrange as tiny) from protein conformation:
given some inter-atomic distances,
columns of a K × n matrix Y. We rewrite Eq. (2.21) as follows: find the positions of the atoms. Above:
color map of distance matrix; below:
∀{u, v} ∈ E y> > > 2
u yu − 2yu yv + yv yv = duv , (2.33) the actual shape of the molecule.
42 mathematical programming

where G = (V, E) is a simple undirected graph, and d : E →

R+ . Consider a square n × n symmetric matrix X = ( Xuv ): we
linearize Eq. (2.33) by replacing y>
u yv by Xuv , and obtain the linear
constraints

∀{u, v} ∈ E Xuu − 2Xuv + Xvv = d2uv (2.34)

in X.
We would now like to get as close as possible to state that there
exists a K × n matrix Y such that X − Y > Y = 0. One possible way to
get “as close as possible” is to replace X − Y > Y = 0 by X − Y > Y  0,
which can also be trivially re-written as X − Y > In−1 Y  0. The left
hand side in this equation is known as the Schur complement of X in
the matrix
!
In Y
S= ,
Y> X
Figure 2.26: A solution to the instance
and X − Y > In−1 Y
 0 is equivalent to setting S  0. We thus derived in Fig. 2.25 obtained by the SDP solver
MOSEK [mosek7], next to the actual
the SDP relaxation of the EDGP system Eq. (2.33): shape: as you can see, they are very
 close, which means that the relaxation
∀{u, v} ∈ E Xuu − 2Xuv + Xvv = d2uv  

is almost precise.
!
In Y (2.35)
 0. 
Y> X



Many variants of this SDP relaxation for the EDGP were proposed
in the literature, see e.g. [Man-Cho So and Ye, 2007, Alfakih et al.,
1999].
SDPs are solved in polynomial time (to desired accuracy) by
means of interior point methods (IPM). There are both commercial
and free SDP solvers, see e.g. MOSEK [mosek7] and SeDuMi.
Natively, they require a lot of work to interface to. Their most
convenient interfaces are Python and the YALMIP [Löfberg, 2004]
MATLAB [matlab] package. The observed complexity of most
SDP solvers appears to be around O(n3 ), so SDP solver technology
cannot yet scale to very large problem sizes. For many problems,
however, SDP relaxations seem to be very tight, and help find
approximate solutions which are “close to being feasible” in the
original problem.

2.10.1 Second-order cone programming

Second-Order Cone Programming (SOCP) consists in the mini-
mization of a linear form subject to a second-order conic set. The
standard form is Figure 2.27:qA picture of the SOCP
 cone y3 ≥ y21 + y22 , also known as
min ∑ ci x i 
 ice cream cone, from L. Vandenberghe’s
i ≤r 

 slides on Convex optimization.
i i
∑ Ax = b


i ≤r s (2.36)
ni


x0i ≥ ∑ ( xij )2 , 

∀i ≤ r 

j =1

 systematics and solution methods 43

where, for each i ≤ r, ci is a row vector in Rni , Ai is an m × ni matrix,

b ∈ Rm , n1 , . . . , nr ∈ N, and the decision variable vector x ∈ R∑i ni
is partitioned into r blocks, each of size ni (for i ≤ r).
SOCP is a special case of SDP for which there exist more special-
ized solvers. Many58 convex problems can be cast in this form after 58
For example, a convex quadratic
suitable (and often non-trivial) transformations are applied. See constraint x > Ax + 2b> x + c ≤ 0 can be
transformed to
[Alizadeh and Goldfarb, 2003] for more information. p
k L > x + L −1 b k 2 ≤ b > A −1 b − c
after factoring A = LL> (why?). Also,
2.11 Multi-objective programming the apparently nonconvex constraints
x > x ≤ yz with y, z ≥ 0 is actually
Multi-Objective Programming (MOP) consists in the identi- convex, as it can be re-written as the
SOCP constraint
fication of the Pareto59 region (see Fig. 2.28) of many objective q
functions subject to a feasible set. The standard form is y + z ≥ 4x2 + (y − z)2

 (why?).
min f 1 ( x ) 
 59
A Pareto optimum is a solution x ∗ to
.. .. 
 Eq. (2.37) such that no other solution
. .


x 0 exists which can improve one

min f q ( x ) (2.37) objective without worsening another.

g( x ) ≤ 0 


 The Pareto set (or Pareto region) is the
 set of Pareto optima.
x ∈ X,


where f 1 , . . . , f q : Rn → R, g : Rn → Rm , and there is an integer

0 ≤ q ≤ n such that X ⊆ Zq × Rn−q .
The motivation for studying MOPs is that many social or busi-
ness problems have to do with some human decision maker who
can authoritatively decide, after seeing the cost of an optimal solu-
tion, to accept to relax some constraints in order to further decrease
the cost. In particular, decision makers often want to see what is the
trade-off between a cost and some other goal.
Any constrained problem can be turned to a MOP: take e.g. Figure 2.28: The objective function
values of three Pareto sets of a MILP
min{ f ( x ) | g( x ) ≤ 0, } (based on varying one of the param-
eters) for deciding what products
should be discontinued (or “killed”).
replace the zero in the right hand side by a scalar λ, and then solve
Both killing off products and selling
products can yield monetary returns
min{( f ( x ), λ) | g( x ) ≤ λ}. (the former through cutting costs),
but at the expense of each other
Conversely, many practical solution techniques for MOP are [Giakoumakis et al., 2012].
based on replacing all but the i-th objective (min f p ( x ) | p 6= i )
by constraints f p ( x ) ≤ λ p for p 6= i, then fixing (λ p | p 6= i )
at some iteratively decreasing λ0 . For each given λ0 , one solves a
single-objective problem

min{ f i ( x ) | ∀ p 6= i f p ( x ) ≤ λ p ∧ g( x ) ≤ 0 ∧ x ∈ X } (?)

in order to find minimal values for λ0 as (?) goes from feasible to

infeasible. For bi-objective problems, plotting the pairs ( f i ( x ∗ ), λ0 )
yields an approximation of the Pareto set (see Fig. 2.28).
No general-purpose established solvers currently exist for MOPs:
in a certain sense, the problem is too difficult for general-purpose
approaches, since the solution is not actually one point, as in single-
objective problems, but a possibly infinite or even uncountable
Pareto set. See [Ehrgott, 2005] for more information.
44 mathematical programming

2.11.1 Maximum flow with minimum capacity

We model the following problem:

an oil company intends to maximize the flow of oil on their distribu-

tion network G = (V, A) from the production node s to the customer
node t, whilst minimizing the total cost of installing pipe capacities
Cij for each arc (i, j) ∈ A.

We suppose that arc capacities have a unit cost, i.e. the total cost of
installing arc capacities is

∑ Cij . (2.38)
(i,j)∈ A

If we had not been told to minimize the total installed capacity, this
would be a standard Max Flow problem (see Sect. 1.7.2):

∑

max xsj 
x ≥0 (s,j)∈ A




∑ xis = 0



(i,s)∈ A
∀i ∈ V r {s, t} ∑ xij = ∑ x ji 



(i,j)∈ A ( j,i )∈ A 


∀(i, j) ∈ A xij ≤ Cij .


Now all we need to do is to consider the symbols C as decision

variables instead of parameters, and minimize Eq. (2.38) as a second
objective.

max ∑ xsj 
 cost
(s,j)∈ A




min ∑ Cij




(i,j)∈ A 


∑ xis = 0




(i,s)∈ A
(2.39)
∀i ∈ V r {s, t} ∑ xij = ∑ x ji 


(i,j)∈ A ( j,i )∈ A 


∀(i, j) ∈ A xij ≤ Cij





∀(i, j) ∈ A xij ≥ 0





∀(i, j) ∈ A Cij ≥ 0. 

In order to find the Pareto set of this problem, we must change

one of the two objectives, e.g. the total capacity installation cost
Eq. (2.38), into a constraint:
flow
∑ Cij ≤ λ,
(i,j)∈ A Figure 2.29: Objective values of the
Pareto set of the bi-objective network
where λ ranges over a given interval, e.g. [0, M| A|] (where M is the flow problem. The straight line arises
by the fact that the two objective
maximum capacity any arc can have installed). functions force the constraint xij ≤ Cij
to be active on all arcs (s, j), thereby
making xij = Cij on those arcs.
2.12 Bilevel programming

Bilevel Programming (BLevP) consists in the optimization60 of an 60

Called the upper-level problem.
objective function subject to some decision variables representing
the optimum of another MP formulation.61 The standard form is: 61
Called the lower-level problem.
 systematics and solution methods 45


min f ( x, y) 
x,y 


≤ 0

g( x, y) (2.40)
y ∈ arg min{φx (y) | γx (y) ≤ 0 ∧ y ∈ ζ } 



x ∈ X,


where x is a sequence of n decision variables, y is a sequence of p

decision variables, f : Rn+ p → R, g : Rn+ p → Rm , φx : Rq → R,
γx : Rq → R` , and there are integers 0 ≤ s ≤ n and 0 ≤ t ≤ p
such that ζ ⊆ Zt × R p−t and X ⊆ Zs × Rn−s . Note that φx , γx are
functions which are parametrized by the decision variables x of the
upper-level problem.
The motivation62 for bilevel programs comes from studying how 62
Typical application: a firm wants to
optimization is carried out when multiple stakeholders, ordered plan operations (upper-level) subject to
a certain unit revenue based on prices
hierarchically, make decisions to optimize different goals. As in the which are decided (lower-level) by
case of MOP, solutions in bilevel programming are not necessarily negotiations with customers.
well defined, as they may depend on whether lower-level decision
makers take decisions favorably (the case of a division of a larger
firm) or adversarially (competing entities ordered hierarchically).
Accordingly, there are no general-purpose solvers — some assump-
tions must be made on the structure of the problem.63 See [Colson 63
A promising solution approach, in
et al., 2005] for more information. the case the lower-level problem is a
MILP, consists in approximating the
lower level problem by means of a
polytope where each face is generated
2.12.1 Lower-level LP dynamically and added as a constraint
to the upper-level problem.
Certain bilevel formulations can be transformed to single-level.
Assume that:

• the objective f ( x, y) is only a function of x;

• among the constraints g( x, y) ≤ 0 only one is actually a function

of both x and y, specifically

min c( x ) · y ≥ α,
y

where c( x ) is an n-vector of functions of x and α is a parameter;

• the lower-level problem has the form:


min c( x ) · y 
y ≥0 (2.41)
A( x ) · y = b ( x ), 

where A( x ) is an m × n matrix having components in function of

x, and b is an m-vector of functions of x.

Remark that, when x is fixed, Eq. (2.41) is an LP.

Under these conditions, Eq. (2.40) can be formulated as a single-
level problem by replacing the lower-level LP with its dual. For x
fixed, the dual of Eq. (2.41) is:
)
max λ · b( x )
λ (2.42)
λ · A( x ) ≤ c ( x ),
46 mathematical programming

By LP duality we can replace Eq. (2.41) by Eq. (2.42), and obtain:

max λ · b( x ) ≥ α. (2.43)
λ

Now stop and consider Eq. (2.43) within the context of the
upper-level problem: it is telling us that the maximum of a certain
function must be at least α. So if we find any λ which satisfies
λ · b( x ) ≥ α, the maximum will certainly also be ≥ α. This means we
can forget the maxλ . The problem can therefore be written as:

min f (x) 
x,λ 


g( x ) ≤ 0



λ · b( x ) ≥ α (2.44)

λ · A( x ) ≤

c( x ) 



x ∈ X,


which is a single-level MP.

2.12.2 Minimum capacity maximum flow

Let us re-examine the example in Sect. (2.11.1) from the following
bilevel point of view: the higher-level stakeholders (say, the Chief
Financial Officer)64 aims at reducing costs over all operations while 64
Why the heck do CFOs always aim
making sure that each client receives a share of the flow by upper- at making the shareholders richer?
How about providing good service
bounding it by a threshold α, whereas a specific client manager to the individual customer? How
wants to keep his/her own client happy and maximize the flow to about making employees happier (and
hence more productive)? Capitalism
that client. took a very nasty turn in the last,
The formulation is as follows:65 say twenty years from this point of
view (from many other points of
∑ Cij

min  view the world is a better place
C ≥0 (i,j)∈ A 

 now than in the 1960s-1980s). For

F (C ) ≤ α


 this and other rants of mine, read
kafkatoday.wordpress.com.


F (C ) = max ∑ xsj


x ≥0

 65
Can this formulation be made single-
(s,j)∈ A
level? If so, use AMPL or Python to
∑ xis = 0 
 solve it. If not, explain why.
(i,s)∈ A



∑ ∑

∀i ∈ V r {s, t} xij = x ji




(i,j)∈ A ( j,i )∈ A 


∀(i, j) ∈ A xij ≤ Cij .


2.13 Semi-infinite programming

Semi-Infinite Programming (SIP) consists in the minimization of

an objective function subject to an infinite number of constraints.
The standard form is

min f (x) 

∀ σ ∈ S gσ ( x ) ≤ 0 (2.45)

∀β ∈ T x β ∈ Xβ , 

where f : Rn → R, for all α ∈ S we have gσ : Rn → R, for all β ∈ T,

X β is either Z or R, and either S or T is an infinite or uncountable
set of indices.
 systematics and solution methods 47

SIPs are mostly studied in the context of optimal control prob-

lems, where T is uncountable, as well as in robust optimization,
where S is infinite or uncountable. See [López and Still, 2007] for
more details.
Optimal control problems have both decision variables and state
variables. One can interpret the decision variables66 as parameters 66
Also known as control variables in this
to a simulation based on the ODE/PDE constraints, which are setting.

themselves expressed in terms of decision and state variables, of

which there are usually uncountably many. These problems are usu-
ally cast in terms of minimizing an integral (which might express
the length of a path, or the total amount of energy consumed by a
process) subject to differential equation constraints. Optimal control
problems are rarely so simple that they can be solved analytically.
Numerical solution methods usually call for differential equations
to be replaced by finite differences equations, and for integrals to be
replaced by sums. See [Evans, v. 0.2] for more information.
Robust optimization is a set of techniques for making sure the
solutions of a MP will hold for uniform uncertainty in certain
parameters. Suppose a MP formulation has the constraint g( p, x ) ≤
0 where p is a parameter vector and x a decision variable vector. A
robust counterpart to this constraint is:

∀ p ∈ [ p L , pU ] g( p, x ) ≤ 0,

where p is uniformly distributed in [ p L , pU ]. This type of problems

can rarely be solved in general form. For certain type of constraints,
however, ingenious solution techniques have been devised. If g
is a monotonic function, for example, it suffices to impose the
constraint at the interval endpoints, which reduces the uncountable
Figure 2.30: Typical application of
set [ p L , pU ] to the finite set { p L , pU }. Possibly the best known paper optimal control: balancing a robot
in this field is [Bertsimas and Sim, 2004]. whilst it walks [Majumdar et al., 2014].
Honestly, at first sight the robot looks
like it’s falling. I blame the middle
2.13.1 LP under uncertainty author (shown below the picture) ,.

Consider the MP

sup{c> x | ∀ a j ∈ K j
x
∑ a j x j ≤ b }, (2.46)
j≤n

where x ∈ Rn is an n-vector of decision variables, K = (K j | j ≤ n)

is a sequence of convex subsets of Rm , and b ∈ Rm , c ∈ Rn are
parameter vectors. Note that the convex sets K j need to be encoded
in a finite way (e.g., if each K j is a polytope, as a list of vertices or
facets). Note that for each j ≤ n, K j might be uncountable. So this is
a SIP with uncountably many constraints.
Consider now the LP
max c> x,
Ax ≤b
x ≥0

where A is an m × n matrix having a j as its j-th column. Eq. (2.46)

is a good representation of this LP whenever a j are subject to
uncertainty in the convex set K j . It is shown in [Soyster, 1973] that
48 mathematical programming

Eq. (2.46) is equivalent to the LP

max{c> x | Āx ≤ b ∧ x ≥ 0}, (2.47)

where Ā = ( āij ) is an m × n matrix such that:

∀ j ≤ n, i ≤ m āij = sup{ aij | a j ∈ K j }. (2.48)

Hence, in order to solve Eq. (2.46), it suffices to solve the mn con-

vex problems in Eq. (2.48), then construct the matrix Ā with their
solutions, and finally solve Eq. (2.47).
The proof establishes an equivalence between feasible solutions
of Eq. (2.47) and Eq. (2.46). Consider first x feasible in Eq. (2.47),
i.e. Āx ≤ b. By construction of Ā, we have A ≤ Ā for all A ∈
∏ j≤n K j , implying Ax ≤ Āx ≤ b for each x ≥ 0 and hence Ax ≤ b,
meaning x is also feasible in Eq. (2.46). Conversely, if x is a solution
to Eq. (2.46), ∑ a j x j ≤ b holds for any a j ∈ K j , which means it also
j≤n
holds for the supremum, implying Āx ≤ b, as claimed. Now, since
the feasible regions are the same, and the objective function is the
same, the optima are also the same.

2.14 Summary

This chapter is a systematic study of some important MP formula-

tion classes, with some of their properties, applications, solution
methods and software packages. Figure 2.31: A convincing picture
showing a higher profit solution
• Linear Programming (LP), methods, some applications traded off with a more robust but
lower profit one (from www.eyeon.nl).

• Mixed-Integer Linear Programming (MILP), complexity

and some solution methods; some applications, the Power
Generation problem, the Traveling Salesman Problem (TSP)

• Integer Linear Programming (ILP), the Scheduling problem

• Binary Linear Programming (BLP), applications: assignment,

vertex cover, set covering, set packing, set partitioning, stables
and cliques, other combinatorial optimization problems

• convex Nonlinear Programming (cNLP), complexity and

some solution methods; applications: linear regression, function
interpolation

• Nonlinear Programming (NLP), circle packing, kissing num-

ber problem, local and global solution methods

• convex Mixed-Integer Nonlinear Programming (cMINLP),

cardinality constrained portfolio selection, some methods

• Mixed-Integer Nonlinear Programming (MINLP), universal-

ity and some methods
 systematics and solution methods 49

• Quadratic formulations: convex Quadratic Programming

(cQP) and restricted linear regression, (nonconvex) Quadratic
Programming (QP) and definite/indefinite matrices, Motzkin-
Straus’ clique formulation, Binary Quadratic Programming
(BQP) and the Max Cut problem, Quadratically Constrained
Quadratic Programming (QCQP) and Euclidean distance
geometry

• Semidefinite Programming (SDP), QP/QCQP relaxations,

matrix factorization, some methods

• Second-Order Cone Programming (SOCP)

• Multi-Objective Programming (MOP) and trade-offs; some

solution methods

• Bilevel Programming (BLevP) and hierarchical optimization

• Semi-Infinite Programming (SIP) and control problems.

3
Reformulations

Reformulations are symbolic transformations in MP. We usually Figure 3.1: The preface of the special
issue 157 of Discrete Applied Mathe-
assume these transformations can be carried out in polynomial matics dedicated to Reformulations in
time. In Ch. 2 we listed many different types of MP. Some are easier Mathematical Programming [Liberti
and Maculan, 2009].
to solve than others; specifically, some are motivated by a real-
life application, and others by the existence of an efficient solver.
Usually, real-life problems naturally yield MP classes which are
NP-hard, while we are interested in defining the largest MP class
solvable by each efficient (i.e. polytime) solver we have. Ideally,
we would like to reformulate1 a given formulation belonging to a 1
I do not mean that any NP-hard
“hard” class to an equivalent one in an “easy” class. problem can just be reformulated
in polytime to a problem in P, and
Hard problems will remain hard,2 but certain formulations be- no-one knows yet if this is possible or
long to a hard class simply because we did not formulate them in not: the P 6= NP is the most important
open problem in computer science.
the best possible way.3 This chapter is about transforming MPs into 2
This is because if P is NP-hard and
other MPs with desirable properties, or features. Such transforma- Q is obtained from P by means of a
tions are called reformulations. In this chapter, we list a few useful polynomial time reformulation, then
by definition Q is also NP-hard.
and elementary reformulations. 3
More precisely, NP-hard problems
Informally speaking, a reformulation of a MP P is another MP contain infinite subclasses of instances
Q, such that Q can be obtained from P in a shorter time than is which constitute a problem in P,
e.g. the class of all complete graphs
needed to solve P, and such that the solution of Q is in some way is clearly a trivially easy subclass of
useful to solving P. We list four main types of reformulation: exact, Clique.
narrowing, relaxation, approximation. More information can be
found in [Liberti, 2009, Liberti et al., 2009a].

3.0.1 Definition
• A reformulation is exact if there is an efficiently computable surjective4
map φ : G( Q) → G( P). Sometimes we require that φ also extends to
local optima (L( Q) → L( P)), or feasible solutions (F ( Q) → F ( P)),
Figure 3.2: Exact reformulation.
or both. 4
We require φ to be surjective so that
• A reformulation is a narrowing if φ maps G( Q) to a nontrivial5 subset we can retrieve all optima of P from
the optima of Q.
of G( P). Again, we might want to extend φ to local optima or feasible 5
In narrowings, we are allowed to
solutions. lose some optima of the original
problem P; narrowings are mostly
• A reformulation is a relaxation if the objective function value at the used in symmetric MPs, where we are
global optima of Q provides a bound to the objective function value at only interested in one representative
the global optima of P in the optimization direction. optimum per symmetric orbit.

• A reformulation is an approximation6 if Q is a formulation of a se- 6

Note that, aside from the asymptotic
behaviour, there may be no relation
between Q and P; in other words,
solving Q really only provides a
heuristic method for solving P.
52 mathematical programming

quence Q1 , Q2 , . . . such that lim G( Qt ) is an exact or narrowing

t→∞
reformulation of P.

min − f
3.1 Elementary reformulations − min − f
max f
3.1.1 Objective function direction and constraint sense

Figure 3.3: Objective function direc-

We list this for completeness, but we already mentioned that tion.
any problem in the form maxx f ( x ) can be written as − minx (− f ( x )),
and vice versa.7 For the same reasons, any constraint in the form 7
I often forget to take the negative of
g( x ) ≤ 0 can be written as − g( x ) ≥ 0 (and vice versa. the optimal value after the optimiza-
tion process. Pay attention!

3.1.2 Liftings, restrictions and projections

We define here three important classes of auxiliary problems:

liftings, restrictions and projections.

• A lifting adds auxiliary variables to the original problem.

• A restriction replaces some of the variables by parameters.8 8

In other words, some of the variables
are fixed.
• A projection Q eliminates some variables from the original prob-
lem P in such a way that F ( Q) is a projection9 of F ( P) on the 9
This means that there is a mapping
subspace spanned by the variables of Q. π : F ( P) → F ( Q) such that ∀y ∈
F ( Q) there is x ∈ F ( P) with π ( x ) = y.

3.1.3 Equations to inequalities

Any equality constraint g( x ) = 0 can be replaced by pairs of

inequalities10 of opposite sign: 10
This can be done to conform to some
problem form. For example, an LP in
standard form can be reformulated to
g( x ) ≤ 0
an LP in canonical form by applying
g( x ) ≥ 0. this reformulation, and then by
turning each constraint of the form
g( x ) ≥ 0 to − g( x ) ≤ 0.
3.1.4 Inequalities to equations

An inequality g( x ) ≤ 0 in a problem P can be transformed to an

equation by means of a lifting: we add an auxiliary variable to Q,
called a slack variable11 s ≥ 0, constrained to be non-negative, and 11
Or surplus variable, depending on its
write g( x ) + s = 0. sign restriction.

3.1.5 Absolute value terms

Consider a problem P with an absolute value term |α( x )|. We can

re-write P to a problem Q without the | · | operator as follows:

• add two12 variables t+ ≥ 0, t− ≥ 0 12

If there are m absolute values in
the MP, you have to carry out this
• replace |α( x )| by the linear term t+ + t− reformulation m times, i.e. you have to
add 2m additional variables.
 reformulations 53

• add the constraint t+ − t− = α( x )

• add the constraint t+ t− = 0.
The last constraint ensures that only one in t+ , t− can be nonzero
at any time, which means that if α( x ) < 0 then t− = α( x ) and if
α( x ) > 0 then t+ = α( x ), and consequently that t+ + t− has the
same value as |α( x )|.

3.1.6 Product of exponential terms

We can replace a product of exponential terms13 ∏i e f i ( x) by an 13

Exponential terms like these are not
additional variable w ≥ 0 and the constraint ln(w) = ∑i f i ( x ). so common in applications, but they
may occasionally happen, e.g. when
expressing probability of chains of
3.1.7 Binary to continuous variables events.

We can replace any binary variable x ∈ {0, 1} to a continuous

variable by removing the integrality constraints and adding the
quadratic constraint14 x2 = x, a quadratic equation which only has 14
This constraint is sometimes also
0, 1 as solutions. written as x ( x − 1) ≥ 0 ∧ x ≤ 1 [Sutou
and Dai, 2002].
In principle, this would reduce all binary MPs to QCQPs. How-
ever, most NLP solvers do not handle the constraint x2 = x well,
empirically speaking.15 In practice, it is much better to deal with 15
Many researchers have used this re-
the binary constraint and use branching. One notable exception formulation, but mostly for theoretical
rather than practical reasons.
arises when the resulting QCQP is used to derive an SDP relaxation
of the original BLP.

3.1.8 Discrete to binary variables

A discrete variable x ∈ { p1 , . . . , pk } can be reformulated

by relaxing it to a continuous variable,16 then adding a vector 16
Essentially, this reformulation pro-
y = (y1 , . . . , yk ) of k binary variables, as well as the constraints: vides an assignment of the allowable
values to the relaxed variable by means
∑ yh = 1 of assignment constraints.
h≤k

∑ ph yh = x
h≤k

to the original problem.

3.1.9 Integer to binary variables

If p 1 , . . . , p k are consecutive integers { L, L + 1, . . . , L + k − 1},

then we can write x by means of a binary encoding.17 Let b be the 17
Many researchers have used this, see
smallest integer such that k ≤ 2b . Then add a vector z = (z1 , . . . , zb ) e.g. [Boulle, 2004, Vielma, 2015].

of b binary variables, relax x to a continuous variable, and add the

constraints:
b −1
x = L+ ∑ z h 2h
h =0
to the original problem.
54 mathematical programming

3.1.10 Feasibility to optimization problems

We can reformulate a feasibility problem

P ≡ { x ∈ Rn | ∀ i ≤ m g ( x ) ≤ 0 }

to the following optimization problem:18


min ∑ yi 

i≤m 
∀i ≤ m g( x ) ≤ yi  (3.1)

y ≥ 0,

Figure 3.4: Feasibility to optimization:
where y1 , . . . , ym are additional variables. the feasible region gi ( x ) ≤ 0 (i ≤ 3) is
shown in dark, and the level sets for
the slack variables y1 , y2 , y3 are shown
3.1.11 Minimization of the maximum of finitely many functions for values 1, 2.
18
Why would anyone want to do this?
Well, it so happens that most local
NLP solvers are good at improving
An objective function of the form19 solutions which are already feasible,
but terrible at finding a feasible
min max f i ( x ) solution from scratch. This is a way to
i ≤n make your solver’s life easier.
19
Note that the first min operator is
can be reformulated to with respect to the decision variables x,
but the second max is with respect to
min t the finite set of functions { f 1 , . . . , f n }.
This is a necessary condition for
∀i ≤ n t ≥ f i ( x ), applying this reformulation: if the
inner optimization operator is also
where t is an added (continuous) variable. The maximization of the with respect to some decision variables,
minimum of finitely many functions can treated in a similar way.20 this reformulation cannot be applied.
20
How can you reformulate the min-
imization of the minimum of finitely
many functions, or the maximization
3.2 Exact linearizations of the maximum of finitely many
functions?

These are reformulations transforming various nonlinear

terms to linear forms, which may involve additional mixed-integer
variables.

3.2.1 Product of binary variables

Consider a product term xi x j occurring in a MP P (for some

variable indices i, j), where xi , x j ∈ {0, 1}. We can linearize this
product exactly as follows:

• replace21 xi x j by an additional variable yij ∈ [0, 1]; 21

This symbolic replacement is actually
known as “linearization”. More
• add the following constraints: generally, a linearization replaces a
whole nonlinear term by means of a
yij ≤ xi (3.2) single additional variable.

yij ≤ xj (3.3)
yij ≥ xi + x j − 1. (3.4)

Constraints (3.2)-(3.4) are called Fortet’s constraints [Fortet, 1960].

We have the following result.22 22
Prove this proposition (hint: Fig. 3.5).
 reformulations 55

3.2.1 Proposition
For any i, j with

Sij = {( xi , x j , yij ) ∈ {0, 1}2 × [0, 1] | yij = xi x j }

Tij = {( xi , x j , yij ) ∈ {0, 1}2 × [0, 1] | Eq. (3.2)-(3.4)}

we have Sij = Tij .

This technique can be extended to terms of the form y I = ∏i∈ I xi

where xi ∈ {0, 1} for each i ∈ I. The linearization constraints are:23

∀i ≤ I yI ≤ xi (3.5)
Figure 3.5: The reason why Fortet’s
yI ≥ ∑ xi − | I | + 1. (3.6) reformulation works: the constraints
i∈ I form the convex envelope of the set
{( xi , x j , yij ) | yij = xi x j ∧ xi , x j ∈ {0, 1}},
i.e. the smallest convex set containing
3.2.2 Product of a binary variable by a bounded continuous variable the given set.
23
Formulate and prove the general-
ization of Prop. 3.2.1 in the case of | I |
We tackle terms of the form xy where x ∈ {0, 1}, y ∈ [ L, U ], binary variables.
and L ≤ U are given scalar parameters. It suffices to replace the
term xy by an additional variable z, and then add the following
linearization constraints:24 24
Prove their correctness.

z ≥ Lx (3.7)
z ≤ Ux (3.8)
z ≥ y − max(| L|, |U |)(1 − x ) (3.9)
z ≤ y + max(| L|, |U |)(1 − x ). (3.10)

3.2.3 Linear complementarity

A linear complementarity constraint25 has the form x > y = 0, where 25
Linear complementarity constraints
x, y ∈ [0, M ]n are two distinct vectors in Rn . Since x, y ≥ 0 every are at the basis of a standard MP
formulation called Linear Comple-
term x j y j in mentarity Problem (LCP), consisting
∑ xj yj = 0 in an LP subject to a linear complemen-
tarity constraint. LCPs are NP-hard,
j≤n
since they can encode binary variables
must be zero, i.e. the linear complementarity constraint is equiva- and hence SAT, but local optima can
be obtained by means of local NLP
lent to x j y j = 0 for each j ≤ n. This constraint actually expresses
solvers. If the LCP is a reformulation
a combinatorial property, i.e. that at most one between x j and y j is of a MILP, local optima may be feasible
nonzero. We can model this using binary variables z j ∈ {0, 1} for MILP solutions.

each j ≤ n, and adding the following constraints:26 26

Prove their correctness.

xj ≤ Mz j (3.11)
yj ≤ M (1 − z j ). (3.12)

Note that this reformulation provides an exact linearization for the

reformulation in Sect. 3.1.5.27 27
Why?

3.2.4 Minimization of absolute values

Suppose the objective function of P is the minimization of a sum

f ( x ) = · · · + |h( x )| + · · ·
56 mathematical programming

This can be reformulated as in Sect. 3.1.5, and will result in

f ( x ) = · · · + (t+ + t− ) + · · ·

subject to the constraint t+ − t− = h( x ).

Since we are minimizing f , at least one of the global optima will
have either t+ = 0 or t− = 0 even without the complementarity
constraint t+ t− = 0. Whether this optimum will be found by the
solver28 really depends on how the solver works. For example, if 28
It would be more desirable to assert
the problem reformulation is a MILP, and solutions at every node of properties which hold for any solver,
so as not to depend on the nature of
the BB are obtained using the simplex method, which finds optima the solver, which may vary greatly.
at vertices of the feasible polyhedron, this is going to be the case.29 On the other hand, getting rid of the
problematic product term t+ t− is a
We can also consider t− as a surplus variable of t+ ≥ h( x ), and great benefit.
t+ as a slack variable of t− ≤ h( x ), and re-write the above as 29
Prove this.

f ( x ) = · · · + t+ + t− + · · ·

subject to
t− ≤ h( x ) ≤ t+ .
Finally, since −|h( x )| ≤ h( x ) ≤ | h( x )|, we can simply replace
t− by −t+ in the constraints (and rename t+ as t for simplicity), to
obtain

f (x) = · · · + t +··· (3.13)

−t ≤ h( x ) ≤ t (3.14)
t ≥ 0, (3.15)

which reduces the number of added variables by half. Note that t

here represents |h( x )|, so we only need to sum it once on the ob-
jective function, i.e. we do not want to replace t− in the objective,
otherwise we would end up with 2t, which would count |h( x )|
twice, and is therefore wrong. Note also that Eq. (3.13)-(3.15) can
also be used when |h( x )| appears in terms other than just the objec-
tive function: consider for example the case of diagonal dominance
of an n × n matrix A = ( aij ) (see Sect. 6.6.2)

∀i ≤ n aii ≥ ∑ |aij |.
j 6 =i

We can reformulate this to:

∀i ≤ n aii ≥ ∑ tij (3.16)

j 6 =i
∀i, j ≤ n − tij ≤ aij ≤ tij (3.17)
∀i, j ≤ n tij ≥ 0. (3.18)

Suppose ( a, t) are feasible in the reformulation ((3.16)-(3.18)): then

we have
aii ≥ ∑ tij ≥ ∑ | aij |
j 6 =i j 6 =i

as claimed, where the first inequality is Eq. (3.16), the second

follows because Eq. (3.17) implies tij ≥ | aij |.
 reformulations 57

3.2.5 Linear fractional terms

Consider a problem P where one of the terms is

a> x + b
τ= ,
c> x + d

where a ∈ Rn , b, d ∈ R are parameters and x is a vector of n

continuous decision variables which might otherwise only appear
as Ax = q elsewhere in the problem.
We linearize the linear fractional term by adding n variables α j
xj
which play the role of c> x +d
and a further variable β which plays
1 Figure 3.6: A small Linear Fractional
the role of c> x +d
.
Programming (LFP) instance. A
Then τ can be replaced by a> α + βb, the system Ax = q can be typical application is to maximize
replaced by Aα = βq, and a new constraint efficiency: if a production line uses
xi units of product i and spends
C ( x ) = c0 + ∑i ci xi in order to make
c> α + βd = 1 a profit P( x ) = p0 + ∑i pi xi , then the
objective function is maxx P( x )/C ( x ).
can be added to the formulation.30 30
Prove that, if the feasible set is
bounded on the x variable and does
not contain any x such that c> x = −d,
3.3 Advanced examples then this reformulation is exact. Give
examples of formulations where this
reformulation is not exact.

Let us see two advanced examples where we apply the reformula-

tions given above.

3.3.1 The Hyperplane Clustering Problem

Consider the

Hyperplane Clustering Problem (HCP) [Dhyani, 2007, Caporossi

et al., 2004]. Given a set of points p = { pi | 1 ≤ i ≤ m} in Rd ,
determine a set of n hyperplanes w = {w j1 χ1 + . . . + w jd χd = w0j | 1 ≤
j ≤ n} in Rd and an assignment of points to hyperplanes such that
the sum of the distances from the hyperplanes to their assigned
points are minimized.

The problem can be modelled as follows:

• Parameters. The set of parameters is given by: p ∈ Rm×d , and m,

n, d ∈ N.
Figure 3.7: Hyperplane clustering
• Variables. We consider the hyperplane coefficient variables w ∈ [Dhyani, 2009].

Rn×d , the hyperplane constants w0 ∈ Rn , and the 0-1 assignment

variables x ∈ {0, 1}m×n .

• Objective function. We minimize the total distance, weighted by

the assignment variable:

min ∑ ∑ |w j pi − w0j |xij .

i ≤m j≤n
58 mathematical programming

• Constraints. We consider assignment constraints: each point must

be assigned to exactly one hyperplane:

∀i ≤ m ∑ xij = 1,
j≤n

and the hyperplanes must be nontrivial:

∀j ≤ n ∑ |w jk | = 1
k≤d

(since, otherwise, the trivial solution with w = 0, w0 = 0 would

be optimal).

This is a MINLP formulation because of the presence of the nonlin-

ear terms (absolute values and products in the objective function)
and of the binary assignment variables. We shall now apply several
of the elementary reformulations presented in this chapter to obtain
a MILP reformulation.

1. Because x is nonnegative and because we are going to solve the

reformulated MILP to global optimality, we can linearize the
minimization of absolute values as in Sect. 3.2.4: Figure 3.8: A HCP instance solved
with two and three hyperplanar
clusters [Amaldi et al., 2013].
min ∑ (tij+ xij + tij− xij )
i ≤m
j≤n

∀i ≤ m ∑ xij = 1
j≤n
∀ j ≤ n |w j |1 = 1
∀i ≤ m, j ≤ n tij+ − tij− = w j pi − w0j ,

where, for all i ≤ m and j ≤ n, tij+ , tij− ∈ [0, M ] are continuous

additional variables bounded above by a large enough constant
M.31 31
The upper bound M is enforced
without loss of generality because
2. Reformulate the products tij+ xij and tij− xij according to Sect. 3.2.2: w, w0 can be scaled arbitrarily.

min ∑ (yij+ + yij− )

i ≤m
j≤n

∀i ≤ m ∑ xij = 1
j≤n
∀ j |w j |1 = 1
∀i ≤ m, j ≤ n tij+ − tij− = w j pi − w0j
∀i ≤ m, j ≤ n yij+ ≤ min( Mxij , tij+ )
∀i ≤ m, j ≤ n yij+ ≥ Mxij + tij+ − M
∀i ≤ m, j ≤ n yij− ≤ min( Mxij , tij− )
∀i ≤ m, j ≤ n yij− ≥ Mxij + tij− − M,

where yij+ , yij− ∈ [0, M ] are continuous additional variables.

Figure 3.9: HCP with slabs of varying

width [Dhyani, 2009].
 reformulations 59

3. Reformulate each term |w jk | (for 1 ≤ k ≤ d) according to

Sect. 3.1.5:

min ∑ (yij+ + yij− )

i≤m
j≤n

∀i ≤ m ∑ xij = 1
j≤n

∀i ≤ m, j ≤ n tij+ − tij− = w j pi − w0j

∀i ≤ m, j ≤ n yij+ ≤ min( Mxij , tij+ )
∀i ≤ m, j ≤ n yij+ ≥ Mxij + tij+ − M
∀i ≤ m, j ≤ n yij− ≤ min( Mxij , tij− )
∀i ≤ m, j ≤ n yij− ≥ Mxij + tij− − M
∀j ≤ n ∑ (u+jk + u−jk ) = 1
k≤d
−
∀ j ≤ n, k ≤ d u+
jk − u jk = w jk
−
∀ j ≤ n, k ≤ d u+
jk u jk = 0,

−
where u+ jk , u jk ∈ [0, M ] are continuous variables for all j and
k ≤ d. 32 32
Again, the upper bound M does not
lose generality.
4. The last constraints above are complementarity constraints,33 33
They can be collectively written as
−
which we can reformulate according to Sect. 3.2.3: ∑ u+
jk u jk = 0. Why?
j≤n
k≤d

min ∑ (yij+ + yij− )

i≤m
j≤n

∀i ≤ m ∑ xij = 1
j≤n

∀i ≤ m, j ≤ n tij+ − tij− = w j pi − w0j

∀i ≤ m, j ≤ n yij+ ≤ min( Mxij , tij+ )
∀i ≤ m, j ≤ n yij+ ≥ Mxij + tij+ − M
∀i ≤ m, j ≤ n yij− ≤ min( Mxij , tij− )
∀i ≤ m, j ≤ n yij− ≥ Mxij + tij− − M
∀j ≤ n ∑ (u+jk + u−jk ) = 1
k≤d
−
∀ j ≤ n, k ≤ d u+
jk − u jk = w jk
∀ j ≤ n, k ≤ d u+
jk ≤ Mz jk
∀ j ≤ n, k ≤ d u−
jk ≤ M(1 − z jk ),

where z jk ∈ {0, 1} are binary variables for all j ≤ n and k ≤ d.

This reformulation allows us to solve P by using a MILP solver —

these have desirable properties with respect to MINLP solvers, such
as numerical stability, scalability and an optimality guarantee.
The AMPL .mod file for the MILP reformulation of the HCP is as
follows.

## parameters
param m integer, > 0;
60 mathematical programming

param n integer, > 0;

param dim integer, > 0;
set N := 1..n; # planes
set M := 1..m; # points
set D:=1..dim; # spatial dimensions
param p{M,D}; # the points
param P{d in D} := (max{i in M} p[i,d]) - (min{i in M} p[i,d]); # ranges
# used for approximate linearization
param small default 1e-6;
param big default 1e3;
# w (hyperplane coefficients) variable bounds
param wL := -1; #-big;
param w0L := -big;
param wU := 1; #big;
param w0U := big;

## variables
var w0{N} <= w0U, >= w0L;
var w{N,D} <= wU, >= wL;
var x{M,N} binary;
var tplus{M,N} >= 0, <= big;
var tminus{M,N} >= 0, <= big;
var yplus{M,N} >= 0, <= big;
var yminus{M,N} >= 0, <= big;
var uplus{N,D} >= 0, <= big;
var uminus{N,D} >= 0, <= big;
var z{N,D} binary;

## objective function
minimize fitting_error: sum{i in M, j in N} (yplus[i,j] + yminus[i,j]);

## constraints
subject to assignment {i in M} : sum{j in N} x[i,j] = 1;
subject to minabs_ref {i in M, j in N} :
tplus[i,j] - tminus[i,j] = sum{k in D} w[j,k] * p[i,k] - w0[j];
subject to yplus_lin1 {i in M, j in N} : yplus[i,j] <= big * x[i,j];
subject to yplus_lin2 {i in M, j in N} : yplus[i,j] <= tplus[i,j];
subject to yplus_lin3 {i in M, j in N} :
yplus[i,j] >= big * x[i,j] + tplus[i,j] - big;
subject to yminus_lin1 {i in M, j in N} : yminus[i,j] <= big * x[i,j];
subject to yminus_lin2 {i in M, j in N} : yminus[i,j] <= tminus[i,j];
subject to yminus_lin3 {i in M, j in N} :
yminus[i,j] >= big * x[i,j] + tminus[i,j] - big;
subject to one_norm1_ref{j in N}: sum{k in D} (uplus[j,k] + uminus[j,k]) = 1;
subject to one_norm2_ref{j in N, k in D}: uplus[j,k] - uminus[j,k] = w[j,k];
subject to one_norm3_ref{j in N, k in D}: uplus[j,k] <= big*z[j,k];
subject to one_norm4_ref{j in N, k in D}: uminus[j,k] <= big*(1 - z[j,k]);

A small instance consisting of 8 points and 2 planes in R2 , with

p = {(1, 7), (1, 1), (2, 2), (4, 3), (4, 5), (8, 3), (10, 1), (10, 5)} is solved
to optimality by the ILOG CPLEX solver [IBM, 2014] to produce the
following output:
Assignment of points to hyperplanar clusters:
cluster 1 = { 2 3 4 8 }
cluster 2 = { 1 5 6 7 }
The hyperplane clusters
 reformulations 61

1: (-0.272727) x_1 + (0.727273) x_2 + (-0.909091) = 0

2: (0.4) x_1 + (0.6) x_2 + (-4.6) = 0

3.3.2 Selection of software components

Large software systems consist of a complex architecture of

interdependent, modular software components. These may either
be built or bought off-the-shelf. The decision of whether to build
or buy software components influences the cost, delivery time and Figure 3.10: Developing software
modules or selecting them off-the-
reliability of the whole system, and should therefore be taken in an shelf? (From vogella.com).
optimal way [Cortellessa et al., 2006].
Consider a software architecture with n component slots. Let Ii
be the set of off-the-shelf components and Ji the set of purpose-built
components that can be plugged in the i-th component slot, and
assume Ii ∩ Ji = ∅. Let T be the maximum assembly time and R
be the minimum reliability level. We want to select a sequence of
n off-the-shelf or purpose-built components compatible with the
software architecture requirements that minimize the total cost,
whilst satisfying delivery time and reliability constraints.
This problem can be modelled as follows.

• Parameters:

1. Let N ∈ N;
11
0
1
0
0
1
2. for all i ≤ n, si is the expected number of calls to the i-th
component; 15

2
3. for all i ≤ n, j ∈ Ii , cij is the cost, dij is the delivery time, and IWF 11 CE 7

CDB
µij the probability of failure on demand of the j-th off-the- 3
distance
4
indicator

shelf component for slot i; value value 12

TDB
CSE 8
4. for all i ≤ n, j ∈ Ji , c̄ij is the cost, tij is the estimated devel- 5
CLS
13

opment time, τij the average time required to perform a test 6

HRDB
OWF

case, pij is the probability that the instance is faulty, and bij the 14

1
0 CLDB
1
0
09
1
testability of the j-th purpose-built component for slot i. 0
1 10 1
0

Figure 3.11: MP can be used to ratio-

• Variables: nalize interfaces in software architec-
tures, modelled as a graph, directed or
undirected. After some years of devel-
1. Let xij = 1 if component j ∈ Ii ∪ Ji is chosen for slot i ≤ n, and opment, a complex software package
0 otherwise; may start to display a “spaghetti”-like
architecture such as the one shown
2. Let Nij ∈ Z be the (non-negative) number of tests to be above. This can be simplified by insert-
performed on the purpose-built component j ∈ Ji for i ≤ n: we ing interface modules whose purpose
is that of dispatching to the correct
assume Nij ∈ {0, . . . , N }. module. If we identify a subgraph H
which is fully connected, we can add
• Objective function. We minimize the total cost, i.e. the cost of the an interface ι and replace the clique
off-the-shelf components cij and the cost of the purpose-built with a star centered at ι and connected
to all of the nodes of H (see Fig. 3.12).
components c̄ij (tij + τij Nij ):
!
min ∑ ∑ cij xij + ∑ c̄ij (tij + τij Nij )xij .
i ≤n j∈ Ii jinJi
62 mathematical programming

• Constraints:

1. assignment constraints: each component slot in the architec-

ture must be filled by exactly one software component

∀i ≤ n ∑ xij = 1;
j∈ Ii ∪ Ji

2. delivery time constraints: the delivery time for an off-the-

shelf component is simply dij , whereas for purpose-built
components it is tij + τij Nij

∀i ≤ n ∑ dij xij + ∑ (tij + τij Nij )xij ≤ T;

j∈ Ii j∈ Ji

3. reliability constraints: the probability of failure on demand

of off-the shelf components is µij , whereas for purpose-built
components it is given by

pij bij (1 − bij )(1−bij ) Nij 11

0
ϑij = (1−bij ) Nij
, 1
0
0
1
(1 − pij ) + pij (1 − bij ) 15

so the probability that no failure occurs during the execution 2

11 12
7
IWF CDB CE
TDB
of the i-th component is 3 4
distance indicator 16
! value value

∑ µij xij + ∑ ϑij xij

DBI
si CSE 8
j∈ Ii j∈ Ji 5
ϕi = e , CLS

13 14
6 OWF
whence the constraint is HRDB CLDB

1
0 1
0
0
1 1
0
09
1 10

∏ ϕi ≥ R; Figure 3.12: Given a software archi-

i≤n
tecture (directed) graph G = (V, A),
we suppose that arcs are colored by a
notice we have three classes of reliability constraints involving
coloring function µ : A → {1, . . . , K }.
two sets of additional variables ϑ, ϕ. These colors may denote any given
type of semantic relationship between
This problem is a MINLP P with no continuous variables. Next, the modules: they provide a way for
the software architect to define rele-
we reformulate P to a MILP. vant subgraphs which can actually
be replaced by an interface. What
1. Consider the term g = ∏i≤n ϕi and apply the reformulation in we can do automatically with MP
Sect. 3.1.6 to P, to obtain the formulation P1 as follows: is to identify the densest subgraphs
H = (U, F ), which we define as
! subgraphs where | F | − |U | is maxi-
min ∑ ∑ cij xij + ∑ c̄ij (tij + τij Nij )xij mum, where all the arcs are colored
by a given k. To achieve this, for any
i ≤n j∈ Ii j∈ Ji
given color k we solve the following
∀i ≤ n ∑ xij = 1 BQP: max ∑ xu xv − ∑ xv
x ∈{0,1}|V | (u,v)∈ A v ∈V
j∈ Ii ∪ Ji
subject to ∀ (u, v) ∈ A xu xv ≤
∀i ≤ n ∑ dij xij + ∑ (tij + τij Nij )xij ≤ T min(max(0, µuv − k + 1), max(0, k −
j∈ Ii j∈ Ji µuv + 1)). The set of vertices U of the
densest subgraph having color k is
pij bij (1 − bij )(1−bij ) Nij defined by the indicator vector x: if
= ϑij |U | < | F | we replace F by the arcs
(1 − pij ) + pij (1 − bij )(1−bij ) Nij {(ι, u) | u ∈ U }. We can solve this BQP
w ≥ R for every color k ≤ K.
!
∑ si ∑ µij xij + ∑ ϑij xij = ln(w),
i≤n j∈ Ii j∈ Ji
 reformulations 63

and observe that w ≥ R implies ln(w) ≥ ln( R) because the ln

function is monotonically increasing, so the last two constraints
can be grouped into a simpler one not involving logarithms of
problem variables:
!
∑ si ∑ µij xij + ∑ ϑij xij ≥ ln( R).
i ≤n j∈ Ii j∈ Ji

2. We now make use of the fact that Nij is an integer variable

for all i ≤ n, j ∈ Ji , and reformulate the integer variables to
binary variables as per Sect. 3.1.8. For k ∈ {0, . . . , N } we add
assignment variables νijk so that νijk = 1 if Nij = k and 0 otherwise.
Now for all k ∈ {0, . . . , N } we compute the constants ϑk =
(1−bij )k
pij bij (1−bij )
(1−bij )k , which we add to the problem parameters.
(1− pij )+ pij (1−bij )
We remove the constraints defining ϑij in function of Nij : since
the following constraints are valid:

∀i ≤ n, j ∈ Ji ∑ νijk = 1 (3.19)
k≤ N

∀i ≤ n, j ∈ Ji ∑ kνijk = Nij (3.20)

k≤ N

∀i ≤ n, j ∈ Ji ∑ ϑk νijk = ϑij , (3.21)

k≤ N

the constraints in Eq. (3.20) are used to replace Nij , and those in
Eq. (3.21) to replace ϑij .34 We obtain: 34
Constraints (3.19) are simply added
to the formulation.
!
min ∑ ∑ cij xij + ∑ c̄ij (tij + τij ∑ kνijk ) xij
i ≤n j∈ Ii j∈ Ji k≤ N

∀i ≤ n ∑ xij = 1
j∈ Ii ∪ Ji
11
00
∑ dij xij + ∑ (tij + τij ∑ kνijk ) xij
1
00
11
∀i ≤ n ≤ T 00
11

j∈ Ii j∈ Ji k≤ N 15
!
∑ si ∑ µij xij + ∑ xij ∑
2

ϑk νijk
11 12
≥ ln( R) IWF CDB TDB
CE 7

i ≤n j∈ Ii j∈ Ji k≤ N 3 4
distance indicator 16

∑
value value
∀i ≤ n, j ∈ Ji νijk = 1. DBI
8
k≤ N 5
CLS
CSE

13 14
3. We distribute products over sums in the formulation to obtain 6 OWF
HRDB CLDB

the binary product sets { xij νijk | k ≤ N } for all i ≤ n, j ∈ Ji : by 0

1
1
0
09
1 10 1
0
0
1

repeatedly applying the reformulation in Sect. 3.2.1 to all binary C1 C2

products of binary variables, we get a MILP reformulation Q of Figure 3.13: After interfacing, we can
P where all the variables are binary. solve a graph partitioning problem
on G = (V, A) so as to identify
We remark that Q derived above has many more variables and semantically related clusters, for
which there might conceivably exist an
constraints than P. More compact reformulations are applicable off-the-shelf solution.
in Step 3 because of the presence of the assignment constraints
[Liberti, 2007].
Reformulation Q essentially rests on linearization variables wijk
which replace the quadratic terms xij νijk throughout the formulation.
64 mathematical programming

A semantic interpretation of step 3 is as follows. We notice that for

i ≤ n, j ∈ Ji , if xij = 1, then xij = ∑k νijk (because only one value k
will be selected), and if xij = 0, then xij = ∑k νijk (because no value k
will be selected). This means that

∀i ≤ n, j ∈ Ji xij = ∑ νijk (3.22)

k≤ N

is a valid problem constraint. We use it to replace xij everywhere

in the formulation where it appears with j ∈ Ii , obtaining a opt-
reformulation with xij for j ∈ Ii and quadratic terms νijk νlhp . Now,
because of (3.19), these are zero when (i, j) 6= (l, p) or k 6= h and
are equal to νijk when (i, j) = (l, p) and k = h, so they can be
linearized exactly by replacing them by either 0 or νijk according
to their indices. What this really means is that the reformulation
Q, obtained through a series of automatic reformulation steps, is a
semantically different formulation defined in terms of the following
decision variables:
(
1 if j ∈ Ii is assigned to i
∀i ≤ n, j ∈ Ii xij =
0 otherwise.

 1 if j ∈ Ji is assigned to i and there

k
∀i ≤ n, j ∈ Ji , k ≤ N νij = are k tests to be performed

 0 otherwise.

This is an important hint to the importance of automatic reformula-

tion in problem analysis: it is a syntactical operation, the result of
which, when interpreted, can suggest a new meaning.

3.4 Summary

This chapter is about reformulations as a means to change a for-

mulation so it can be solved using a given solver, so that some
optimality properties are preserved.

• Motivation and main definitions

• Elementary reformulations: changing objective direction and

constraint sense, lifting – restrictions – projections, turning equa-
tions into inequalities and vice versa, absolute values, product
of exponentials, turning binary variables to continuous, turning
discrete variables to binary, turning integer variables to binary,
turning feasibility to optimization problems, minimization of a
maximum

• Exact linearizations: product of binary variables, product of a

binary variable by a continuous one, linear complementarity,
minimization of absolute values, linear fractional terms

• Applying reformulations to turn the hyperplane clustering

problem from MINLP to MILP.
 Part II

In-depth topics
4
Constraint programming

Constraint Programming (CP) is a declarative programming lan-

guage for describing feasibility problems, such as e.g. finding the
solution of a given sudoku game, or of a crossword puzzle, or of
a game of eternity (see Fig. 4.1), or a neutral arrangement of nine
queens on a chessboard.1
The CP language consists of sets, parameters, decision variables
and constraints: like MP but without objective functions. Although Figure 4.1: A 4 × 4 eternity board [Dürr,
2011].
today decision variables in CP can be both discrete and continuous, 1
For some reason which completely
“native” CP only offered discrete decision variables.2 escapes me, constraint programmers
Let p be the parameter vector, which defines the input of the are very fond of games and puzzles, to
the point that it is relatively hard to
problem being solved. Let D1 , . . . , Dn be the domains of the deci- find any real application of CP, aside
p p from scheduling that is.
sion variables x1 , . . . , xn . Let C1 ( x ), . . . , Cm ( x ) be the constraints,
expressed as logical statements which can evaluate to YES or NO.
2
In fact, CP was originally only used
with explicitly bounded decision
Denote as x 0 = C ( p) a solution of the CP formulation, with x 0 = ∅ variables, which prevents the language
if C ( p) is infeasible. from being Turing-complete. Even
p with bounded variables, CP can still
Although a constraint Ci (for i ≤ m) is commonly written in describe NP-hard problems, though:
a humanly readable form, such as AllDifferent( xi1 , . . . , xik ) mean- try reducing SAT to CP.
p
ing “xi1 , . . . , xik must be assigned different values”, Ci is formally
defined as a k-ary relation Ri ⊆ Di1 × · · · × Dik .
p
p C2
C1 x2
x3
4.1 The dual CP x1

p p
We remark that Ci need not be explicitly cast in function of all the C4
p C3
decision variables. In fact, it turns out that every CP formulation x5
x4 x6
C ( p) can be reduced to one which has at most unary and binary
constraints, by reduction to the dual CP, denoted C ∗ ( p). For each
p
i ≤ m, consider the subset Si ⊆ {1, . . . , n} of variable indices that Ci
Figure 4.2: The constraint/variable
depends on.
incidence structure of the CP for-
The dual CP has m dual variables y1 , . . . , ym with domains mulation can be represented as a
R1 , . . . , Rm and the dual constraints: hypergraph.

∀ i < j ≤ m y i [ Si ∩ S j ] = y j [ Si ∩ S j ] . (4.1)

Note that each dual variable yi takes values from the relation Ri ,
which is itself a subset of a cartesian product of the Di ’s. Hence you
should think of each dual variable as a vector or a list, which means
that y = (y1 , . . . , ym ) is a jagged array.3 The meaning of Eq. (4.1) is 3
A jagged array is a basic data struc-
ture: it consists of a list of lists of
different sizes.
68 mathematical programming

p p
that, for each pair of constraints Ci , Cj , we want the corresponding y1 y2
{2, 3}
dual variables yi , y j to take on values from Ri and R j which agree
on Si ∩ S j . Note that formally each dual constraint is binary, since it
only involves two dual variables. Note also that when Si ∩ S j = ∅, {3

{3}
}
the constraint is empty, which means it is always satisfied and we
can formally remove it from the dual CP. Thus, Eq. (4.1) contains
O(m2 ) constraints in the worst case, but in practice there could be
y4 y3
fewer than that.

4.1.1 Proposition Figure 4.3: A graph representation

of the constraint/variable incidence
C ∗ ( p) is an exact reformulation of C ( p) and vice versa. structure of the dual formulation to
the CP of Fig. 4.2. Since all constraints
Proof. Let x 0 = C ( p) be a feasible solution of the original CP for- are binary, the hyperedges are all
p simple edges, labeled with Si ∩ S j .
mulation. This means that, for all i ≤ m, Ci ( xi01 , . . . , xi0 ) evaluates p
k Note that the unary constraint C4
to YES. Define yi0 = ( xi01 , . . . , xi0 ). We claim y0 = (y10 , . . . , y0m ) sat- in the original CP becomes a unary
k
isfies all constraints Eq. (4.1). Consider two constraints i < j with constraint in the dual, which simply
p
0 = x 0 = y0 , which proves specifies the action of C4 on x6 as the
Si ∩ S j 6= ∅. For each h ∈ Si ∩ S j , yih h jh restriction of R4 so that D6 becomes
the claim. Conversely, let y0 = C ∗ ( p) be a feasible solution for the p
C4 ( D6 ) (the restriction of D6 induced
p
dual CP, and aim at constructing a solution x 0 of C ( p). Pick any by C4 ).
h ≤ n: if there are no i < j such that h ∈ Si ∩ S j , then set xh0 to any
p
element of Dh compatible with Ci . Otherwise, for any i < j such
that h ∈ Si ∩ S j , feasibility of y0 with respect to the dual constraints
ensure that yih0 = y0 , so set x 0 = y0 . Now consider any constraint
jh h ih
p
Ci ( x ) of the original CP, for some i ≤ m, and aim to show that
p
Ci ( x 0 ) evaluates to YES: if h ∈ Si and h 6∈ S j for any j 6= i, then xh0 is,
p
by construction, part of a satisfying solution for Ci ; if h ∈ Si ∩ S j for
some j 6= i, then xh = yih 0 = y0 is in the projection of both R and R
jh i j
to the h-th coordinate. Hence they are also part of a satisfying solu-
p
tion for Ci . Having constructed a bijection between solutions of the
two CP formulations, we can use it to define the map φ mentioned
in Defn. 3.0.1. 2

4.2 CP and MP

CP relates to computer science as MP relates to applied mathemat-

ics. Both CP and MP are Turing complete declarative programming
languages as long as their decision variables are not explicitly
bounded. Both contain NP-hard subclasses of formulations, and
hence are NP-hard when seen as formal decision problems. MP
was born out of mathematics, whereas CP stems from the develop-
ment of logic into computer science. The academic communities
around CP and MP are still different, although their intersection
Figure 4.4: The CPAIOR 2011 confer-
is growing, insofar as solution methods from CP and MP can be
ence poster, see www.andrew.cmu.edu/
successfully mixed. user/vanhoeve/cpaior.
Differently from MP, sets in CP are used both for indexing vari-
able symbols in structured formulations and for defining domains
where the decision variables can range. Also differently from MP,
constraints in CP are put together from a richer set of operators
than the one usually employed in MP. In fact, much of the research
 constraint programming 69

in CP focuses on creating new operators which can be efficiently

dealt with in the typical CP solution algorithm.
Another important difference between CP and MP is that,
whereas in MP we have many different solvers for different MP
subclasses, in CP there is essentially one solution approach: a
search based on branching and backtracking.

4.3 The CP solution method

At its most essential, branching and backtracking is limited to work

on bounded variable domains, and looks like the algorithm in
Fig. 4.6.
This algorithm generates a search tree where each node has
as many child nodes as the number of elements in the domain of
the branching variable. The algorithm can be refined by means of Figure 4.5: Branching/backtracking on
several per-node procedures. The most important ones are domain the 4 queens problem (from ktiml.mff.
reduction and pruning. The former consists in various heuristic cuni.cz).

procedures for attempting to remove as many elements as possible

from the variable domains without losing any part of the feasible function CPsolve(J, x 0 )
// J = indices of unfixed vars
region. As for the latter, when every element can be removed from // x 0 = vector of fixed var. values
a domain, the current search node is pruned. if J = ∅ then
return x 0
end if
4.3.1 Domain reduction and consistency choose branching var. index j ∈ J
for v ∈ D j do
A set of k domains Di1 , . . . , Dik is k-consistent with respect to a fix x 0j ← v
p if some constraint evaluates to
constraint Ci ( xi1 , . . . , xik ) if for each (k − 1)-tuple
NO then
return ∅
( a i 1 , . . . , a i j − 1 , a i j + 1 , . . . , a i k ) ∈ Di 1 × · · · × Di j − 1 , Di j + 1 × · · · × Di k − 1 else
return CPsolve(J ∪ { j}, x 0 )
p end if
there is ai j ∈ Di j such that Ci ( ai1 , . . . , aik ) evaluates to YES. A CP
end for
instance is k-consistent if all of its cartesian domain k-products Figure 4.6: CPsolve(J, x 0 )
are k-consistent with respect to each k-ary constraint. The cases of
k ∈ {1, 2} are the most important: the case k = 1 because it can
be dealt through pre-processing, and the case k = 2 because every
CP can be reduced (by means of duality, see Sect. 4.1) to the case of
binary constraints.
Consider for example a domain D1 = {2, 3, 5, 6, 9} for the vari-
p
able x1 , and the unary constraint C1 ≡ [ L ≤ x1 ≤ U ], where L ≤ U
are parameters which are given as part of the input p. Suppose for
some given p = p0 we have L = 4 and U = 7. Then D1 is consis-
p0
tent with respect to C1 if D1 is reduced to {5, 6}, namely only the
values which satisfy 4 ≤ x1 ≤ 7. Figure 4.7: Patrick Winston ex-
plaining 1-consistency (from
Consistency with respect to unary constraints is also known as
MITOpenCourseWare).
1-consistency or node-consistency. As mentioned above, it can be
carried out as a pre-processing step to the solution process: since
unary constraints cannot encode relations between variables, once
the domain becomes consistent w.r.t. a unary constraint, it must
necessarily keep being node-consistent throughout the solution
process. Suppose, to get a contradiction, that D j is node-consistent
p
with respect to the unary constraint Ci ( x j ), but that, due to some
70 mathematical programming

p
other constraint C` involving x j , D j becomes node-inconsistent
p
w.r.t. Ci . Since constraints can never add values to domains, the
p
effect of C` on D j must yield a domain D 0j ⊆ D j . Of course, if D 0j is
p
node-inconsistent w.r.t Ci , any superset of D 0j must also be node-
inconsistent, which means that D j is, against the initial assumption.
Consistency with respect to binary constraints of the form
p
Ci ( x j , xh ) is also known as 2-consistency or arc-consistency. The do-
p
mains D j , Dh are 2-consistent with respect to Ci if for each a j ∈ D j
p
there is ah ∈ Dh such that Ci ( a j , ah ) evaluates to YES. Consider
for example the domain D2 = {1, 2, 3, 4} for x2 and the constraint
x1 < x2 : then D1 , D2 are arc-consistent with respect to x1 < x2 if D1
is reduced to {2, 3} and D2 to {3, 4}.
Domain reduction through consistency is usually performed be-
fore choosing the branching variable index in the search algorithm
(Fig. 4.6).

4.3.2 Pruning and solving Figure 4.8: Branching in CP (from

www.info.ucl.ac.be/~pschaus).
After a certain amount of branching and domain reduction, there
might4 be leaf nodes of the search tree where the domain product 4
If all the domains have a finite
D = D1 × Dn is either empty, or consists of exactly one element number of values, then you can replace
“might” by “will”.
a = ( a1 , . . . , an ). In the former case, the leaf node is pruned, i.e. no
child nodes are further appended to the pruned node. In the latter
case, the solution x = a is returned to the user, and the algorithm
terminates.
Next, we illustrate how the AllDifferent constraint impacts pruning
and finding solutions in CP. Suppose D1 = . . . , = Dn = D such that
| D | = n, and x1 , . . . , xn ∈ D. By stating

AllDifferent( x1 , . . . , xn )

we are really saying that x1 , . . . , xn encode a permutation of D.

We know from Sect. 2.2 and 2.4 that permutations correspond to
linear assignments, which are special cases of matchings.5 Finding 5
A matching M in a graph G =
∗ (V, E) is a set of edges which are
matchings in graphs can be done in polytime [Edmonds, 1965] , pairwise non-incident. The relevant
which implies that we can efficiently test search tree nodes for optimization problem is to find
matchings of maximum size. A linear
pruning or finding candidate solutions. assignment is a maximum perfect
More specifically, the matching arising from an AllDifferent con- matching in a bipartite graph (also see
Sidenote 13 in Ch. 2).
straint is defined on a bipartite graph ( x, D, E) between the n-
variable vector x and the n-valued domain D; E denotes the set
of variable/value assignments which are feasible with respect to
the current node of the search tree. At this point we invoke Hall’s
theorem6 6
Also known as stable marriage theo-
rem. You can look at [Dürr, 2011] for a
4.3.1 Theorem ([Hall, 1935]) short proof.

There exists a perfect matching7 in a bipartite graph (U, V, E) if and only 7

A matching is perfect if every vertex
if is adjacent to exactly one edge in
the matching. Referred to a bipartite
∀S ⊆ U |S| ≤ |Γ(S)|, (4.2) graph (U, V, E) the existence of a
perfect matching obviously implies
where Γ(S) ⊆ V is the set of vertices in V adjacent to the vertices in S. that |U | = |V |.
 constraint programming 71

By Thm. 4.3.1, there is no perfect matching if the contrapositive of

Eq. (4.2) is true, namely

∃S ⊆ U |S| > |Γ(S)|,

and it turns out that the certificate S can be found in polytime.

This offers the perfect tool for pruning a node using the AllDiffer-
ent constraint: at some level of the search tree, nodes will be created
where the edge set E is sparse enough (edges having been removed
at prior nodes) so as not to allow perfect matchings, with this
condition being efficiently verifiable. Conversely, since maximum
matchings can also be found in polytime, the AllDifferent constraint
offers an efficient way to find candidate solutions to the CP.

4.4 Objective functions in CP

CP formulations do not natively have objective functions f ( x ) to Figure 4.9: An unforeseen glitch in
optimize, but there are at least two approaches in order to handle Hall’s stable marriage theorem (from
snipview.com).
them.
A simple approach consists in enforcing a new constraint
f ( x ) ≤ d, and then update the value of d with the objective function
value of the best solution found so far during the solution process
(Sect. 4.3). Since a complete search tree exploration lists all of the
feasible solutions, the best optimum at the end of the exploration is
the global optimum.
In the case of discrete bounded variable domains another ap- while L < U do
proach consists in using bisection, which we explain in more detail. d ← ( L + U )/2
x̄ ← C¯( p, d)
Let C ( p) be a CP formulation, with p a parameter vector represent- if x̄ = ∅ then
ing the instance; and let x 0 = C ( p), with x 0 = ∅ if p is an infeasible L←d
else
instance. U←d
Given some lower bound f L and upper bound f U such that end if
f +f end while
f L ≤ minx f ( x ) ≤ f U , we select the midpoint d = L 2 U and
return x̄
consider the formulation C¯( p, d) consisting of all the constraints of
C ( p) together with the additional constraint f ( x ) ≤ d. We compute Figure 4.10: The bisection algorithm
(see Fig. 1.16).
x̄ = C¯( p, d): if x̄ = ∅ we update f L ← d, otherwise f U ← d,
finally we update the midpoint and repeat. This process terminates
whenever f L = f U , in which case x̄ is a feasible point in C ( p) which
minimizes f ( x ).
Bisection may fail to terminate unless f L , f U can only take finitely
many values (e.g. f L , f U ∈ N).8 Since CP formulations are mostly 8
We remark that f L can only increase
defined over integer or discrete decision variables, this requirement and f U can only decrease during the
bisection algorithm, hence if both are
is often satisfied. integer they can only range between
The complexity of bisection is O(t log2 (|U − L|)) where t is the their initial values.

complexity of calling C¯( p, d), which makes it attractive in case

C¯( p, d) can be solved efficiently.

4.5 Some surprising constraints in CP

Modelingwise, we already mentioned that CP offers a lot more

variety than MP in the constraints it can deal with.
72 mathematical programming

Name Constraint Why is it unusual for MP?

The 6= relation is not admitted in MP. It can be modelled using ≤, ≥
AllDifferent( x) ∀i < j ≤ n ( xi 6 = x j ) and additional binary variables.
Logical connectives are not always admitted in MP (in particular,
the implication is not). They can modelled using additional binary
Implies( x, g, i, j) gi ( x ) ≤ 0 ⇒ g j ( x ) ≤ 0 variables, linked to the truth value of each logical statement by means
of “big M”s and cardinality constraints.
Constraint disjunctions is not admitted in MP. It can nonetheless be
( gi ( x ) ≤ 0 )
W
Disjunction( x, g) modelled using additional binary variables yi , additional constraints
i≤m gi ( x ) ≤ M (1 − yi ), and one more additional constraint ∑i yi ≥ 1.
Exactly( x, I, k, g) |{i ∈ I | gi ( x ) ≤ 0}| = k Cardinality constraints on the number of satisfied constraints in a set
AtLeast( x, I, k, g) |{i ∈ I | gi ( x ) ≤ 0}| ≥ k are not admitted in MP. They can be modelled using additional slack
|{i ∈ I | gi ( x ) ≤ 0}| ≤ k variables gi ( x ) ≤ Myi and imposing the cardinality constraint on ∑i yi .
AtMost( x, I, k, g)
Decision variables can never be used for indexing in MP. This
Element(x, L, i, j) L xi = x j situation can be modelled using additional variables y`v = 1 if
and only if L` = v ∧ xi = ` ∧ x j = v.

4.6 Sudoku

As a toy example, we propose a CP formulation for solving a

Sudoku puzzle. This is seen as the following decision problem.
√
Sudoku. Given n ∈ N such that n ∈ N, and a partially defined n × n
array M = (mij ) where, for some given pairs (i, j) ∈ S, mij specifies
an integer in N = {1, . . . , n}, is there an n × n fully defined array
X = ( xij ) such that:

• ∀(i, j) ∈ S xij = mij

• every integer in N appears in each column of X
• every integer in N appears in each row of X
√ √
• every integer in N appears in each n × n sub-matrix of X
√ √
whose upper-left element has indices (k n, ` n) for some inte-
√
gers k, ` ∈ R = {0, . . . , n − 1}?
√
We introduce integer parameters n, r ← n, sets N = {1, . . . , n}
and R = {0, . . . , r − 1}, and a partially defined square array mij for
i, j ∈ . The constraints are:
Figure 4.11: A “hard” Sudoku instance
∀i ∈ N, j ∈ N xij = mij (from theguardian.com).

∀i ∈ N AllDifferent( xi1 , . . . , xin )

∀j ∈ N AllDifferent( x1j , . . . , xnj )
∀k ∈ R, ` ∈ R AllDifferent( xij | kr + 1 ≤ i ≤ kr + r ∧ `r + 1 ≤ c ≤ `r + r ).

4.6.1 AMPL code

The CP formulation is encoded by means of the following AMPL
sudoku_cp.mod file.

param n integer, >0;

param r integer, := sqrt(n);
set N := 1..n;
set R := 0..r-1;
param m{N,N} integer, default 0;
var x{N,N} integer, >= 1, <= n;
subject to completeM{i in N, j in N : m[i,j] > 0}: x[i,j] = m[i,j];
subject to rows{i in N}: alldiff{j in N} x[i,j];
subject to cols{j in N}: alldiff{i in N} x[i,j];
 constraint programming 73

subject to squares{k in R, l in R}:

alldiff{i in k*r+1..k*r+r, j in l*r+1..l*r+r} x[i,j];

We can specify an instance by means of the following sudoku_cp.dat

file.
param n := 9;
param m: 1 2 3 4 5 6 7 8 9 :=
1 2 . 4 . . . . 5 .
2 . 9 . . . 8 . . .
3 8 . . 1 . . . 2 .
4 . . 7 3 6 . . 8 .
5 . . 8 . . . 2 . .
6 . 3 . . 2 1 7 . .
7 . 8 . . . 3 . . 4
8 . . . 5 . . . 6 .
9 . 4 . . . . 8 . 5
;

Finally, the following sudoku_cp.run file calls the solver and

displays the solution.

model sudoku_cp.mod;
data sudoku_cp.dat;
option solver ilogcp;
option ilogcp_options "logverbosity=terse";
solve;
printf "x:\n";
for{i in N} {
for{j in N} {
printf "%2d ", x[i,j];
}
printf "\n";
}
printf "\n";

The resulting solution is:

x:
2 7 4 9 3 6 1 5 8
6 9 1 2 5 8 4 3 7
8 5 3 1 4 7 6 2 9 Figure 4.12: A weird Sudoku variant
4 2 7 3 6 9 5 8 1 (from saxi753.deviantart.com).
Change the CP formulation in this
1 6 8 4 7 5 2 9 3
section to handle this instance, and
9 3 5 8 2 1 7 4 6 find a solution.
5 8 2 6 1 3 9 7 4
7 1 9 5 8 4 3 6 2
3 4 6 7 9 2 8 1 5

4.7 Summary

This chapter is about constraint programming, which is a different

modeling paradigm than MP, focusing on feasibility rather than
optimality, on discrete rather than continuous variables, and on a
single solver parametrized on constraint structures instead of many
solvers, each for a different standard form.
5
Maximum cut

The problem which names this chapter is one of the best known
problems in combinatorial optimization: finding the cutset of
maximum weight in a weighted undirected graph.1 It has two main 1
We already mentioned the un-
motivating applications: finding the minimum energy magnetic weighted variant in Sect. 2.9.3.

spin values of the atoms in a given crystal, and knowing where

to drill holes in electrical circuits so that wires that must intersect
do so in different board layers. The first application is commonly
known as Ising model of spin glasses, and the second as Very Large
Scale Integration (VLSI).2 2
Lately, another “application” came
First thing first: the formal definition of this problem is the about, namely benchmarking D-Wave’s
claimed quantum computer, see
following. Fig. 2.22 in Ch. 2.
Max Cut. Given a weighted undirected graph G = (V, E, w) with
w : E → R, find S ⊆ V such that ∑ wij is maximum. 2
i ∈S
j6∈S 3
In graph theoretical terminology, a subset S of vertices is known 1

as a cut, whereas the set of edges with one incident vertex in S and
the other in V r S is a cutset. So the Max Cut problem calls for the 5

cutset of maximum weight (see Fig. 5.1).

4 6
Max Cut is one of the original problems which R. Karp proved
NP-hard [Karp, 1972], which is one of the main reasons for its
importance. The reduction proof is from a problem called Max- Figure 5.1: A graph with unit edge
2-Sat, which aims at determining the maximum number of a weights and its maximum cut. The
conjunction of satisfiable clauses involving a disjunction of two corresponding cutset is the set of edges
intersected by the curved perimeter.
literals each.3 3
A literal is a boolean variable x or
Another reason why Max Cut is so famous is that it is naturally its complement x̄. A logical formula in
conjunctive normal form is a boolean
modelled as a MIQP rather than a MILP, whereas most of the expression in the form C1 ∧ . . . ∧ Cm ,
other “classic” combinatorial optimization problems are naturally for some integer m, where each clause
Ci is a disjunction L1 ∨ . . . ∨ Lni of
modelled as MILP. literals (i.e. each L j is either a boolean
Most importantly, however, I believe that Max Cut is very fa- variable or its complement), for each
mous because of the algorithm that Goemans and Williamson i ≤ m. In Max-2-Sat we require that
ni = 2 for each i ≤ m.
published in 1994. Theirs was the first SDP-based randomized ap-
proximation algorithm for a combinatorial optimization problem.
The so-called “GW algorithm” became an instant success, and has
had a deep impact in theoretical computer science for various rea-
sons. Aside from its theoretical importance, the GW algorithm is
one of the few approximation algorithms which actually works in
practice!
76 mathematical programming

5.1 Approximation algorithms

In order to understand this mysterious business of algorithms

which “surprisingly work4 in practice”, let us see what an approx- 4
The literature is full of approximation
imation algorithm is. Suppose we are trying to solve an instance I algorithms which prove a theoretical
approximation property, but have
of a minimization problem P ≡ min f (I , x ) having minimum ob- never been tested in practice. Every
x ∈X
time I can, I ask presenting authors
jective function value f ∗ . We design an algorithm A, but we cannot
of new approximation algorithms
prove it will find an exact global optimum,5 so it finds a feasible whether they have tested their algo-
point x 0 ∈ X with objective function value f 0 > f ∗ . If we can prove rithms computationally: more often
f0 than not, the answer is “why? what is
that A yields a ratio f ∗ which is bounded above by a quantity φ for the point in that?”.
all instances of P, then A is a φ-approximation algorithm for P. 5
In other words, the algorithm is not
Now let I be an instance of a maximization6 problem P ≡ exact.
6
Since Max Cut is a maximization
max f (I , x ) with optimal objective function value f ∗ . As before, we problem, this is the relevant point of
x ∈X
view.
have an algorithm A which is not exact, so it finds a feasible point
f0
x 0 ∈ X with value7 f 0 < f ∗ . If we can prove that A yields a ratio f ∗ 7
Note the inequality sense has
which is bounded below by a quantity φ for all instances of P, then changed, since we are maximizing.

A is a φ-approximation algorithm for P.

5.1.1 Approximation schemes

The quantity φ is a function φ(|I|) of the size taken to store the
instance I . The best case occurs when φ is a constant, the closer to
1 the better. Among the constant approximation algorithms, those
giving the best solutions are those which, for any given ε > 0,
provide an approximation ratio 1 + ε (for minimization) and 1 − ε
(for maximization) in polynomial time. Such algorithms are called
polynomial-time approximation schemes (PTAS).
In general, we only consider approximation algorithms which
terminate in polynomial time. The reason is that simple, brute-force
exponential time exact algorithms exist to solve any NP-complete
problems and many interesting NP-hard problems. Forsaking an
exactness guarantee is customarily considered acceptable if, in
exchange, we can find an approximate solution in polynomial time.
PTAS, specifically, can take as much as φ(|I|1/ε ) time to termi-
nate, where φ is a polynomial. Obviously, as ε becomes smaller, 1/ε
can become very large. Those PTAS which run in polynomial time
of both |I| and 1/ε are called fully polynomial time approximation
schemes, or FPTAS.

5.2 Randomized algorithms

A randomized algorithm employs a source of randomness as part

of its input. We are interested in Monte Carlo randomized algo-
rithms, which may output a wrong result with bounded probability.
In order to make this statement more precise, consider a decision
problem P, and an algorithm R which, when presented with a
NO instance of P, outputs NO with certainty, but when presented
with a YES instance, only outputs YES with probability pR >
1
2 . The class of all such problems is called RP (which stands for
 maximum cut 77

“randomized polynomial time”).

How can R help solve an instance I of a problem P? You simply
run R many times. For example, suppose you run it m times on I .
If you get m NO answers, the probability that the answer is YES is
the same probability p̄ that R gets the answer wrong m times. Since
the probability of R(I) being wrong is 1 − pR < 12 , we have

1
p̄ < ,
2m

which can be driven down to zero as closely as we want.8 By con- 8

Note that we could replace 12 with
trast, if R(I) = YES on at least one run, then I must be necessarily any other δ < 1, and still get lim δm .
m→∞
In this sense, taking δ = 0.5 is arbi-
a YES instance, for if it were NO, then R would output NO with trary.
certainty.

5.2.1 Randomized approximation algorithms

However, in this chapter we discuss Max Cut, which is a maxi-
mization problem rather than a decision problem. A randomized
algorithm R in this setting would simply yield an optimum with a
certain probability pR > 0. Unfortunately, the above argument no
longer works: since, in general, we do not know the optimal objec-
tive function value, we cannot tell whether the solution provided
by R is an optimum or not. Even if we go through m independent
runs of R and take the largest value, we cannot say much about
global optimality guarantees.
We can, however, exploit an upper bound U to the optimal ob-
jective function value f ∗ : if we can prove that R yields an average
objective function value f¯ = δU (for some 0 < δ ≤ 1), then, because
U ≥ f ∗ , it follows that Figure 5.2: Karger’s randomized
algorithm for Min Cut contracts edges
f¯ ≥ δ f ∗ . at random until only one edge remains
— which corresponds to a cut in the
This means that, on average, R performs at least as well as a δ- original graph. This extremely simple
approximation algorithm. algorithm returns a minimum cut
with probability 2/(n(n − 1)) (where
For minimization problems we can exploit a lower bound L to f ∗ , n = |V |), which is much better than
and prove that the average objective function value of R is f¯ = ∆L the one obtained by sampling a cut at
random (from Wikipedia). Unlike Max
with ∆ ≥ 1. By L ≤ f ∗ it follows that f¯ ≤ ∆ f ∗ , which means Cut, however, a polytime algorithm
that R performs, on average, at least as well as a ∆-approximation for Min Cut is known (Which one?)
algorithm. — this goes to show that randomized
algorithms can sometimes also be
Finally, a randomized optimization algorithm which performs useful for problems which we can
at least as well as a given approximation algorithm on average is a solve “efficiently” (it all depends how
one defines this term!). Write a Python
randomized approximation algorithm. code to evaluate whether the Karger’s
algorithm is better or worse than a
5.2.1 Example deterministic Min Cut algorithm, on a
significant test set.
Consider the following (trivial9 ) randomized approximation algorithm R
for the unweighted10 version of Max Cut: for each i ∈ V ( G ), let i ∈ S 9
This algorithm is trivial because it
with probability 0.5. Let R( G ) denote the cutset size corresponding to the does not even look at the edges of the
graph, which actually determine the
cut S determined by R, and Opt( G ) be the size of the maximum cutset. maximum cutset.
We show that 10
I.e. Max Cut where all weights are
1 one.
E(R( G )) ≥ Opt( G ),
2
78 mathematical programming

where E is the expectation of R( G ) over an infinite number of runs:

E(R( G )) = ∑ Prob((i ∈ S ∧ j 6∈ S) ∨ (i 6∈ S ∧ j ∈ S))

{i,j}∈ E( G )
1
= ∑ 2
{i,j}∈ E( G )
1 1
= | E| ≥ Opt( G ),
2 2
where the second line follows from the first because the only other possibil-
ities for i, j are (i ∈ S ∧ j ∈ S) ∨ (i 6∈ S ∧ j 6∈ S), and they are equally
likely, and the third line follows because no cutset can have size larger than
the total number of edges. 2

5.3 MP formulations
Figure 5.3: You should be all familiar
Let us see a few MP formulations for the Max Cut problem.
with the Max Flow = Min Cut theo-
rem, which states that the maximum
flow (see Sect. 1.7.2) in a directed
5.3.1 A natural formulation
graph with given arc capacities is the
same as the capacity of a minimum
We start with a “natural” formulation, based on two sets of decision cut. Explain why we cannot solve Max
variables, both for cut and cutset. Cut on an undirected graph by simply
replacing each edge by two antipar-
• Let V be the vertex set and E the edge set. allel arcs with capacities equal to the
negative weights, and then solving a
• Let w : E → R+ be the edge weight parameters. Max Flow LP to find the minimum
cut. (Picture from faculty.ycp.edu).
• For each {i, j} ∈ E let zij = 1 if {i, j} is in the maximum cutset,
and 0 otherwise (decision variables).

• For each i ∈ V, let yi = 1 if i ∈ S, and 0 otherwise (decision

variables).

• The objective function is

max ∑
s,z∈{0,1} {i,j}∈ E
wij zij ,

which maximizes the size of the cutset relative to S.

• The constraints link the meaning of z and s variables:

∀{i, j} ∈ E zij = yi (1 − y j ) + (1 − yi )y j .

Note that the RHS can only be zero or one. If i, j are both in
S or not in S, we have yi = y j = 1 or yi = y j = 0, so the
RHS evaluates to zero, which implies that the edge {i, j} is not
counted in the maximum cutset. If i ∈ S and j 6∈ S, then the first
term of the RHS is equal to 1, and the second is equal to 0, so the
RHS is equal to 1, which implies that the edge {i, j} is counted.
Similarly for the remaining case i 6∈ S and j ∈ S.

This is a Binary Quadratically Constrained Program (BQCP), which

we have not really considered in past chapters. Expanding the RHS
of the constraints yields

yi − yi y j + y j − yi y j = yi + y j − 2yi y j ,
 maximum cut 79

which implies that the quadratic part only arises because of a

product between two binary variables, which we can linearize
exactly by Sect. 3.2.1. So we can write this BQCP as a BLP and solve
it using CPLEX.

5.3.2 A BQP formulation

In the formulation of Sect. 5.3.1 we can simply replace the occur-
rence of zij in the objective function by the RHS yi + y j − 2yi y j , and
obtain:
max ∑ wij (yi + y j − 2yi y j ),
y∈{0,1}n {i,j}∈ E
(5.1)

where n = |V |, which is a BQP.

5.3.3 Another quadratic formulation

We are going to exploit the following equation: Figure 5.4: The Ising model of a
spin glass (from http://www3.nd.
∀y ∈ {0, 1}n 2(yi + y j − 2yi y j ) = 1 − (2yi − 1)(2y j − 1), edu/~mcbg). There are n magnetic
impurities (atoms, represented by
graph vertices V) in an external
which allows us to rewrite Eq. (5.1) as
magnetic field, represented by a
dummy vertex in V. For each i < j
1
2 {i,j∑
max wij (1 − (2yi − 1)(2y j − 1)). in V the magnetic interaction is Jij .
y∈{0,1}n }∈ E The model dictates that the spin
xi ∈ {−1, 1} of each atom i is such
Now, if yi ∈ {0, 1} for any i ∈ V, then xi = 2yi − 1 is a variable that the energy of the system, given
by − ∑i< j Jij xi x j , is minimum. A few
which ranges in {−1, 1}, since xi = 1 if yi = 1 and xi = −1 if yi = 0. further steps transform this to a Max
Thus, we can formulate the Max Cut problem as: Cut problem.

1
2 {i,j∑
max wij (1 − xi x j ). (5.2)
x ∈{−1,1}n }∈ E

Note that Eq. (5.2) is also a natural formulation (see Fig. 5.4): for
each i ∈ V, let xi = 1 if i ∈ S and xi = −1 if i ∈ V r S. Then, for each
{i, j} ∈ E, xi x j = −1 if and only if {i, j} is in the maximum cutset,
and the weight of the corresponding edge is 12 wij (1 − (−1)) = wij .

5.3.4 An SDP relaxation

In this section we derive a well-known SDP relaxation of the Max
Cut problem. The strategy is as follows: we reformulate the prob-
lem exactly to a matrix problem with a rank constraint, which we
then relax.
First, we slightly change the input of the problem: instead of a
simple, undirected, edge-weighted graph G = (V, E), we consider
a complete simple directed graph Ḡ = (V, A), where wij = w ji = 0
for each {i, j} 6∈ E. Since we are adding the weight wij twice (every
edge {i, j} ∈ E is replaced by two antiparallel arcs (i, j) and ( j, i ) in
A), we have to multiply the objective function by another factor 12 .
This means we can write another quadratic Max Cut formulation
as follows:
1
max ∑ wij (1 − xi x j ).
x ∈{−1,1}n 4 i,j∈V
(5.3)
80 mathematical programming

We now carry out a reformulation which we have never seen

before, and which might appear very weird: we replace each scalar
decision variable xi (for i ∈ V) by a vector of decision variables
vi ∈ Sn−1 , i.e. vi ∈ Rn and kvi k2 = 1. Every product xi x j is replaced
by the scalar product vi · v j , thereby obtaining:

1
4 i,j∑
max wij (1 − vi · v j ). (5.4)
v ∈Sn −1 ∈V

Note that any solution to Eq. (5.4) is an n × n real symmetric matrix

V having vi as its i-th column. It should be easy to see that this
is not an exact reformulation. We can make it exact, however, by Figure 5.5: Replacing a scalar variable
enforcing the condition that, for any solution V, there exists a with a vector variable gives more de-
grees of freedom, so the feasible region
column vector u, with each component in S0 , such that uu> = VV > .
is enlarged. In other words, it is a re-
If this is the case, then for each i 6= j we have vi> v j = vi · v j = ui u j , laxation of the original problem (from
and since S0 = {−1, 1}, we have recovered Eq. (5.4). some lecture notes of R. O’Donnell).

By Prop. 5.3.1, it suffices that rank V = 1.

5.3.1 Proposition
For any n × n real symmetric matrix V, rank V = 1 if and only if there
exists a vector u ∈ Rn such that uu> = VV > .

Proof. Suppose V has rank 1. This happens if and only if every

column is a scalar multiple of a single column, i.e. we can write V
as (u1 t u2 t · · · un t) for some column vector t ∈ Rn which we can
choose as having unit Euclidean norm without loss of generality.11 11
Why is there no loss of generality in
In turn, this is equivalent to stating that the (i, j)-th component of assuming ktk2 = 1?

VV > is ui u j ktk22 = ui u j , which happens if and only if VV > = uu> .

The converse direction is trivial.12 2 12
Why? Prove it!

We now consider the Gram matrix of V (see Sect. 2.10), denoted

Gram(V ), i.e. the n × n matrix having vi · v j as its (i, j)-th compo-
nent. The Gram matrix has two remarkable properties, given in
Prop. 5.3.2 below.

5.3.2 Proposition
For any real n × n symmetric matrix V, we have: (i) Gram(V )  0; (ii)
rank Gram(V ) = rank V.

Proof. First, we remark that Gram(V ) = V > V = VV > .

(i) For any x ∈ Rn , we have

x > Gram(V ) x = x > V > Vx = (Vx )> Vx = kVx k2 ≥ 0,

implying Gram(V )  0, i.e. the Gram matrix is PSD.13 13

The converse also holds: any PSD
(ii) For any matrices A, B we know that the columns of AB matrix is the Gram matrix of some
other matrix. Prove it.
are linear combinations of the columns of A, and the rows of AB
are linear combinations of the rows of A: hence, the columns of
Gram(V ) = V > V are a linear combination of the columns of V > ,
i.e. the rows of V, and the rows of Gram(V ) are a linear combina-
tion of the rows of V. Since the row rank and column rank of any
matrix coincide, Gram(V ) has the same rank as V, as claimed. 2
 maximum cut 81

5.3.3 Corollary
If X = ( Xij ) = Gram(V ) and rank X = 1, then there exists a vector u
such that Xij = ui u j .

Proof. Since rank X = 1 and X is the Gram matrix of V, then

rank V = 1 by Prop. 5.3.2, which, by Prop. 5.3.1 implies the existence
of u such that uu> = VV > = Gram(V ) = X. 2

By Cor. 5.3.3, we can reformulate Eq. (5.4) as:

1
∑ wij (1 − Xij )

max 4 
i,j∈V




Diag( X ) = I (5.5)
X  0 


rank X = 1,


where Diag( X ) is the matrix consisting of the diagonal entries of X,

and I is the identity matrix.
Lastly, we look at the objective function of Eq. (5.5) more closely:

∑ wij (1 − Xij )
i,j∈V

= ∑ wij 1 − ∑ wij Xij

i,j∈V i,j∈V
Diag( X ) = 1
⇒ ∀i ∈ V Xii = 1 = ∑ wij Xii − ∑ wij Xij
i,j∈V i,j∈V
!
= ∑ ∑ wij Xii − A • X
i ∈V j ∈V
= D•X−A•X
= ( D − A) • X,

where A = (wij ) is the weighted adjacency matrix of the input

digraph Ḡ, and D is the matrix having ∑ wij on the i-th diagonal
j ∈V
element, and 0 elsewhere. Note that the matrix L = D − A is
commonly known as the weighted Laplacian of the graph14 G. 14
Why is the weighted Laplacian
Our matrix formulation of the Max Cut is therefore: defined on undirected graphs, yet we
refer it to an input which is a directed
1
 graph?
max 4L • X 


Diag( X ) = I

(5.6)
X  0  

rank X = 1.


An SDP relaxation of Eq. (5.6) can now be easily derived by drop-

ping the constraint rank X = 1:

1
max 4L •
X 

Diag( X ) = I (5.7)

X  0. 

As mentioned in Sect. 2.10, SDPs can be solved in polynomial time

to any desired ε accuracy by means of IPM.
82 mathematical programming

5.4 The Goemans-Williamson algorithm

Since SDPs can be solved in polytime to any desired accuracy,15 one 15

We remark that there exists an
could dream of an ideal world where one can just efficiently solve IPM for SDP which converges in
O(|I|3.5 log(1/ε)) [Boyd and Van-
the SDP relaxation of any instance I of Max Cut, and the solution denberghe, 2004], where |I| is the
X ∗ simply happens to have rank 1: then we can just pick the correct size of the SDP instance. Does this
make IPM an FPTAS? It would ap-
vector x: since, as remarked, k xi k2 = 1 reduces to xi ∈ {−1, 1} for pear so, but no-one mentions it in
each i ≤ n, we can simply read the maximum cutset off the SDP the literature. As far as I can tell, it
solution. We live, alas, in an imperfect world where relaxations is because the ε tolerance is not only
applicable to optimality, but also to
are often inexact, however. The main innovation of the Goemans- feasibility, i.e. IPMs for SDP might
Williamson algorithm (GW) is a randomized algorithm for finding yield a solution with eigenvalues ≥ −ε,
or satisfying the constraints with a ε
a decently-sized cutset from X ∗ . This algorithm is often called error. The definition of FPTAS applies
“randomized rounding”, since some Xij∗ might well be fractional in the approximation to the objective
Eq. (5.7), whereas we would need it to be either 1 or −1 in order for function, but requires strict feasibility.

X ∗ to represent a valid solution of Eq. (5.2).

In summary, the GW is as follows:

1. Solve the SDP in Eq. (5.7) to find X ∗

2. Perform “randomized rounding” to find x 0 ∈ {−1, 1}n represent-
ing a “good” cut.

By “good”, the GW algorithm means that, on average, the random-

ized rounding algorithm will yield a cutset having size α Opt(I),
where α > 0.878. Since 0.878 is quite close to 1, the GW is, at least
theoretically, a pretty good randomized approximation algorithm.

5.4.1 Randomized rounding

Consider the SDP relaxation solution X ∗ : since it is a symmetric
matrix, it has a real factorization P> ΛP, where Λ has all of the
eigenvalues of X ∗ on the diagonal (and zero elsewhere), and P
is the unitary matrix formed by the corresponding eigenvectors,
arranged column-wise. Since X ∗ is PSD, Λ ≥ 0, which means that
√
the square root matrix Λ is real. So
√ >√ √ >√
X ∗ = P> ΛP = P> Λ ΛP = ( ΛP) ΛP = V > V,
√
where V = ΛP, with vi being the i-th column of V.
Note that, since X ∗ satisfies Diag( X ∗ ) = 1 (i.e. Xii∗ = 1 for each
i ≤ n), we have Xii∗ = vi · vi = kvi k2 = 1, which implies that the
vectors vi are on Sn−1 . Our job would be over if we could “wisely” Figure 5.6: Factorizing the SDP
choose a hyperplane H, passing through the origin, which sepa- solution yields a bunch of vectors
on the spherical surface (from some
rates the vectors vi into two parts: on the one side, say whenever lecture notes of R. O’Donnell).
i ∈ S0 , we set all i’s such that xi0 = 1, and on the other side we put
the rest, namely i’s with xi0 = −1. By “wisely”, we mean of course
that this magical hyperplane should yield a Max Cut solution x 0
which is at least as good as α times the maximum cutset.
Geometrically, let r be the (unit) normal vector to the magical
hyperplane: then those vectors vi which form an angle ξ between
−90 and 90 degrees with H will be in S0 , and those which form an
angle ξ between 90 and 270 degrees with H will not be in S0 . The
 maximum cut 83

algebraic connection between an angle and two vectors is the scalar

product: specifically,

∀i ≤ n r · vi = kr k2 kvi k2 cos(ξ ) = cos(ξ ),

since r and all of the vi ’s are unit vectors. Thus, i ∈ S0 if and only if
r · vi ≥ 0, which implies that we can define

x 0 = sgn(r > V ),

with the convention that sgn(0) = 1.

Now, the average value of the cutset size obtained by random-
ized rounding is:
 

E  ∑ wij  = ∑ wij Prob(xi0 6= x0j ). (5.8) Figure 5.7: For every pair of vec-
 
tors, we can look at the situation
i ∈S0 i< j
j6∈S0 in 2D (from some lecture notes of
R. O’Donnell).
So we need to evaluate the probability that, given a vector r picked
uniformly at random on Sn−1 , its orthogonal hyperplane H forms
angles as specified above with two given vectors vi and v j .
Since we are only talking about two vectors, we can focus on the
plane η spanned by those two vectors. Obviously, the projection of
Sn−1 on η is a circle C centered at the origin. We look at the line λ
on η (through the origin) spanned by r. The question, much simpler
to behold now that it is in 2D, becomes: what is the probability that a
line crossing a circle separates two given radii vi , v j ?
This question really belongs to elementary geometry (see
Fig. 5.7). In order to separate vi , v j , the radius defined by λ has to
belong to the smaller slice of circle delimited by vi and v j . The prob-
ability of this happening is the same as the extent of the (smaller)
angle between vi and v j divided by 2π. Only, λ really identifies a di-
ameter, so we could turn λ around by π and get “the other side” of
λ to define another radius. So we have to multiply this probability
by a factor of two:

arccos(vi · v j )
∀i < j Prob( xi0 6= x 0j ) = . (5.9)
π
Since we do not know Opt(I), in order to evaluate the approxi-
mation ratio of the GW we can only compare the objective function
value of the rounding x 0 with the SDP relaxation value, which, as
an upper bound to Opt(I), is such that
Figure 5.8: When rounding is worse
1 1
∑
2 i< j
wij (1 − xi0 x 0j ) ≤ Opt(I) ≤ ∑ wij (1 − vi · v j ).
2 i< j
(5.10) than SDP: how much worse? (From
some lecture notes of R. O’Donnell)

By Eq. (5.8)-(5.9), the average value of 12 (1 − xi0 x 0j ) over all i < j is

1
· v j )). Now we look at Eq. (5.10) term-wise: how much
π (arccos( vi
worse can arccos(vi · v j )/π be with respect to (1 − vi · v j )/2?
Let us look at vi · v j as a quantity ρ, and plot16 the two functions 16
See Fig. 5.8.

1 1
arccos ρ and (1 − ρ ),
π 2
84 mathematical programming

for ρ ∈ [−1, 1]. Note that for ρ ≥ 0, π1 arccos ρ ≥ 21 (1 − ρ), which

implies that randomized rounding performs better than the SDP
solution. For ρ < 0 this is no longer the case. How bad can random-
ized rounding perform with respect to SDP? The minimum of the
gap between the two functions has value:

(arccos ρ)/π
α = min ≈ 0.87854.
ρ∈[−1,1] (1/2)(1 − ρ)

This means that, on each pair i, j ∈ V, randomized rounding can

do no worse that α times the solution of the SDP. Hence:
!
1 α
E ∑ wij (1 − xi0 x 0j ) ≥
4 i,j∑
wij (1 − Xij∗ ) ≥ α Opt(I), (5.11)
4 i,j∈V ∈V

as claimed.

5.4.2 Sampling a uniform random vector from Sn−1

It only remains to discuss a method for sampling uniform random
vectors from the surface of a unit sphere in Rn . A very well known
method consists in sampling n independent values r10 , . . . , rn0 from a
>
standard normal distribution N (0, 1), then letting r 0 = (r10 , . . . , rn0 )
0
and r = krr0 k . By the lemma below, r is uniformly distributed on
2
Sn −1 .

5.4.1 Lemma
The vector r is uniformly distributed on Sn−1 .
Figure 5.9: Most “wild guess methods”
Proof. Since r10 , . . . , rn0 are independently sampled, their joint proba- for uniformly sampling on a sphere
bility distribution function is simply the product of each individual turn out to be incorrect (above) or,
distribution function: if correct (below), inefficient (from
www.bogotobogo.com).
1 0 2 1 0
∏ √2π e(ri ) /2 = (2π ) n/2
e−kr k2 /2 .
i ≤n

The probability that r 0 is in some measurable set A is:

1
Z
0
pA = e−kr k2 /2 dr 0 .
(2π )n/2 A

Now consider any rotation matrix U (such that U > U = I) applied

to A. The probability that r 0 is in U > A is:

1
Z
0
pU A = e−kr k2 /2 dr 0
(2π )n/2 U> A
1
Z
> r0 k
= e−kU 2 /2 dr 0
(2π )n/2 A
1
Z
> Uk 0k
= e−kU 2 kr 2 /2 dr 0
(2π )n/2 A
1
Z
0
= e−k I k2 kr k2 /2 dr 0
(2π )n/2 A
1
Z
0
= e−kr k2 /2 dr 0 = p A .
(2π )n/2 A
 maximum cut 85

Thus, if s 6= t are points in Sn−1 , the probability that r 0 = s is the

same as the probability that r 0 = t. Hence r, the normalized version
of r 0 , is uniformly distributed on the unit sphere, as claimed. 2

5.4.3 What can go wrong in practice?

We said at the beginning of this chapter that GW is an approxima-
tion algorithm which actually works in practice. We shall see this in
Sect. 5.5. One word of caution, however: as mentioned in Sidenote
15, it is possible that SDP solvers would find slightly infeasible
solutions, due to their error tolerance, which is in turn due to float-
ing point computations. So X ∗ might have some slightly negative
eigenvalues, or not satisfy Diag( X ∗ ) = 1 exactly. This has to be dealt
with at the implementation level, rather than algorithmically.
For example, the closest PSD matrix A to a given symmetric
matrix B can be obtained by factorization: let B = P> ΛP, where Λ
is a diagonal matrix with the eigenvalues (ordered from largest to
smallest) on the diagonal, and the columns of P are the correspond-
ing eigenvectors. If Λii ≥ 0 for all i ≤ n, we have a certificate that
B is PSD. However, if there is ` ≤ n such that Λ` < 0 (even though
perhaps |Λ` | is tiny), then we define Λ+ as Λ, with all eigenvalues
from the `-th to the n-th replaced by zeros. We then compute17 17
Note that, since this computation
A = P> Λ+ P. also involves floating point operations,
some error might make some eigen-
Since GW really only relies in the factorization of X ∗ into V > V, value of A slightly negative: this is
you can also address these issues by designing a factorization a game of hide-and-seek, where you
have to artfully tune the number of
algorithm which is robust to slightly negative eigenvalues in X ∗ . very small eigenvalues which you zero
Most implementations (e.g. in NumPy or Matlab) are not robust in in Λ+ .
this sense.

5.5 Python implementation

We now present and discuss a Python program which solves:

• the MIQP formulation in Eq. (5.1), using Pyomo calling CPLEX18 ; 18

Note that the objective function in
Eq. (5.1) is quadratic: even though
• the MILP formulation derived from applying the binary variable CPLEX excels at solving MILPs, it
product reformulation (Sect. 3.2.1) to Eq. (5.1), using Pyomo can solve convex MIQPs quite well
(including those with convex nonlinear
calling CPLEX; constraints). I know, I know: the
objective of Eq. (5.1) is not even
• the SDP relaxation in Eq. (5.7) using the alternative19 Picos MP
convex! Ever since version 12.6, CPLEX
modelling environment in Python, which interfaces to SDP has been able to solve also nonconvex
solvers (through the Python library CvxOpt); MIQPs with nonconvex objective
functions (the constrains have to be
• the Max Cut problem using our own implementation of the GW convex quadratic, however), by means
of a special-purpose sBB-type solver.
algorithm.
We need an alternative because
19

Pyomo currently does not interface to

5.5.1 Preamble SDP solvers.

We import the following libraries:

import time
import sys
86 mathematical programming

import math
import numpy as np
from pyomo.environ import *
from pyomo.opt import SolverStatus, TerminationCondition
import pyomo.opt
import cvxopt as cvx
import cvxopt.lapack
import picos as pic
import networkx as nx

We first discuss the standard libraries. We use the time library to

collect CPU timings in order to perform an empirical comparison of
the different solution approaches. The sys library allows us to pass
a command line argument to our Python program. The math library
gives direct access to functions such as sqrt, which computes the
square root of its argument. The numpy library is necessary for
many floating point computations, including linear algebra; note
we import numpy as the shortened string np, which makes our code
somewhat neater.
We also import some specific MP-related libraries. We already
saw the import sequence for Pyomo. We then import the cvxopt
library and its cvxopt.lapack linear algebra module. We do not use
these libraries explicitly, but they are used by picos, the MP python
modelling environment which is specifically suitable for convex
optimization, and can interface with SDP and SOCP solvers. Finally,
we store and manipulate graphs using the networkx library, which
we import as the shortened string nx.
We rely on a few globally defined variables:

GWitn = 10000
MyEpsilon = 1e-6
cplexTimeLimit = 100

GWitn is the maximum number of iterations of the randomized

rounding phase of the GW algorithm. MyEpsilon is used as a float-
ing point error tolerance. cplexTimeLimit is the maximum number
of seconds of user20 CPU that CPLEX is allowed to run for. Note 20
CPU timings in most operating
that CPLEX, as well as many other MP solvers, does not check the systems (OS) nowadays are reported
as either user, system or total. The
system clock very often, probably in an attempt to devote every former is the actual amount of seconds
single CPU cycle to finding the solution efficiently. As a result, even that the CPU devotes to running the
specific given task being monitored,
if you set a time limit of 100 seconds, CPLEX might well go on whereas the latter is user+system.
crunching numbers for considerably more time. Usually, people are mostly interested
in user CPU time, since the time the
computer spends doing something
5.5.2 Functions else is not relevant to code efficiency —
one might consider devoting a whole
Next, we briefly introduce the functions we use. In create_maxcut_miqp computer to running the relevant
we take an undirected graph G (in the form of a networkx object) task in single-user mode. With the
advent of multi-core computers, these
and use the Pyomo library to model a MIQP formulation of Max
time statistics have become fuzzier:
Cut. is the time taken to transfer control
or register data from one core to
def create_maxcut_miqp(G): another classified as system or user?
maxcut = ConcreteModel() After all the parallelized code would
# 0-1 vars: 0 if node in S, 1 if in \bar{S} not work without these control and
maxcut.y = Var(G.nodes(), within = Binary) data transfers. I believe different
OSes employ different definitions of
user and system times in multi-core
computers.
 maximum cut 87

# objective: max sum_{ij in E} w_ij * (y_i + y_j - 2 y_i*y_j)

def max_cut(model):
return sum(G[i][j][’weight’] * (model.y[i] + model.y[j] - \
2*model.y[i]*model.y[j]) for (i,j) in G.edges())
maxcut.maxcutobj = Objective(rule = max_cut, sense = maximize)
return maxcut

In create_maxcut_milp we do the same, but we add Fortet’s

inequalities to MIQP formulation, and linearize the products yi y j ,
replacing each bilinear term with a new variable zij (Sect 3.2.1).
def create_maxcut_milp(G):
maxcut = ConcreteModel()
# 0-1 vars: 0 if node in S, 1 if in \bar{S}
maxcut.y = Var(G.nodes(), within = Binary)
# linearization variables z_ij = y_iy_j
maxcut.z = Var(G.edges(), within = PercentFraction)
# objective: max sum_{ij in E} w_ij * (y_i + y_j - 2 z_ij)
def max_cut(model):
return sum(G[i][j][’weight’] * (model.y[i] + model.y[j] - \
2*model.z[i,j]) for (i,j) in G.edges())
maxcut.maxcutobj = Objective(rule = max_cut, sense = maximize)
# Fortet’s linearization constraints
def fortet1(model, i,j):
return model.z[i,j] <= model.y[i]
maxcut.fortet1 = Constraint(G.edges(), rule = fortet1)
def fortet2(model, i,j):
return model.z[i,j] <= model.y[j]
maxcut.fortet2 = Constraint(G.edges(), rule = fortet2)
def fortet3(model, i,j):
return model.z[i,j] >= model.y[i] + model.y[j] - 1
maxcut.fortet3 = Constraint(G.edges(), rule = fortet3)
return maxcut

For the SDP relaxation of Max Cut, we employ a different MP

modelling environment in Python, namely Picos. Note that the
argument of create_maxcut_sdp is not the graph, but its Laplacian
matrix. You can hopefully make out what the various instructions
do by the comments.

def create_maxcut_sdp(Laplacian):
n = Laplacian.shape[0]
# create max cut problem SDP with picos
maxcut = pic.Problem()
# SDP variable: n x n matrix
X = maxcut.add_variable(’X’, (n,n), ’symmetric’)
# constraint: ones on the diagonal
maxcut.add_constraint(pic.tools.diag_vect(X) == 1)
# constraint: X positive semidefinite
maxcut.add_constraint(X >> 0)
# objective function
L = pic.new_param(’L’, Laplacian)
# note that A|B = tr(AB)
maxcut.set_objective(’max’, L|X)
return maxcut

In the GW algorithm the output of the SDP relaxation phase is

X ∗ , but the input to the randomized rounding phase is a matrix
factor V of X ∗ . The rank_factor function computes this factor.
88 mathematical programming

Note that we try and bound the “negative zero” eigenvalues due to
floating point errors.

def rank_factor(X, rk):

n = X.shape[0]
evl, evc = np.linalg.eigh(X) # extract eigeninfo
# get closest nonneg eigenvals (small neg number --> 0)
evl[evl < 0] = 0
if rk < n:
V = np.transpose(evc[:,-rk:])
for k in range(rk):
V[k] *= math.sqrt(evl[-rk-1])
else:
V = np.transpose(evc[::-1])
for k in range(n):
V[k] *= math.sqrt(evl[::-1][k])
return V

The function closest_sdp removes small negative eigenvalues

from a matrix X which should be PSD. We factor X it using eigen-
vectors and eigenvalues, we replace negative eigenvalues by zeros,
then we multiply the factor and its transpose to retrieve an updated
matrix X which is PSD. This somehow replicates some of the in-
structions found in the function rank_factor above. The point is
that, as emphasized in Sect. 5.4.3, multiplying the factors involves
many floating point operations, which might introduce new float-
ing point errors. Replicating some of these “cleaning” actions helps
reduce the chances of an error in the call to square root functions.21 21
Honestly, I’m not really sure this
further “cleaning step” is wholly
justified — I must have inserted this
def closest_psd(X): code in consequence of some negative
square root error occurring in the code
evl, evc = np.linalg.eigh(X) # extract eigeninfo
on some instance I was testing. Try
# get closest nonneg eigenvals (some -1e-27 --> 0) a version of the program where you
evl[evl < 0] = 0 do not call this function, and see what
return evc.T.dot(np.diag(evl).dot(evc)) happens.

We now come to the randomized rounding algorithm: the func-

tion gw_randomized_rounding is wholly commented, so it should be
easy to understand.

def gw_randomized_rounding(V, objX, L, iterations):

n = V.shape[0]
# repeat max iterations times
count = 0
obj = 0
print "maxcut(GW/rnd): improv_itn objfun"
while (count < iterations):
r = np.random.normal(0,1,n) # rnd unfrm vector on unit sphere
r /= np.linalg.norm(r) # normalization
x = np.sign(np.dot(V,r)) # signs of V . r
x[x >= 0] = 1 # in case there’s some zeros
o = np.dot(x.T,np.dot(L,x)) # objective function value
if o > obj:
x_cut = x
obj = o
print count, "\t", obj
count += 1
return (x_cut, obj)
 maximum cut 89

5.5.3 Main
We now step through the instructions of the “main” part of our
Python program. First, we read the input file having name given as
the first argument on the comand line.

# read edge list file as input

if len(sys.argv) < 2:
# not enough args on cmd line, error
print "error: need an edge list filename on cmd line"
quit()
# each line of input is arc (i,j) weight w: i j w
G = nx.read_weighted_edgelist(sys.argv[1], nodetype = int)
# make G undirected
G = nx.Graph(G)

For example, the triangle graph with unit weights is:

1 2 1
1 3 1
2 3 1

Secondly, we solve the MIQP formulation using the standard

Pyomo constructs (see Sect. 1.8.3). Figure 5.10: The logo of PICOS (from
picos.zib.de).
miqp = create_maxcut_miqp(G)
solver = pyomo.opt.SolverFactory(’cplexamp’, solver_io = ’nl’)
solver.options[’timelimit’] = cplexTimeLimit
solver.options[’mipdisplay’] = 2
t0 = time.time() # start the stopwatch
results = solver.solve(miqp, keepfiles = False, tee = True)
t1 = time.time() # stop it
miqpcpu = t1 - t0 # this is the time taken by the solver
miqp.solutions.load_from(results) # this is an indicator vector
# retrieve the set of the indicator vect. as a dictionary (key:val)
ymiqp = {i:miqp.y[i].value for i in G.nodes()}
miqpcut = [i for i,y in ymiqp.iteritems() if y == 1]
miqpobj = miqp.maxcutobj()

Note the use of the time Python library: we call time.time()

around the solver.solve() command, then we take the difference
of the two timings to compute the duration of solver.solve().
Third, we solve the MIQP formulation using the standard Pyomo
constructs (see Sect. 1.8.3).

milp = create_maxcut_milp(G)
t0 = time.time()
results = solver.solve(milp, keepfiles = False, tee = True)
t1 = time.time()
milpcpu = t1 - t0
milp.solutions.load_from(results)
ymilp = {i:milp.y[i].value for i in G.nodes()}
milpcut = [i for i,y in ymilp.iteritems() if y == 1]
milpobj = milp.maxcutobj()

Fourth, we solve the SDP relaxation. Since we modelled the SDP

using Picos, the instructions are different.

L = np.array(1/4.*nx.laplacian_matrix(G).todense()) # compute Laplacian

sdp = create_maxcut_sdp(L) # create max cut SDP relaxation
90 mathematical programming

t0 = time.time() # start watch

sdp.solve(verbose = 0) # solve
t1 = time.time() # stop watch
sdpcpu = t1 - t0 # time difference
sdpobj = sdp.obj_value() # retrieve obj fun value
Xsdp = closest_psd(np.array(sdp.get_valued_variable(’X’))) # retrieve solution

Fifth, we run factor Xsdp and apply randomized rounding.

t0 = time.time()
N = len(G.nodes())
V = rank_factor(Xsdp, N)
(gwcut,gwobj) = gw_randomized_rounding(V, sdp.obj_value(), L, GWitn)
t1 = time.time()
gwcpu = t1 - t0

Lastly, we output our comparison statistics: objective function

value and CPU time.

print "maxcut(out): MIQPobj=", miqpobj, ", MIQPcpu=", miqpcpu

print "maxcut(out): MILPobj=", milpobj, ", MILPcpu=", milpcpu
print "maxcut(out): SDPobj=", sdpobj, ", SDPcpu=", sdpcpu
print "maxcut(out): GWobj=", gwobj, ", GWcpu=", gwcpu

This code resides in a text file called maxcut.py, and can be run
using the command line

python maxcut.py file.edg

where file.edg is the instance file with the edge list, as detailed
above (two integers and a floating point number per line, where the
integers are the indices of the vertices adjacent to each edge and the
floating point is its weight).

5.5.4 Comparison on a set of random weighted graphs

I have created22 a set of file in .edg format, each storing a weighted 22
You should do this too! A piece of
graph generated using the Erdős-Renyi random model. This is advice: don’t do this by hand — use
Python, or another scripting language.
as follows: given an integer n and a real p ∈ [0, 1], generate a
graph with n vertices, where each edge has probability p of being
created. I set the range of n to {10, 20, . . . , 90}, and the range of p to
{0.3, 0.5, 0.7}.
I then launched the maxcut.py Python program on each of the
files, saved the output to a similarly-named .out file, and then
proceeded to collect23 the output data in the table below. Note 23
Again, don’t do this by hand! Parse
that, since the time limit for CPLEX is set to 100s, CPLEX does the .out file using a program, and
output a tabular format — or simply
not have the time to solve all instances will not be solved to global edit the maxcut.py script to write its
optimality. output in tabular format.
The first two columns of the table contain the generation param-
eters for the random graph. The following four columns contain
the objective function values of MIQP, MILP, SDP and randomized
rounding (RR) applied to the SDP solution. The last four columns
contain the user CPU time taken to solve MIQP, MILP, SDP and
perform RR (the time for RR includes factoring the output of the
SDP solution). Thus, the time taken to run the GW algorithm is
 maximum cut 91

represented by the (row-wise) sum of the values in the last two

columns. Remember that only MIQP, MILP and RR actually output
a feasible solution for the Max Cut (the SDP relaxation outputs a
matrix which in general does not have rank 1).
It is clear that the GW makes a very reasonable concession24 to 24
Is this always the case? Generate
the quality of the obtained cut, in exchange for huge savings on other kinds of graphs and see for
yourself! For example, how about
the CPU time. This is why I wrote that this is an approximation square mesh grid graphs with unit
algorithm which actually works in practice! weights?

instance objective function value user CPU time

n p MIQP MILP SDP RR MIQP MILP SDP RR
10 0.3 7.82 7.82 8.38 7.82 0.1 0.0 0.1 0.2
10 0.5 9.87 9.87 10.15 9.87 0.1 0.1 0.0 0.2
10 0.7 15.09 15.09 15.20 15.09 0.1 0.1 0.0 0.2
20 0.3 28.54 28.54 28.93 27.29 0.1 0.1 0.1 0.2
20 0.5 36.55 36.55 37.94 35.40 0.2 0.2 0.1 0.2
20 0.7 44.88 44.88 46.44 44.44 0.2 0.3 0.1 0.2
30 0.3 52.80 52.80 55.14 47.82 0.3 0.5 0.1 0.2
30 0.5 74.82 74.82 77.01 68.36 0.5 1.3 0.1 0.2
30 0.7 100.49 100.49 102.67 93.31 1.9 3.4 0.1 0.2
40 0.3 85.55 85.55 89.07 75.80 1.1 2.0 0.1 0.2
40 0.5 134.52 134.52 138.69 124.54 4.4 6.4 0.1 0.2
40 0.7 175.29 175.29 179.88 162.63 46.2 100.2 0.1 0.2
50 0.3 148.99 148.99 151.83 127.68 2.6 3.8 0.1 0.2
50 0.5 206.02 206.02 212.62 182.69 21.6 33.7 0.1 0.2
50 0.7 263.75 263.20 269.46 243.14 100.1 100.2 0.1 0.2
60 0.3 201.12 201.12 209.16 178.03 38.0 100.1 0.1 0.3
60 0.5 281.69 280.52 291.98 256.08 100.1 100.2 0.2 0.3
60 0.7 370.38 367.92 382.96 348.29 100.1 100.3 0.2 0.2
70 0.3 253.52 252.85 262.39 223.67 100.1 100.2 0.1 0.3
70 0.5 394.33 383.24 403.63 352.04 100.1 100.3 0.1 0.3
70 0.7 497.64 487.58 509.99 461.83 100.1 100.4 0.2 0.3
80 0.3 309.52 309.52 321.58 276.06 100.1 100.2 0.1 0.3
80 0.5 490.93 493.07 514.59 455.93 100.1 100.3 0.2 0.4
80 0.7 648.89 628.01 665.94 603.25 100.2 100.5 0.2 0.3
90 0.3 405.39 400.41 424.94 355.79 100.1 100.2 0.2 0.3 Implement the trivial randomized
90 0.5 608.89 590.78 631.28 560.55 100.1 100.5 0.2 0.4 approximation algorithm from Exam-
90 0.7 818.97 800.58 848.35 764.77 100.2 100.5 0.2 0.3 ple 5.2.1, and compare it to the other
algorithms on unweighted Max Cut
instances. What is its performance like,
with respect to RR? Would you define
it as “an algorithm which actually
5.6 Summary works in practice”?

This chapter focuses on the Max Cut problem. It presents applica-

tions, MP formulations and a very famous randomized approxima-
tion algorithm by Goemans and Williamson. Before delving into the
thick of this algorithm, it explains what approximation algorithms
are, what randomized algorithms are, and what the union of the
two concepts is.
6
Distance Geometry David Hilbert
Leopold Kro-
necker
Ernst Zermelo

Distance Geometry (DG) is a sub-field of geometry based on the Kurt Gödel

Adolf Fraenkel
concept of points and distances rather than points, lines, planes, hy- Karl Menger

perplanes or general manifolds. By far the most common setting is

where the distances are Euclidean, although early axiomatizations
of DG worked with general (semi)metrics [Menger, 1931, Blumen-
thal, 1953]. This novel axiomatization of geometry was proposed
at a time where the “old” way of doing mathematics (read: up to
1900) was being formalized1 into our current setting. To be honest, 1
Up to 1900, many proofs had ele-
mathematicians still use a lot of hand-waving [Krantz, 2010], but ments of “hand-waving” in them. At
the end of the 19th century, David
now at least they know they could do things completely formally if Hilbert decided he’d had enough
they wanted to.2 of being unable to prove anyone
wrong formally (specifically Leopold
Kronecker, who kept saying that exis-
tential proofs did not prove anything
6.1 The fundamental problem of DG – only constructive proofs were good),
and decided to use his extensive in-
fluence to steer the community into
The reason why DG and MP are related is through the fundamental defining things formally once and for
problem of DG: all. His valiant attempt succeeded to
a large extent, insofar as we now have
Distance Geometry Problem (DGP). Given a positive integer K and the Zermelo-Fraenkel-Choice (ZFC)
system of axioms, which appear to be
a simple undirected weighted graph G = (V, E, d) where d : E → R+
enough to do almost anything of inter-
is an edge weight function, is there a realization3 x : V → RK such est in mathematics, but was ultimately
that the constraints: crushed by Gödel’s incompleteness
theorem in its aim of having a finitistic
∀{i, j} ∈ V k xi − x j k = dij (6.1) mechanical procedure for proving
every true sentence from the axioms.

are satisfied?
2
What if the only way to write a
certain proof formally takes longer
than the maximum extent of the
When the norm in Eq. (6.1) is the 2-norm, the DGP is also called human life? Well, use computers!
Euclidean DGP (EDGP), which was introduced in Sect. 2.6. What about reading a proof? Today we
have formal proofs, partly or wholly
Because it consists of a set of constraints for which we need written out by computers, which
to find a solution, the DGP is a CP problem (see Ch. 4). On the no-one can possibly hope to read in a
lifetime, much less understand. Can
other hand, CP problems usually have integer, bounded variables,
they still be called proofs? This is
whereas the DGP has continuous unbounded ones. So it makes similar to the old “does the universe
more sense to see it as a feasibility NLP, i.e. an NLP with zero still exist if no-one’s around to notice?”

objective function.
As mentioned in Sect. 2.6, it is difficult to solve DGPs as systems
3
A realization of a graph is a function,
mapping each vertex to a vector of
of nonlinear constraints, as given in Eq. (6.1). Usually, MP solvers a Euclidean space, which satisfies
are better at improving optimality than ensuring feasibility. A nice Eq. (6.1).
feature of DGPs is that we can write them as a minimization of
94 mathematical programming

constraint errors:

min
x
∑ | k xi − x j k − dij |. (6.2)
{i,j}∈ E

Evidently, x ∗ is feasible in Eq. (6.1) if and only if it is globally

optimum in Eq. (6.2), with zero optimal objective function value.

6.2 Some applications of the DGP

As mentioned in Sect. 2.6, the DGP has many applications to sci-

ence and engineering [Liberti et al., 2014]. We always assume that
the input graph G is connected (otherwise the problem can be
decomposed and solved for each component separately).

6.2.1 Clock synchronization

Let K = 1 and V be a set of clocks. An edge {i, j} is in E if the time
difference between clocks i and j is dij . Suppose that each clock i
knows its own time as well as the weighted graph G. Then a solu-
tion x ∗ ∈ R1 = R is a consistent evaluation of the absolute times of Figure 6.1: Clock synchronization
(from www.mdpi.com).
each clock in V. This application arises in network synchronization
protocols [Singer, 2011].

6.2.2 Sensor network localization

Let K = 2 and V be a set of mobile devices in a wireless network at
a certain given time t. An edge {i, j} is in E if the distance between
i and j is known to be dij at time t. Such distances can be measured
up to a certain threshold, for example by estimating how much
battery power a peer-to-peer communication between i and j takes.
By exploiting network communication, the weighted graph G can Figure 6.2: Sensor network localization
(from www.commsys.isy.liu.se).
be transmitted to a central server. A solution x ∗ ∈ R2 gives a
consistent position of the devices at time t. This application arises
naturally in the localization of sensors in wireless networks [Yemini,
1978, Biswas et al., 2006, Bachrach and Taylor, 2005, Ding et al.,
2010, Cucuringu et al., 2012a].

6.2.3 Protein conformation

Let K = 3 and V be a set of atoms in a molecule such as e.g. a
protein. It is well known that proteins interact with cells because of
their geometrical (as well as chemical) properties: in other words,
their shape matters. Being so small, we cannot directly observe Figure 6.3: Protein conformation from
their shape. We can, however, measure inter-atomic distances up to distance data (slide by Dr. B. Chazotte
a certain threshold, usually 5Å to 6Å [Schlick, 2002], using Nuclear www.campbell.edu).

Magnetic Resonance (NMR) [Wüthrich, 1989]. Although NMR only

really gives a frequency for each triplet (atom type, atom type, length),
which means that we can only know how frequent it is to observe
a certain distance between, say, a hydrogen and a carbon, using
these data we can reconstruct a weighted graph where {i, j} is in E
 distance geometry 95

if the distance dij between atom i and atom j is known. A solution

x ∗ ∈ R3 gives a consistent conformation of the protein [Moré
and Wu, 1999, Reams et al., 1999, Wu et al., 2008, Cucuringu et al.,
2012b, Lavor et al., 2011, Davis et al., 2010, Cassioli et al., 2015].

6.2.4 Unmanned underwater vehicles

Let K = 3 and V be a set of unmanned submarines, such as
e.g. those used by Shell when trying to fix the oil spill in the Gulf of
Mexico. Since GPS cannot be used underwater, the position of each
submarine must be inferred. An edge {i, j} is in E if the distance Figure 6.4: Unmanned underwater
between submarines i and j can be obtained using the sonars. A vehicles (from www.swissint.ch).

solution x ∗ ∈ R3 gives a consistent snapshot of the positions of the

submarines at a certain time instant.

6.3 Formulations of the DGP

The DGP is really a whole class of problems, which depends on

the type of norm used in Eq. (2.21). In general, however, if it is not
explicitly given, most people would mean the Euclidean norm k · k2 .
In other words, most of the times you read DGP, you should think
of EDGP.

6.4 The 1-norm and the max norm

Figure 6.5: The 2-, 1- and ∞-norms.
We just point out that DGPs defined on the 1-norm and the ∞-
norm4 can be linearized exactly to MILPs. We show how to proceed 4
Also known as max norm.
in the case5 k · k = k · k1 . Eq. (6.2) becomes: 5
By adapting the MILP reformulation
  for the k · k1 case, derive a MILP
  reformulation for the k · k∞ case.
min ∑  ∑ | xik − x jk | − dij  .
 
x
{i,j}∈ E k≤K
 

By applying the reformulations in Sect. 3.2.4 and 3.1.5 we obtain

the following nonconvex NLP:

∑ (sij+ + sij− )

min
x,s

{i,j}∈ E



−
= sij+ − sij−

+
∀{i, j} ∈ E ∑ (tijk + tijk ) − dij




k≤K 

−

+
∀k ≤ K, {i, j} ∈ E tijk − tijk = xik − x jk
+ −
∀k ≤ K, {i, j} ∈

E tijk tijk = 0 


+ −

 Figure 6.6: The 1-norm between these
∀{i, j} ∈ E sij , sij ≥ 0 


 two points is 12 (from Wikipedia).
+ −

∀k ≤ K, {i, j} ∈ E tijk , tijk ≥ 0,


+ −
where the only nonlinear constraints are the products tijk tijk = 0. In 6
The weighted diameter of a graph is
theory, we cannot apply the reformulation in Sect. 3.2.3 since the t the maximum, over all pairs of vertices
u, v in the graph, of the weight of the
variables are unbounded. On the other hand, their unboundedness shortest path connecting u and v.
stems from the x variables, which are themselves unbounded — but
simply because we did not bother looking at the problem closely 7
This implication holds since all
translations are congruences, but it
enough. Suppose that the weighted diameter6 of the input graph G requires us to add another family of
is γ. Then no realization can have k xi − x j k ≥ γ, which also implies7 (linear) constraints to the formulation.
Which ones?
96 mathematical programming

k xi k ≤ γ for all i ∈ V, and in turn −γ ≤ xik ≤ γ for all i ∈ V and

k ∈ K. So, using interval arithmetic on the linear constraints, we can
bound all variables x, s, t by a suitably large8 constant M. We can 8
For best computational results, this
now apply the reformulation in Sect. 3.2.3, which yields the MILP: constant should be as small as possible
for each instance to be solved.
∑ (sij+ + sij− )

min 
x,s {i,j}∈ E



− −

+ +
∀{i, j} ∈ E ∑ (tijk + tijk ) − dij = sij − sij




k≤K



+ −
∀k ≤ K, {i, j} ∈ E tijk − tijk = xik − x jk





+
∀k ≤ K, {i, j} ∈ E tijk ≤ Mzijk (6.3)
− 
∀k ≤ K, {i, j} ∈ E tijk ≤ M(1 − zijk ) 



+ −

∀{i, j} ∈ E sij , sij ≥ 0




+ −

∀k ≤ K, {i, j} ∈ E tijk , tijk ≥ 0





∀k ≤ K, {i, j} ∈ E zijk ∈ {0, 1}.


For the rest of this chapter we shall focus on the 2-norm, i.e. we
shall mean EDGP whenever we write DGP.

6.4.1 The 2-norm

We can square both sides of Eq. (6.1) and obtain another NLP with
exactly the same set of solutions:

∀{i, j} ∈ V k xi − x j k22 = d2ij , (6.4)

which we recognize as a QCQP with zero objective function: in-

deed, the right hand side of Eq. (6.4) can be written as:

k xi − x j k22 = ( xi − x j ) · ( xi − x j )
= xi · xi + x j · x j − 2xi · x j
= k xi k22 + k x j k22 − 2xi · x j
= ∑ 2
xik + ∑ x2jk − 2 ∑ xik x jk ,
k≤K k≤K k≤K

which is a multivariate polynomial of second degree. An SDP

relaxation of Eq. (6.4) was derived in Sect. 2.10, see Eq. (2.35).
Applying Eq. (6.2) to Eq. (6.4), we obtain the Global Optimiza-
tion (GO) formulation9 9
Note that the absolute values in
Eq. (6.2) have been replaced by a
min
x
∑ (k xi − x j k22 − d2ij )2 , (6.5) square: obviously the set of global
optima of Eq. (6.2) is the same as the
{i,j}∈ E
one for Eq. (6.5).
which is an unconstrained quartic polyonomial minimization prob-
lem. This formulation is at the basis of several different solution
methods for the DGP [Lavor et al., 2006, Liberti et al., 2009b].

6.5 Euclidean Distance Matrices

A Euclidean distance matrix (EDM) is an n × n zero-diagonal sym-

metric matrix D = (dij ) for which there exist an integer K and a
realization x = ( x1 , . . . , xn )> ∈ RnK such that ∀i, j ≤ n we have
k xi − x j k2 = dij . Oftentimes, people write EDM when they really
mean square EDM, i.e. D = (d2ij ).
 distance geometry 97

6.5.1 The Gram matrix from the square EDM

Since

∀i ≤ j ≤ n k xi − x j k22 = k xi k22 + k x j k22 − 2xi · x j , (6.6)

the square EDM D is intimately related to the Gram matrix B =

( xi · x j ) (see Sect. 2.10 and 5.3.4) of the realization x ∈ RnK . In this
section we show10 how to compute B in function of D. 10
The sequence of transformations
We first translate x so that its barycenter11 is at the origin: that follow is fastidiously detailed and
precise. It is more or less taken from
[Cox et al., 1997], with some added
∑ xi = 0. (6.7) steps of my own. The reason I tried to
i ≤n put in as many details as possible is
that I was tired to re-work the details
Now we remark that, for each i, j ≤ n, we have: out every time I read this proof with
some missing steps.
d2ij = k xi − x j k2 = ( xi − x j ) · ( xi − x j ) = xi · xi + x j · x j − 2xi · x j . (6.8) 11
The barycenter of a sequence
( x1 , . . . , xn ) of points in RK is the
Next, we “invert” Eq. (6.8) to express xi · x j in function of d2ij : we vector n1 ∑ xi . The barycenter is also
i ≤n
sum Eq. (6.8) over all values of i ∈ {1, . . . , n}, obtaining: known as centroid.
!
∑ d2ij = ∑ (xi · xi ) + n(x j · x j ) − 2 ∑ xi · xj. (6.9)
i ≤n i ≤n i ≤n

By Eq. (6.7), the rightmost term in the right hand side of Eq. (6.9) is
zero. On dividing through by n, we have

1 1
n ∑ d2ij = n ∑ (xi · xi ) + x j · x j . (6.10)
i ≤n i ≤n

Similarly for j ∈ {1, . . . , n}, we obtain:

1 1
n ∑ d2ij = xi · xi + n ∑ (x j · x j ). (6.11)
j≤n j≤n

We now sum Eq. (6.10) over all j, getting:

1 1
n ∑ d2ij = n n ∑ (xi · xi ) + ∑ (x j · x j ) = 2 ∑ (xi · xi ) (6.12)
i ≤n i ≤n j≤n i≤n
j≤n

(the last equality in Eq. (6.12) holds because the same quantity
f (k ) = xk · xk is being summed over the same range {1, . . . , n}, with
the symbol k replaced by the symbol i first and j next). We then
divide through by n to get:

1 2
n2 ∑ d2ij = n ∑ (xi · xi ). (6.13)
i ≤n i ≤n
j≤n

We rearrange Eq. (6.8), (6.11), (6.10) as follows:

2xi · x j = xi · xi + x j · x j − d2ij (6.14)

1 1
xi · xi =
n ∑ d2ij − n ∑ (x j · x j ) (6.15)
j≤n j≤n
1 1
xj · xj =
n ∑ d2ij − n ∑ (xi · xi ), (6.16)
i≤n i ≤n
98 mathematical programming

and replace the left hand side terms of Eq. (6.15)-(6.16) into Eq. (6.14)
to obtain:
1 1 2
2xi · x j =
n ∑ d2ik + n ∑ d2kj − d2ij − n ∑ (xk · xk ), (6.17)
k≤n k≤n k≤n

whence, on substituting the last term using Eq. (6.13), we have:

1 1
2xi · x j =
n ∑ (d2ik + d2kj ) − d2ij − n2 ∑ d2hk . (6.18)
k≤n h≤n
k≤n

Eq. (6.18) can also be written12 in matrix form as: 12

Prove it.

1
B = − JDJ, (6.19)
2
where J = In − n1 1 · 1> and 1 = (1, . . . , 1).
| {z }
n

6.5.2 Is a given matrix a square EDM?

We can now establish that the problem of determining whether a
given matrix is a square EDM can be solved in polytime as follows:

1. given D, compute B as per Sect. 6.5.1

2. compute the eigenvalues of B

3. if all eigenvalues are non-negative, then B  0, so we can factor

B as in Sect. 5.4.1, to obtain x such that B = xx > , as required:
output YES13 This proves that any PSD matrix is a
13

Gram matrix of some realization.

4. if at least one eigenvalue is negative, then output NO.

We note that all of the steps can be carried out in polynomial time,
and remark that the above algorithm can also be used to ascertain
whether D is a (non-square) EDM or not.14 14
Why?

6.5.3 The rank of a square EDM

EDMs and their square counterparts have a fundamental difference,
as remarked in the recent survey [Dokmanić et al., 2015]: whereas
EDMs may have any rank greater or equal to rank( x ), square EDMs
have rank in the set15 {rank( x ), . . . , rank( x ) + 2} . By Eq. (6.6), we So, if x has full rank K, the square
15

can write the EDM D of the realization x ∈ RnK as: EDM of x has rank in {K, K + 1, K + 2}.

>
D = 1diag( xx > ) − 2xx > + diag( xx > )1> ,

where 1 is the (column) vector of all ones, and diag(v) is the (col-
umn) vector of the diagonal of v. This implies16 16
Note that Eq. (6.20) is the matrix
equation inverse of Eq. (6.19).
D = 1diag( B)> − 2B + diag( B)1> , (6.20)

where B is the Gram matrix of x. Now, by Prop. 5.3.2(ii), we know

that rank( B) = rank( x ), and both the first and the last term in the
17
I.e., if M = M0 + M00 then rank( M ) ≤
RHS of Eq. (6.20) have rank 1. The claim follows by rank inequali- rank( M0 ) + rank( M00 ). In other words,
ties.17 the rank(·) function is sub-additive.
 distance geometry 99

The reason why this matters in practice is the following. Usually,

given an EDM, you really want to find the realization it corre-
sponds to. If you square every component of your given EDM, and
then compute its rank r (perhaps only considering eigenvalues
having absolute value larger than a given “floating point error tol-
erance”), r − 2 will be a lower bound on the affine dimension K 0
of the realization giving rise to your EDM. In most applications, n
will be very large, but K will be very small, see e.g. Sect. 6.2 where
K ∈ {1, 2, 3}. If you find r − 2  K, and you cannot impute this to
a floating point error in eigenvalue computations, chances are your
given EDM is wrong.

Figure 6.7: There are many different

6.6 The EDM Completion Problem completion problems: see above an
example of rank-1 completion (from
The following problem is closely related to the EDGP. Wikipedia.

Euclidean Distance Matrix Completion Problem (EDMCP).

Given a partially specified18 n × n zero-diagonal symmetric matrix, 18
A partially specified matrix (or partial
determine whether it can be completed to an EDM (or to a square matrix) over a number field F is a
EDM). matrix over F ∪ {?}. By convention,
a ? entry is unspecified. Here is an
example of a partial matrix:
It is easy to show that a partially specified zero-diagonal square  
0 ? 1 1
symmetric matrix D over R+ carries the same information as a  ? 0 4 9 
simple undirected edge-weighted graph G = (V, E, d): every  1 4 0 4 .
 

component dij 6= ? in D corresponds to an edge {i, j} ∈ E with 1 9 4 0

weight dij , whereas ? components corresponds to edges not in E. Can this matrix be completed to an
EDM?
So is the EDMCP just another name for the EDGP? The difference
between EDGP and EDMCP is in the integer K: in the former,
K > 0 is given as part of the input, while in the latter it is part of
the output. Specifically, while the graph information is the same,
the EDGP is also given K, and then asks if, for that K, the given
partial matrix can be completed to a distance matrix of a realization
in RK . The EDMCP, on the other hand, asks if there is any K such
that the given partial matrix can be completed to a distance matrix
of a realization in RK .
This difference is subtle, but has a very deep implication: the
EDGP is known to be NP-hard,19 whereas we do not know the
complexity status of the EDMCP: no-one has found a polytime
algorithm, nor a proof of NP-hardness.
Figure 6.8: An EDMCP instance
(above) and the equivalent EDGP
6.6.1 An SDP formulation for the EDMCP instance (below).
19
By reduction from the Partition
By Prop. 5.3.2, every Gram matrix is PSD, and, by Sidenotes 13 in problem, see [Saxe, 1979].

Ch. 5 and 13 in this chapter, every PSD matrix is the Gram matrix
of some other matrix (or realization). If we were given a partially
specified Gram matrix B = (bij ), we could complete20 it by solving 20
This is called the PSD Completion
Problem (PSDCP).
the following SDP:
)
∀i, j ≤ n : bij 6= ? Xij = bij
(6.21)
X  0.
100 mathematical programming

Any solution X ∗ to Eq. (6.21) is a PSD matrix having bij in the

components specified by B.
However, we are not given B but the square EDM matrix D =
2
(dij ). We therefore use Eq. (6.6) and replace xi · x j by Xij (for all
i, j ≤ n):
)
∀i, j ≤ n : d2ij 6= ? Xii + X jj − 2Xij = d2ij
(6.22)
X  0.

Note the similarity of the first constraint of Eq. (6.22) with Eq. (2.34),
written in terms of weighted graphs instead of partial matrices.
Any solution X ∗ to Eq. (6.22) is a PSD matrix, and hence a Gram
matrix, such that Xii + X jj − 2Xij = d2ij in the components specified
by D. In other words, if we apply Eq. (6.20) with B replaced by X ∗ ,
we get an EDM with the specified entries.
Compare the two SDP formulations for the EDGP (Eq. (2.35))
and for the EDMCP (Eq. (6.22)). The former is different, in that
we use the Schur complement:21 while in the EDGP we require a 21
The Schur complement being PSD
PSD solution of a given rank K, in the EDMCP we only want a PSD states X  xx > , which is a relaxation
of X = xx > . The fact that x (and hence
solution of any rank (or, in other words, any PSD solution). This is its rank) is explicitly mentioned in the
why Eq. (2.35) is only a relaxation of the EDGP, whereas Eq. (6.22) SDP makes the relaxation tighter with
respect to the rank constraint.
is an exact formulation of the EDMCP.
Since SDPs can be solved in polytime and the SDP in Eq. (6.22) is
an exact formulation for the EDMCP, why am I saying that no-one
found a polytime algorithm for solving the EDMCP yet? Essentially,
because of floating point errors: the IPMs that solve SDPs are
approximate methods: they find a matrix which is approximately
feasible with respect to the constraints and approximately PSD (see
Sect. 5.4.3). Worst-case complexity classes such as P or NP, however,
are defined for precise solutions only. As discussed in Sidenote 15
in Ch. 5, approximate feasibility disqualifies the algorithm from
being defined as an “approximation algorithm”.

6.6.2 Inner LP approximation of SDPs

SDPs are nice because they usually provide tight relaxations and
can be solved in polytime. On the other hand, current technology
can solve relatively small SDPs, say with matrices of up to around
1000 components. In practice, it is not uncommon to see DGP appli-
cations where K ∈ {1, 2, 3} but n ≥ 1000. This would yield an SDP
variable matrix with at least one million components. Since SDP
solvers are usually based on IPMs, at each iteration the solver has
to deal with O(106 ) floating point numbers. Unless the structure
of the problem can be exploited to simplify the formulation, SDP
solvers will generally fail. In Sect. 6.7 we will present a heuristic
algorithm which scales better.
Here we discuss an extremely recent method [Majumdar et al.,
Figure 6.9: Amir Ali Ahmadi is the
2014] which can be used to approximate an SDP by means of an main architect behind the work in
LP. Since the LP is based on an inner approximation, any solution X 0 SDP approximations by DD matrices.
You’ve already seen his mug in
found by the LP is also a feasible solution for the SDP. This method
Fig. 2.30.
 distance geometry 101

is based on diagonal dominance.22 22

An n × n matrix A = ( aij ) is
Recall that an SDP is simply an LP defined on a matrix X of diagonally dominant (DD) if Eq. (6.23)
holds.
decision variables, subject to a further PSD constraint X  0. We
replace the PSD constraint by the DD constraint:

∀i ≤ n aii ≥ ∑ |aij |, (6.23)

j 6 =i

and remark that any matrix X which is DD is also SDP; this follows
directly from Gershgorin’s theorem.23 Lastly, we reformulate the 23
The proof on Wikipedia is very clear,
absolute values in Eq. (6.23) using Eq. (3.16)-(3.18) in Sect. 3.2.4, search for “Gershgorin circle theorem”.

obtaining the following LP, which will yield solutions of the SDP
relaxation in Eq. (2.35):

= d2ij 

∀{i, j} ∈ E Xii + X jj − 2Xij 

∀i ∈ V ∑ tij ≤ Xii (6.24)
j 6 =i 

∀i 6 = j ∈ V −tij ≤ Xij ≤ tij .


Note that this is a feasibility LP. We can solve it “as such”, or try to
make the entries of X as small as possible (which might reduce its
rank if many of them become zero) by using min ∑ij tij .

6.6.3 Refining the DD approximation

An iterative refinement method for improving the approximation
quality of the solution X of Eq. (6.24) with respect to the SDP is
described in [Ahmadi and Hall, 2015]. We present it here in full
generality, applied to any SDP.
For any n × n matrix U, we have U > U  0. This follows because
any Gram matrix is PSD, and, trivially, U > U = Gram(U ). Moreover,
for any X  0, we have
U > XU  0, Figure 6.10: An example of diagonal
dominance (from www.voonik.com).
since X being PSD means that it is the Gram matrix of some other
matrix V, so

U > XU = U > V > VU = (VU )> VU  0

as the Gram matrix of VU must be PSD. This means that the set

D(U ) = {U > XU | X is DD}

only contains PSD matrices.

Let us go back to the general SDP formulation in Eq. (2.32). As
mentioned above, we obtain an inner approximation of this SDP
by replacing X  0 by “X is DD”, which can be reformulated to an
LP. This LP has no “adjustable parameters” by which we can try
and improve the approximation; in other words, it is an LP rather
than a family of LPs — and this is one of those cases where having
a parametrized family of LP approximations could help in adapting
the approximation to the original SDP. By replacing “X is DD” by
102 mathematical programming

X ∈ D(U ) we achieve exactly this, and let our LP approximation

depend on the matrix U.

min C • X 

∀i ≤ m Ai • X = bi (6.25)

X ∈ D(U ), 

We can now leverage the parameter matrix U to attempt to

improve the approximation.
The iterative algorithm proposed in [Ahmadi and Hall, 2015]
defines a sequence of matrices U 0 , U 1 , . . . , U ` , . . . as follows:
Figure 6.11: An introductory slide on
parametric LP (from users.soe.ucsc.
U0 = I (6.26)
edu/~kross).
U` = factor( X `−1 ), (6.27)

where, factor denotes a factor24 of the argument matrix, and for 24

The factor can be found in any
each k ∈ N, X ` is the solution to the LP number of ways. The paper suggests
 using Cholesky factors where
min C • X 
 X `−1 = LL> ,
i
∀i ≤ m A • X = b i (6.28)
with L a lower triangular matrix.
`

X ∈ D(U ), 

which is the same as Eq. (6.25) with U replaced by U ` . Eq. (6.28)

actually defines a sequence of approximating LPs to Eq. (2.32).

6.6.1 Proposition
The approximation X ` to the solution of Eq. (2.32) is no worse than X `−1 .

Proof. It suffices to show that X `−1 is feasible for the `-th approxi-
mating LP in the sequence, since this shows that the feasible sets F`
of the approximating LP sequence form a chain

F0 ⊆ · · · ⊆ F ` ⊆ · · ·
Since all are inner approximations of the feasible region of Eq. (2.32),
the larger the feasible region, the better the approximation. Now,
> >
by Eq. (6.27) we have that X `−1 = (U ` ) U ` = (U ` ) IU ` , and since
the identity matrix I is trivially DD, we have that X `−1 ∈ D(U ` ).
Moreover, since X `−1 solves the (` − 1)-th approximating LP in the
sequence Eq. (6.28), we have ∀i ≤ m Ai • X `−1 = bi , which proves
the claim. 2

6.6.4 Iterative DD approximation for the EDMCP

The application of the results of Sect. 6.6.3 to Eq. (6.24) yields the
following iterative algorithm.
1. Initialize U 0 = I and ` = 0

2. Solve the following LP in the matrix variables X, Y and T ≥ 0:


∀i, j : d2ij 6= ? Xii + X jj − 2Xij = d2ij 


∀i ≤ n ∑ tij ≤ Yii 


j 6 =i (6.29)
−T ≤ Y ≤ T  

>

(U ` ) YU ` = X


 distance geometry 103

and let ( X ` , Y ` , T ` ) be its solution

3. Let U `+1 be a Cholesky factor of of X `

4. Increase ` and loop from Step 2.

The termination conditions are based on either ` being smaller than

a pre-determined `max threshold, or CPU time, or k X ` − X `−1 k being
too small.

Figure 6.12: DD and SDD inner

6.6.5 Working on the dual approximations of the PSD set
I + xA + yB  0
What if the original SDP formulation is used to compute a guar-
anteed bound to some problem P? Then inner approximations (from aaa.princeton.edu).

of the SDP feasible region will not help, since only an optimum
of the SDP (or a bound to this optimum) will provide the bound
guarantee for P.
In such cases, we leverage the fact that both LPs and SDPs have a
strong duality theory. In particular:

• the dual of an LP is another LP, and the dual of an SDP is an-

other SDP;

• the optimal objective function values of the LP and its dual

(respectively of the SDP and its dual) are equal;

• any feasible solution of the LP (respectively, SDP) dual is a

guaranteed bound on the objective function value of the primal.

So all we need to do is to compute the dual of the original SDP, Figure 6.13: A geometric view of a
and then apply the DD inner approximation to the dual. This will minimization SDP dual: maximizing
yield a feasible solution in the dual SDP, which provides a valid over the direction vectors perpendicu-
lar to the outer approximation (from
guaranteed bound for the original SDP (and hence, in turn, for P). [Dattorro, 2015]).

6.6.6 Can we improve the DD condition?

All DD matrices are SDP, but not every SDP matrix is DD. In fact,
X being DD is quite a restrictive condition on X. It can be relaxed
somewhat by requiring that there is a diagonal matrix ∆ with
strictly positive diagonal components such that ∆X∆ is DD. A
matrix X with this property is called scaled diagonally dominant
(SDD).
The relevant property of SDD matrices to this discussion is that
every SDD matrix is PSD, and every DD matrix is also (trivially,
with ∆ = I) an SDD matrix. So they represent a set of matrices
between DD and PSD matrices.
Luckily, it turns out that X is SDD can be written as a SOCP (see
Sect. 2.10.1). Methods similar to those discussed above can also
be derived, with LPs replaced by SOCPs. The trade-off is that, in
exchange for improving the approximation of the PSD matrix cone,
and hence the solution quality, we have to spend more CPU time
computing the solution of each SOCP.
104 mathematical programming

6.7 The Isomap heuristic

The Isomap algorithm is usually known as a dimensionality reduc-

tion method, i.e. an algorithm which transforms a set X ⊆ Rn into
another set Y ⊆ Rm with m < n, and such that some properties of
X are preserved, or approximately preserved, in Y. In summary, the
Isomap method is as follows: given X,

1. compute a partial EDM D on X (for example by only considering

distances smaller than a certain threshold);

2. complete D to some approximate distance matrix D̄, for example

by running a weighted shortest path algorithm on the weighted
graph corresponding to D;

3. use Sect. 6.5.1 to find the (approximate) Gram matrix B̄ corre-

sponding to D̄;

4. in general, since D̄ is usually not an EDM, B̄ is not a Gram

matrix, in particular it is not PSD and hence fails to have a real
Figure 6.14: The Isomap method for
factoring YY > ; so we find the closest PSD matrix B0 to B̄ (see
DG: complete a partial distance matrix
Sect. 5.4.3) and then factor it as B0 = YY > . using shortest paths, above; then find
an approximate realization, below
This algorithm is successful as long as (from [Tenenbaum et al., 2000]).

m = rank(Y ) < rank( X ) = n.

In practice, though, we would prefer m  n.

6.7.1 Isomap and DG

In order to employ Isomap as an algorithm for the EDMCP we
simply start it from Step 2, so that its input is the partial EDM D.
If we want to apply it to the EDGP, we replace B0 by the closest
Figure 6.15: The Isomap algorithm
PSD matrix to B̄ having given rank K, namely,25 we only keep at solving a DGP instance (left). The
most K non-negative eigevalues. shape of the solution (right) looks
convincingly similar to the original
In general, the Isomap algorithm is neither exact nor approxi-
shape.
mate, and hence only qualifies as a heuristic algorithm. It is nonethe- 25
To make this summary explanation
less very successful in practice [Tenenbaum et al., 2000] since it somewhat more explicit: we factor B̄
into eigenvectors P and eigenvalues
scales well to large sizes.
Λ listed in decreasing order, we zero
all negative ones, and possibly all
the positive ones up until we leave at
6.8 Random projections most K non-negative ones Λ+ , then we
compute B0 = P> Λ+ P.
An important feature of the 2-norm is that it can be approxi-
mately preserved by random projections. Let X ⊆ Rm with
X = { x1 , . . . , xn }, and think of X as an m × n matrix having n
columns x1 , . . . , xn each defining a point in Rm . Then there is a
randomized algorithm which outputs a k × m matrix T, where k is
O( ε12 log n), such that the probability that:

∀i < j ≤ n (1 − ε)k xi − x j k2 ≤ k Txi − Tx j k2 ≤ (1 + ε)k xi − x j k (6.30) Figure 6.16: The JLL won’t work in low
dimensions.
 distance geometry 105

holds can be made arbitrarily close to 1. This is known as the

Johnson-Lindenstrauss Lemma (JLL) [Johnson and Lindenstrauss,
1984], and is one of the cornerstones of clustering large-dimensional
datasets (such as image databases).
The most striking feature26 of the JLL is that k is not a function 26
Edo Liberty (of Yahoo! Labs) once
of m. In other words, the actual number of dimensions you need in told me that the way he sees the JLL is
that, in any Euclidean space, you can
order to “pack” the projections of the vectors in X, so that the pair- fit exponentially many (in function of the
wise distances are approximately preserved to within ε, does not dimension) “approximately pairwise
orthogonal” vectors . Prove that this is
depend on the number of dimensions of X. Another striking fea- equivalent to the JLL.
ture is that the number of dimensions grows only logarithmically in
the number of vectors.27 27
The upshot is that the perfect
A few comments on the JLL are in order. application setting for the JLL is when
X contains a moderate to large number
of vectors in a huge dimensional space.
• O(ε−2 log n) is not really a number, but a short-hand for C ε−2 log n Image or movie databases have this
for some constant C which does not depend on n or m. To use property, which is why the random
projections are used when clustering
the JLL in practice, this constant must be estimated somehow.28
images.
• C ε−2 log n is not always smaller than m — aside from C , it de- 28
There is not much literature on this,
pends on ε and n, of course. But asymptotically, the term in log n unfortunately [Venkatasubramanian
and Wang, 2011].
grows very slowly with respect to how fast ε−2 grows.
• If you want the approximation to be very accurate, then ε should
be very small, so 1/ε2 is going to be very large. If m is huge,
there will be a saving. The trade-off is between ε and m. For
smaller m’s, you would need a larger error tolerance ε.

The randomized algorithm to find T is very simple.

1. Sample each of the km components of T independently from a

normal distribution with mean 0 and standard deviation √1
k

2. Does TX verify Eq. (6.30)? If not, repeat from Step 1.

The elementary proof of the JLL given in [Dasgupta and Gupta,

2002] shows that the number of iterations of this algorithm is O(n).

6.8.1 Usefulness for solving DGPs

A possible use of random projections for solving EDGPs is given by
the observation that it is easier to satisfy Eq. (6.1) if K is large than
if K is small. On the other hand, most applications require a small
K. So, for extremely large-scale EDGP instances, one might try 29
More in general, the JLL is applica-
deploying any heuristic method to find an acceptably good solution ble to any algorithm which only (or
mostly) uses Euclidean distances, such
in a high-dimensional space, and then use a random projection as e.g. the popular k-means heuris-
matrix T to scale the dimensions down to a more acceptable level. tic for solving Euclidean clustering
problems.
One would then need a heuristic such as Isomap (Sect. 6.7) to drive
the dimensions down to a given K.29

6.9 How to adjust an approximately feasible solution

Working with SDP and heuristic solutions exposes us to the issue

of infeasibility: what if our solution x to a EDGP does not satisfy
Eq. (6.1)? A practical and usually acceptably fast way to do so is
106 mathematical programming

to use x as a starting point for a local search with respect to the

GO formulation Eq. (6.5). A good, open-source local NLP solver
which can be employed for this purpose is IPOPT. There is still no
guarantee of feasibility, but at least the infeasibility error should
decrease.

6.10 Summary

This chapter is about Distance Geometry, and introduces the funda-

mental problem of DG. It argues that it is useful for scientific and
engineering applications, it presents formulations for the 1-, ∞- and
2-norms. It introduces Euclidean distance matrices and draws a
parallel between the DGP and the completion problem for partially
specified EDMs. It presents an SDP formulation for this problem,
and discusses a very new method for finding feasible SDP solu-
tions by solving an LP. It then presents the famous dimensionality
reduction Isomap heuristic as a solution algorithm for the DGP,
and briefly introduces random projections which approximately
preserve Euclidean distances.
Bibliography

A. Ahmadi and G. Hall. Sum of squares basis pursuit with

linear and second order cone programming. Technical Report
1510.01597v1, arXiv, 2015.

A. Alfakih, A. Khandani, and H. Wolkowicz. Solving Euclidean dis-

tance matrix completion problems via semidefinite programming.
Computational Optimization and Applications, 12:13–30, 1999.

F. Alizadeh and D. Goldfarb. Second order cone programming.

Mathematical Programming B, 95:3–51, 2003.

E. Amaldi, K. Dhyani, and L. Liberti. A two-phase heuristic for

the bottleneck k-hyperplane clustering problem. Computational
Optimization and Applications, 56:619–633, 2013.

D. Applegate, R. Bixby, V. Chvátal, and W. Cook. The Traveling

Salesman: a Computational Study. Princeton University Press, Prince-
ton, 2007.

J. Bachrach and C. Taylor. Localization in sensor networks. In

I. Stojmenović, editor, Handbook of Sensor Networks, pages 3627–
3643. Wiley, 2005.

P. Belotti. Couenne: a user’s manual. Dept. of Mathematical

Sciences, Clemson University, 2011.

P. Belotti, J. Lee, L. Liberti, F. Margot, and A. Wächter. Branch-

ing and bounds tightening techniques for non-convex MINLP.
Optimization Methods and Software, 24(4):597–634, 2009.

T. Berthold, G. Gamrath, A. Gleixner, S. Heinz, T. Koch, and

Y. Shinano. Solving mixed integer linear and nonlinear problems using
the SCIP Optimization Suite. ZIB, http://scip.zib.de/, 2012.

D. Bertsimas and M. Sim. The price of robustness. Operations

Research, 52(1):35–53, 2004.

P. Biswas, T. Lian, T. Wang, and Y. Ye. Semidefinite programming

based algorithms for sensor network localization. ACM Transactions
in Sensor Networks, 2:188–220, 2006.

L. Blumenthal. Theory and Applications of Distance Geometry. Oxford

University Press, Oxford, 1953.
108 mathematical programming

I. Bomze. Evolution towards the maximum clique. Journal of Global

Optimization, 10:143–164, 1997.

I. Bomze, M. Budinich, P.M. Pardalos, and M. Pelillo. The max-

imum clique problem. In D.-Z. Du and P.M. Pardalos, editors,
Handbook of Combinatorial Optimization, Supp. A, volume supp. A,
pages 1–74. Kluwer Academic Publishers, Dordrecht, 1998.

P. Bonami and J. Lee. BONMIN user’s manual. Technical report, IBM

Corporation, June 2007.

M. Boulle. Compact mathematical formulation for graph partition-

ing. Optimization and Engineering, 5:315–333, 2004.

S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge

University Press, Cambridge, 2004.

C. Bragalli, C. D’Ambrosio, J. Lee, A. Lodi, and P. Toth. On the

optimal design of water distribution networks: a practical minlp
approach. Optimization and Engineering, 13:219–246, 2012.

R. Burkard, M. Dell’Amico, and S. Martello. Assignment problems.

SIAM, Providence, 2009.

E. Candès. The mathematics of sparsity. In S.Y. Jang, Y.R. Kim, D.-

W. Lee, and I. Yie, editors, Proceedings of the International Congress of
Mathematicians, volume I. Kyung Moon SA, Seoul, 2014.

G. Caporossi, D. Alamargot, and D. Chesnet. Using the computer

to study the dyamics of the handwriting processes. In DS 2004
Proceedings, volume 3245 of LNAI, pages 242–254. Springer, 2004.

A. Cassioli, B. Bordeaux, G. Bouvier, A. Mucherino, R. Alves,

L. Liberti, M. Nilges, C. Lavor, and T. Malliavin. An algorithm to
enumerate all possible protein conformations verifying a set of
distance constraints. BMC Bioinformatics, page 16:23, 2015.

B. Colson, P. Marcotte, and G. Savard. Bilevel programming: a

survey. 4OR, 3:87–107, 2005.

V. Cortellessa, F. Marinelli, and P. Potena. Automated selection of

software components based on cost/reliability tradeoff. In V. Gruhn
and F. Oquendo, editors, EWSA 2006, volume 4344 of LNCS, pages
66–81. Springer, 2006.

A. Costa, P. Hansen, and L. Liberti. On the impact of symmetry-

breaking constraints on spatial branch-and-bound for circle pack-
ing in a square. Discrete Applied Mathematics, 161:96–106, 2013.

D. Cox, J. Little, and D. O’Shea. Ideals, Varieties and Algorithms.

Springer, Berlin, second edition, 1997.

M. Cucuringu, Y. Lipman, and A. Singer. Sensor network local-

ization by eigenvector synchronization over the Euclidean group.
ACM Transactions on Sensor Networks, 8:1–42, 2012a.
 bibliography 109

M. Cucuringu, A. Singer, and D. Cowburn. Eigenvector synchro-

nization, graph ridigity and the molecule problem. Information and
Inference: a journal of the IMA, 1:21–67, 2012b.

N. Cutland. Computability: an introduction to recursive function

theory. Cambridge University Press, Cambridge, 1980.

C. D’Ambrosio and A. Lodi. Mixed-integer nonlinear programming

tools: a practical overview. 4OR, 9:329–349, 2011.

G. Dantzig, A. Orden, and P. Wolfe. The generalized simplex

method for minimizing a linear form under linear inequality
restraints. Pacific Journal of Mathematics, 5(2):183–196, 1955.

S. Dasgupta and A. Gupta. An elementary proof of a theorem by

johnson and lindenstrauss. Random Structures and Algorithms, 22:
60–65, 2002.

J. Dattorro. Convex Optimization and Euclidean Distance Geometry.

Meβoo, Palo Alto, 2015.

R. Davis, C. Ernst, and D. Wu. Protein structure determination via

an efficient geometric build-up algorithm. BMC Structural Biology,
10(Suppl 1):S7, 2010.

Ph. Delsarte. Bounds for unrestricted codes by linear programming.

Philips Research Reports, 27:272–289, 1972.

K. Dhyani. Personal communication, 2007.

Kanika Dhyani. Optimization models and algorithms for the hyperplane

clustering problem. PhD thesis, Politecnico di Milano, 2009.

Y. Ding, N. Krislock, J. Qian, and H. Wolkowicz. Sensor network

localization, Euclidean distance matrix completions, and graph
realization. Optimization and Engineering, 11:45–66, 2010.

I. Dokmanić, R. Parhizkar, J. Ranieri, and M. Vetterli. Euclidean

distance matrices: Essential theory, algorithms and applications.
IEEE Signal Processing Magazine, 1053-5888:12–30, Nov. 2015.

C. Dürr. Programmation par contraintes et programmation

mathématique, 2011. Lecture notes, École Polytechnique.

J. Edmonds. Paths, trees and flowers. Canadian Journal of Mathemat-

ics, 17:449–467, 1965.

M. Ehrgott. Multicriteria Optimization. Springer, New York, 2005.

L. Euler. Solutio problematis ad geometriam situs pertinentis.

Commentarii Academiæ Scientiarum Imperialis Petropolitanæ, 8:128–
140, 1736.

L. Evans. An introduction to mathematical optimal control theory. v. 0.2.

math.berkeley.edu/ evans/control.course.pdf.
110 mathematical programming

M. Fischetti and A. Lodi. Local branching. Mathematical Program-

ming, 98:23–37, 2005.

M. Fischetti, F. Glover, and A. Lodi. The feasibility pump. Mathe-

matical Programming, 104(1):91–104, 2005.

M. Flood. The traveling salesman problem. Operations Research, 4

(1):61–75, 1956.

L. Ford and D. Fulkerson. Maximal flow through a network.

Canadian Journal of Mathematics, 8(3):399–404, 1956.

R. Fortet. Applications de l’algèbre de Boole en recherche opéra-

tionelle. Revue Française de Recherche Opérationelle, 4:17–26, 1960.

R. Fourer and D. Gay. The AMPL Book. Duxbury Press, Pacific

Grove, 2002.

V. Giakoumakis, D. Krob, L. Liberti, and F. Roda. Technological

architecture evolutions of information systems: trade-off and
optimization. Concurrent Engineering Research and Applications, 20
(2):127–147, 2012.

K. Gödel. Über formal unentscheidbare Sätze der Principia Mathe-

matica und verwandter Systeme, I. Monatshefte für Mathematik und
Physik, 38:173–198, 1930.

R.E. Gomory. Essentials of an algorithm for integer solutions to

linear programs. Bulletin of the American Mathematical Society, 64(5):
256, 1958.

G. Gutin and A. Punnen, editors. The traveling salesman problem and

its variations. Kluwer, Dordrecht, 2002.

P. Hall. On representatives of subsets. Journal of the London

Mathematical Society, 10(1):26–30, 1935.

D. Harel. On folk theorems. Communications of the ACM, 23(7):

379–389, 1980.

W. Hart, C. Laird, J.-P. Watson, and D. Woodruff. Pyomo — Opti-

mization modelling in Python. Springer, New York, 2012.

IBM. ILOG CPLEX 12.2 User’s Manual. IBM, 2010.

IBM. ILOG CPLEX 12.6 User’s Manual. IBM, 2014.

W. Johnson and J. Lindenstrauss. Extensions of Lipschitz mappings

into a Hilbert space. In G. Hedlund, editor, Conference in Modern
Analysis and Probability, volume 26 of Contemporary Mathematics,
pages 189–206, Providence, 1984. American Mathematical Society.

N. Karmarkar. A new polynomial time algorithm for linear

programming. Combinatorica, 4(4):373–395, 1984.
 bibliography 111

R. Karp. Reducibility among combinatorial problems. In R. Miller

and W. Thatcher, editors, Complexity of Computer Computations,
volume 5 of IBM Research Symposia, pages 85–104, New York, 1972.
Plenum.

L. Khachiyan. Polynomial algorithms in linear programming. USSR

Computational Mathematics and Mathematical Physics, 23(1):53–72,
1980.

H. Konno and A. Wijayanayake. Portfolio optimization problem

under concave transaction costs and minimal transaction unit
constraints. Mathematical Programming, Series B, 89(2):233–250, 2001.

S. Krantz. The Proof is in the pudding. Springer, New York, 2010.

S. Kucherenko, P. Belotti, L. Liberti, and N. Maculan. New formula-

tions for the kissing number problem. Discrete Applied Mathematics,
155(14):1837–1841, 2007.

M. Laurent and F. Vallentin. Semidefinite optimization. Technical

Report Lecture Notes, CWI and TU Delft, 2012.

C. Lavor, L. Liberti, and N. Maculan. Computational experience

with the molecular distance geometry problem. In J. Pintér, editor,
Global Optimization: Scientific and Engineering Case Studies, pages
213–225. Springer, Berlin, 2006.

C. Lavor, A. Mucherino, L. Liberti, and N. Maculan. On the

computation of protein backbones by using artificial backbones of
hydrogens. Journal of Global Optimization, 50:329–344, 2011.

L. Liberti. Writing global optimization software. In L. Liberti and

N. Maculan, editors, Global Optimization: from Theory to Implementa-
tion, pages 211–262. Springer, Berlin, 2006.

L. Liberti. Compact linearization of binary quadratic problems.

4OR, 5(3):231–245, 2007.

L. Liberti. Reformulations in mathematical programming: Defini-

tions and systematics. RAIRO-RO, 43(1):55–86, 2009.

L. Liberti. Symmetry in mathematical programming. In J. Lee and

S. Leyffer, editors, Mixed Integer Nonlinear Programming, volume
154 of IMA, pages 263–286. Springer, New York, 2012.

L. Liberti. Optimization and sustainable development. Computa-

tional Management Science, 12(3):371–395, 2015.

L. Liberti and N. Maculan. Preface to special issue on reformula-

tions in mathematical programming. Discrete Applied Mathematics,
157(6), 2009.

L. Liberti and F. Marinelli. Mathematical programming: Turing

completeness and applications to software analysis. Journal of
Combinatorial Optimization, 28(1):82–104, 2014.
112 mathematical programming

L. Liberti, S. Cafieri, and F. Tarissan. Reformulations in mathemat-

ical programming: A computational approach. In A. Abraham,
A.-E. Hassanien, P. Siarry, and A. Engelbrecht, editors, Founda-
tions of Computational Intelligence Vol. 3, number 203 in Studies in
Computational Intelligence, pages 153–234. Springer, Berlin, 2009a.

L. Liberti, C. Lavor, N. Maculan, and F. Marinelli. Double variable

neighbourhood search with smoothing for the molecular distance
geometry problem. Journal of Global Optimization, 43:207–218, 2009b.

L. Liberti, C. Lavor, N. Maculan, and A. Mucherino. Euclidean

distance geometry and applications. SIAM Review, 56(1):3–69, 2014.

J. Löfberg. YALMIP: A toolbox for modeling and optimization in

MATLAB. In Proceedings of the International Symposium of Computer-
Aided Control Systems Design, volume 1 of CACSD, Taipei, 2004.
IEEE.

M. López and G. Still. Semi-infinite programming. European Journal

of Operational Research, 180(2):491–518, 2007.

N. Maculan, P. Michelon, and J. MacGregor Smith. Bounds on the

kissing numbers in Rn : Mathematical programming formulations.
Technical report, University of Massachusetts, Amherst, USA,
1996.

A. Majumdar, A. Ahmadi, and R. Tedrake. Control and verification

of high-dimensional systems with dsos and sdsos programming. In
Conference on Decision and Control, volume 53, pages 394–401, Los
Angeles, 2014. IEEE.

A. Makhorin. GNU Linear Programming Kit. Free Software Founda-

tion, http://www.gnu.org/software/glpk/, 2003.

A. Man-Cho So and Y. Ye. Theory of semidefinite programming

for sensor network localization. Mathematical Programming B, 109:
367–384, 2007.

H. Markowitz. Portfolio selection. The Journal of Finance, 7(1):77–91,

1952.

matlab. MATLAB R2014a. The MathWorks, Inc., Natick, MA, 2014.

Y. Matiyasevich. Hilbert’s Tenth Problem. MIT Press, Boston, 1993.

J. Matoušek and B. Gärtner. Understanding and using Linear Pro-

gramming. Springer, Berlin, 2007.

K. Menger. New foundation of Euclidean geometry. American

Journal of Mathematics, 53(4):721–745, 1931.

M. Minsky. Computation: Finite and infinite machines. Prentice-Hall,

London, 1967.

J. Moré and Z. Wu. Distance geometry optimization for protein

structures. Journal of Global Optimization, 15:219–234, 1999.
 bibliography 113

mosek7. The mosek manual, Version 7 (Revision 114). Mosek ApS,

2014. (www.mosek.com).

T. Motzkin and E. Straus. Maxima for graphs and a new proof of

a theorem of Turán. Canadian Journal of Mathematics, 17:533–540,
1965.

J. Munkres. Algorithms for the assignment and transportation

problems. Journal of SIAM, 5(1):32–38, 1957.

R. Reams, G. Chatham, W. Glunt, D. McDonald, and T. Hayden. De-

termining protein structure using the distance geometry program
APA. Computers and Chemistry, 23:153–163, 1999.

N.V. Sahinidis and M. Tawarmalani. BARON 7.2.5: Global Optimiza-

tion of Mixed-Integer Nonlinear Programs, User’s Manual, 2005.

J. Saxe. Embeddability of weighted graphs in k-space is strongly

NP-hard. Proceedings of 17th Allerton Conference in Communications,
Control and Computing, pages 480–489, 1979.

T. Schlick. Molecular modelling and simulation: an interdisciplinary

guide. Springer, New York, 2002.

A. Singer. Angular synchronization by eigenvectors and semidefi-

nite programming. Applied and Computational Harmonic Analysis, 30:
20–36, 2011.

A. Soyster. Convex programming with set-inclusive constraints and

applications to inexact linear programming. Operations Research, 21
(5):1154–1157, 1973.

A. Sutou and Y. Dai. Global optimization approach to unequal

sphere packing problems in 3D. Journal of Optimization Theory and
Applications, 114(3):671–694, 2002.

J. Tenenbaum, V. de Silva, and J. Langford. A global geometric

framework for nonlinear dimensionality reduction. Science, 290:
2319–2322, 2000.

A. Turing. On computable numbers, with an application to the

Entscheidungsproblem. Proceedings of the London Mathematical
Society, 42(1):230–265, 1937.

R. Vanderbei. Linear Programming: Foundations and Extensions.

Kluwer, Dordrecht, 2001.

S.A. Vavasis. Nonlinear Optimization: Complexity Issues. Oxford

University Press, Oxford, 1991.

S. Venkatasubramanian and Q. Wang. The Johnson-Lindenstrauss

transform: An empirical study. In Algorithm Engineering and
Experiments, volume 13 of ALENEX, pages 164–173, Providence,
2011. SIAM.
114 mathematical programming

J.P. Vielma. Mixed integer linear programming formulation

techniques. SIAM Review, 57:3–57, 2015.

D. Wu, Z. Wu, and Y. Yuan. Rigid versus unique determination of

protein structures with geometric buildup. Optimization Letters, 2(3):
319–331, 2008.

K. Wüthrich. Protein structure determination in solution by nuclear

magnetic resonance spectroscopy. Science, 243:45–50, 1989.

Y. Yemini. The positioning problem — a draft of an intermedi-

ate summary. In Proceedings of the Conference on Distributed Sensor
Networks, pages 137–145, Pittsburgh, 1978. Carnegie-Mellon Uni-
versity.
Index

k-consistent, 69 inner, SDP, 103 binary

1-consistency, 69 ratio, 76, 83 variable, 55
1-norm, 95 scheme, 76 binary encoding, 53
2-consistency, 70 approximation algorithm, 76, 77, 85 bipartite
2-norm, 96, 104 constant, 76 graph, 70
randomized, 77 bisection, 21, 71
absolute value, 52 approximation error, 36 algorithm, 71
accuracy, 42, 82 approximation quality, 101 complexity, 71
arbitrary, 39 approximation reformualation, 51 bit
adjacency matrix, 39 arc, 19 storage, 20
weighted, 81 antiparallel, 79 blending, 27
adjacent, 31 capacity, 19 BLevP, 44
vertex, 70 arc capacity, 44 BLP, 30, 53, 79
algorithm arc-consistency, 70 BONMIN, 37
analysis, 20 architecture, 61 bound
approximation, 75, 76, 100 spaghetti, 61 guarantee, 103
cMINLP, 37 array guaranteed, 103
exact, 76 bi-dimensional, 24 lower, 71
GW, 85–87 multi-dimensional, 9, 24 upper, 71
iteration, 105 partially defined, 72 bounded above, 76
iterative, 102 slice, 24 bounded below, 76
Monte Carlo, 76 square, 41, 72 bounded probability, 76
polytime, 99, 100 assembly language, 25 BQCP, 78
randomized, 75–77, 104, 105 assignment, 16, 27, 53, 57 BQP, 39, 79
randomized approximation, 77, 91 constraint, 53 branch, 36
randomized rounding, 88 feasible, 70 branch-and-bound, 17, 27
AllDifferent, 70–72 linear, 30 spatial, 36
alphabet, 13 variables, 58 branching, 36, 53, 69
AMPL, 23, 59 assignment constraint, 29 variable, 69
dat file, 24, 25 asymptotic complexity, 20 branching variable, 70
dat syntax, 24 atom, 75, 94, 95 brute force, 76
IDE, 24 position, 41 byte, 14
mod file, 24 axiomatization, 93
run, 24 canonical form
run file, 24 backtracking, 69 LP, 27
angle, 82 BARON, 36, 37, 39, 40 capacity, 18
extent, 83 barrier method, 17 carbon, 94
applied mathematics, 68 barycenter, 97 cardinality constraint, 36
approximate algorithm, 104 battery cartesian, 69
approximate feasibility, 100 power, 94 cartesian product, 24, 28, 36, 37
approximate method, 100 BB, 27, 28, 33, 36, 56 CCMVPS, 38
approximation, 33, 39, 101, 102 node, 36 cell, 13
accurate, 105 big M, 72 certificate, 71
improvement, 103 bijection, 29 PSD, 85
inner, 100–103 bilevel programming, 44, 45 chain, 102
116 mathematical programming

characteristic vector, 39 restrictive, 103 covariance matrix, 36, 38

scaled, 39 cone, 34 CP, 9, 67, 68, 71, 93
chessboard, 67 cone programming dual, 67, 68
Cholesky second-order, 42 formulation, 67, 71, 72
factor, 103 congruence, 95 language, 67
Cholesky factor, 102 conjunction, 75 CP formulation
Church-Turing thesis, 16 conjunctive normal form, 75 original, 68
circle, 34 connectivity, 19 CPLEX, 40, 60, 79, 85, 90
slice, 83 consistency, 69 convex MIQP, 85
circle packing, 34 arc, 70 MILP, 85
circuit, 75 node, 69 nonconvex MIQP, 85
clause, 75 consistent sBB, 85
clauses domain, 69 system clock, 86
satisfiable, 75 constant time, 20 v. 12.6, 85
Clique, 51 constraint, 21, 53, 67, 69, 70, 78, 93 CPU, 14
clique, 33, 61 k-ary, 69 time, 86, 90, 91, 103
largest, 39 additional, 72 timings, 86
max, 33 assignment, 58, 62 user, 86
maximal, 39 binary, 53, 67, 69 CPU time, 20
maximum, 39 cardinality, 36, 72 cQP, 37, 38
clock, 94 convex nonlinear, 85 crossword, 67
synchronization, 94 convex quadratic, 85 crystal, 75
clustering, 57 DD, 101 customer site, 18
Euclidean, 105 disjunction, 72 cut, 39, 75, 77, 91
images, 105 dual, 67 maximum, 39
cMINLP, 36 equality, 52 cutset, 75
cNLP, 33, 35, 36 infinite, 46 maximum, 77–79, 82
code integrality, 21, 53 maximum weight, 75
parallelized, 86 nonconvex, 36 size, 78
code efficiency, 86 nonlinear, 93, 95 cutset size
column, 72, 80, 85 number, 72 average, 83
combinatorial PSD, 41, 101 CvxOpt, 85
property, 55 quadratic, 40, 53 cycle
combinatorial optimization, 75 qualification, 36 disjoint, 29
command line, 89, 90 rank, 41, 79
argument, 86 reliability, 62 Dantzig, 27
commodity, 18, 19 simple, 21 data transfer, 86
comparison unary, 67, 69 DD, 101–103
empirical, 86 uncountable, 47 inner approximation, 103
compiled, 25 uncountably many, 47 decision problem, 20, 27, 68, 72, 76,
complement, 75 constraint matrix, 26 77
complement graph, 33 constraint programming, 9, 67 decision variable, 21, 41, 43, 44, 47,
complementarity, 55 contradiction, 69 67, 68, 78
constraint, 56 contrapositive, 71 binary, 29
linear, 55 control transfer, 86 bounded, 67
complexity control variables, 47 continuous, 29
worst-case, 20, 36, 100 convex discrete, 67
component, 105 function, 33, 34 matrix, 40, 101
composition, 15 set, 33 scalar, 80
computation, 13 subset, 47 symbol, 22
computation model, 14, 15 convex relaxation, 36 vector, 80
computational model, 33 convexity, 33 decision version, 20
computer cost, 61 decomposition, 94
multi-core, 86 minimization, 44 demand, 18
real, 17 unit transportation, 18 derivative, 22
computer science, 68 COUENNE, 37, 39, 40 device, 94
condition Couenne, 36 mobile, 94
 index 117

DG, 93 EDM, 96, 99 set, 22

fundamental problem, 93 partial, 104 solution, 22
DGP, 93, 95 square, 96, 98, 99 feasible point, 35, 36
application, 100 EDMCP, 99, 100 feasible polyhedron, 56
solution methods, 96 algorithm, 104 feasible region, 69
diagonal, 82 efficiency feasible set, 102
diagonal component maximization, 57 feasible solution, 68, 103
strictly positive, 103 eigenvalue, 38, 88, 98 file
diagonal dominance, 56 negative, 88, 98, 104 format, 90
diagonally dominant, 101 negative zero, 88 line, 90
scaled, 103 non-negative, 38, 41, 98 finite difference
diameter, 83 ordered, 85 equation, 47
weighted, 95 slightly negative, 85 fixed cost, 28
dictionary, 13 small negative, 88 fKNP, 35
random, 26 strictly positive, 38 floating point, 85, 86
differentiable, 22 eigenvector, 82, 85, 88 error, 88
digraph, 19, 30, 81 element error tolerance, 86
dimension, 105 one, 70 number, 90
dimensionality reduction, 104 upper-left, 72 operation, 88
direction ellipsoid algorithm, 27 flow, 19, 27, 44
feasible improving, 35 encoding conservation, 20
discrepancy, 36 finite, 47 positive, 20
Disjunction, 72 energy total, 20
disjunction, 75 minimum, 75 form
distance, 93–95 enumerative method, 17 linear, 41
Euclidean, 93, 105 equality constraint, 52 quadratic, 41
euclidean, 36 equation, 40, 52 standard, 27
inter-atomic, 41, 94 quadratic, 53 format
minimum, 57 erase, 13 tabular, 90
pairwise, 105 Erdős-Renyi formula
distance geometry, 35, 93 model, 90 logical, 75
Distance Geometry Problem, 93 error, 88 formulation, 22, 23, 34, 96, 100
distribute floating point, 100 bilevel, 45
products over sums, 63 minimization, 94 exact, 100
distribution tolerance, 33, 85 MIQP, 86
normal, 105 error minimization, 38 MP, 27, 44
domain, 21, 67–70 eternity, 67 natural, 78, 79
bounded, 69 exact algorithm, 104 quadratic, 79
reduction, 69, 70 exact reformulation, 51 SDP, 99, 101, 103
domain product, 70 execution environment, 17 structured, 68
dual, 103 expected reutrn, 36 Fortet
LP, 103 exponential, 22, 53 inequalities, 87
SDP, 103 exponential time, 76 Fortet reformulation, 54
duality, 69 expression Fourier, 27
boolean, 75 FPTAS, 76, 82
economic equilibrium, 27 mathematical, 22 fractional value, 36
edge, 35, 77, 79, 99 frequency, 94
list, 90 facet, 47 function, 20
probability, 90 factor, 79, 102 basic, 15
set, 78 Cholesky, 102, 103 computable, 16
total number, 78 transpose, 88 constant, 15
weight, 78, 90 factorization, 41, 85 convex, 33, 36
weight function, 93 feasibility, 40, 54, 82, 93 nonlinear, 27, 34, 37
EDGP, 35, 41, 93, 95, 99, 100, 105 strict, 82 objective, 21
algorithm, 104 feasibility problem, 27, 32, 67 partial recursive, 14
instance, 41 feasible, 94 projection, 15
relaxation, 42 approximately, 100 successor, 15
118 mathematical programming

trigonometric, 22 hypercube, 30 IPM, 42, 81, 100

function call, 24 hyperedge, 31 SDP, 82
hypergraph, 31, 33 IPOPT, 33, 35, 39, 40, 106
Gödel, 93 cover, 31 irrational, 33
Gödel number, 14, 16 hyperplane, 38, 57, 82, 93 irrational component, 34
gap Hyperplane Clustering Problem, 57 Ising model, 75
minimum, 84 Isomap, 104
general-purpose, 43, 45 ILOG, 60 iterations
geometry, 93 ILP, 30 maximum number, 86
elementary, 83 image database, 105 iterative algorithm, 35
Gershgorin’s theorem, 101 implementation, 85
global optimality, 77 implication, 72 JLL, 105
global optimization, 96 Implies, 72 proof, elementary, 105
global optimum, 94 incompleteness theorem, 93 Johnson-Lindenstrauss lemma, 105
GO, 96, 106 indefinite, 38
goal, 45 index, 29 k-means, 105
gobal solution, 36 indexing, 72 Kantorovich, 27
Goemans-Williamson, 82 indicator vector, 31 kissing number, 35, 37
Gram matrix, 80, 97 inequality Koopmans, 27
partial, 99 Fortet, 87 Kronecker, 93
graph, 35, 39, 61 linear, 38
adjacency matrix, 39 opposite sign, 40, 52 Lagrange coefficient, 36
bipartite, 30, 70 infeasibility, 105 Lagrangian, 36
connected, 94 error, 106 language
diameter, 95 information, 99 AMPL, 24
directed, 19, 62 input, 13, 69, 76, 79, 99 declarative, 17
directed complete, 79 input data, 20 general purpose, 25
drawing, 35 input graph, 94 high-level, 25
edge-weighted, 99 instance, 21–23, 25, 71, 76, 77 imperative, 17, 24, 25
mesh grid, 91 CP, 69 interpreted, 25
random, 90 file, 90 LAP, 30
simple, 31, 33, 42 finitely many, 20 Laplacian
simple undirected, 93 infeasible, 71 matrix, 87
triangle, 89 NO, 20, 35, 76 lattice
undirected, 86 size, 33 integer, 27, 28, 30, 36, 37
undirected edge-weighted, 79 YES, 20, 35, 76 layer, 75
weighted, 90, 94, 100 instance size, 34 LCP, 55
growth small, 35 leaf node, 70
logarithmic, 105 instantiation, 23 left, 13
slow, 105 instruction, 13 LFP, 57
guarantee basic, 14 library
exactness, 76 imperative, 16 standard, 86
GW, 82, 83, 85, 91 integer, 90 lifting, 52
consecutive, 53 line, 93
Hall’s theorem, 70 lattice, 30 linear, 17
halting problem, 14 smallest, 53 linear complementarity, 55
hash table, 20 integral, 47 linear algebra, 41
HCP, 57, 59 integrality constraint, 21 computation, 86
head, 13 integrality constraints, 36 linear assignment, 70
heuristic, 37, 69, 105 interface, 62 linear combination, 80
algorithm, 104 interface module, 61 linear form, 27, 28, 30, 40–42
k-means, 105 interior point, 27, 42 linear fractional
method, 51 method, 27 programming, 57
hierarchical stakeholders, 45 interpreter, 14, 17, 22 linear fractional term, 57
Hilbert, 93 interval linear programming, 27
Hitchcock, 27 endpoint, 47 binary, 30
hydrogen, 94 range, 21 integer, 30
 index 119

mixed-integer, 28 factor, 87, 98, 102 binary, 53

linear regression, 33 Gram, 41, 97, 98, 101 formulation, 22
restricted, 38 identity, 81, 102 modelling environment, 85
linear term, 52 lower triangular, 102 solver, 86, 93
linearization, 40, 41, 54, 55 partial, 99, 100 subclass, 69
exact, 79, 95 PSD, 38, 41, 80, 98, 99, 101 symmetric, 51
product, 54 PSD, closest, 104 MP class, 51
Linux, 24 rank 1, 91 MP formulation, 27
literal, 75 real symmetric, 80 multi-objective, 22
local optimum, 36, 37 SDP, 103 multi-objective programming, 43
localization sequence, 102 MVCCPS, 36
sensor network, 94 symmetric, 80, 85, 99
log-space reductions, 27 unitary, 82 narrowing reformulation, 51
logarithm, 22, 35, 63 zero-diagonal, 96, 99 natural number, 14
logarithmic, 21 matrix problem, 79 negative
logical connective, 72 max clique, 33 definite, 38
logistics, 29 Max Cut, 76, 79, 85 semidefinite, 38
loop, 16, 103 formulation, 78 neighbourhood, 22
lower bound, 77 instance, 82 network
LP, 27, 41, 55, 100–103 MILP formulation, 85 wireless, 94
approximating, 102 MIQP formulation, 85 network flow, 19
canonical form, 52 relaxation, 79 nine queens, 67
dual, 103 SDP relaxation, 85 NLP, 27, 34, 95, 96
family, 101 solution, 82 feasibility, 93
parametric, 102 unweighted, 77, 91 nonconvex, 38
parametrized, 101 max cut, 39 solver, 106
sequence, 102 Max Flow, 20, 44 NLP solver, 53
solution, 102 Max-2-Sat, 75 NMR, 35, 94
standard form, 52 maximization, 22 Nobel prize, 36
LP relaxation, 36 mean, 105 node, 19, 38, 69
megawatt, 28 child, 69
machine code, 25 memory, 14, 34 customer, 44
MacOSX, 24 method leaf, 70
magnetic spin, 75 iterative, 101 prior, 71
manifold, 93 metric, 93 production, 44
Markowitz, 36 midpoint, 71 pruned, 70
matching, 70 MILP, 27, 28, 36, 56, 58, 63, 75, 90, 95 search tree, 70
maximum, 70, 71 reformulation, 59 node-consistency, 69
perfect, 30, 70 solver, 59 node-consistent, 69
matching problem, 32 minimalization, 15, 16 nonconvex, 34, 38
material minimization, 22 NLP, 35
balance, 20 error, 93 nonconvex MINLP, 37
mathematical programming, 9, 17 maximum, 54 nonconvex NLP, 36
MATLAB, 42 unconstrained, 96 nonconvexity, 36
MatLab, 85 MINLP, 36, 37, 58, 62 nonlinear, 17
matrix, 38, 41, 99, 101 solver, 59 nonlinear programming, 34
adjacency, 39 MIQP, 75, 90 convex, 33
binary, 39 mixed-integer convex mixed-integer, 36
Cholesky, 102 nonlinear programming, 37 mixed-integer, 37
closest PSD, 85, 104 modelling, 15 nonlinear term, 54
component, 45 modelling language, 28 nonzero, 53
covariance, 38 molecule, 94 norm, 34
DD, 103 monotonically increasing, 63 Euclidean, 34, 95
diagonal, 81, 85, 103 Monte Carlo, 76 infinity, 95
diagonanly dominant, 101 MOP, 22, 43, 45 max, 95
distance, 99 MOSEK, 42 one, 95
Euclidean distance, 96 MP, 9, 21, 27, 55, 62, 68, 71, 93 two, 96
120 mathematical programming

NP, 20, 100 OS, 86 input, 67

hardest, 20 output, 13, 99 maximization, 76, 77
NP-complete, 76 format, 90 minimization, 76, 77
NP-hard, 20, 28, 32–35, 51, 55, 67, 68, optimization, 22, 70
76, 99 P, 20, 33, 100 transportation, 18
proof, 99 P6=NP, 51 problem structure, 100
nuclear magnetic resonance, 35, 94 p.r., 14, 17 processor, 30
number of bits, 34 pairwise intersection, 34 product, 80, 95
number theory, 15 parameter, 21, 22, 24, 44, 47, 52, 67, binary, 63
NumPy, 85 69, 78 binary variables, 79
integer, 72 reformulation, 55
objective, 23, 45 symbol, 22, 23 scalar, 80
approximation, 82 tensor, 24 product term, 56
coefficients, 26 vector, 47, 67, 71 production facility, 18
direction, 27 parameter matrix, 102 program, 17
nonconvex, 36 Pareto main, 89
objective function, 21, 44, 46, 67, 71, optimum, 43 MP, 17
78, 79, 81 region, 43 programming language, 37, 67
average value, 77 set, 43 declarative, 68
nonconvex, 85 Pareto set, 22, 44 projection, 34, 52
optimal value, 103 infinite, 43 random, 104
quadratic, 85 partitioning proof
value, 90, 94 graph, 63 hand-waving, 93
zero, 93 path, 19 protein, 94, 95
open source, 23 path length, 47 conformation, 94
operating system, 86 PECS, 34 protein conformation, 35, 41
operation permutation, 29, 30, 70 protocol
basic, 15 Picos, 85 synchronization, 94
operator, 52 plane, 93 pruning, 69–71
arithmetical, 22 point, 38, 93 PSD, 41, 80, 99, 101
transcendental, 22 closest, 36 approximately, 100
optimal control, 47 feasible, 76 cone, 103
optimal order, 28 integral, 36 PSD matrix, 38, 88
optimal point, 36 polyhedron, 27, 28, 30, 37–39 PSDCP, 99
optimal value, 36 polynomial PTAS, 76
optimality, 82, 93 quartic, 96 PyOMO, 25
certificate, 34 polynomial size, 34 Pyomo, 85, 86
global, 39 polynomial time, 27, 28, 30, 33, 35, 42 Python, 25, 42, 85, 90
guarantee, 35, 59 reformulation, 51 CvxOpt, 85
local, 39 polytime, 20, 21, 32, 70, 71, 81, 98, cvxopt, 86
optimality condition, 36 100 math, 86
optimality guarantee randomized, 77 MP, 85
global, 37 polytope, 47 networkx, 86
optimization portfolio, 36 numpy, 86
black-box, 22 Markowitz, 38 Picos, 85
problem, 17 position, 95 picos, 86, 87
optimization problem, 15, 20 positive program, 86, 90
optimum, 56, 77 definite, 38 pyomo, 86
global, 22, 33, 35, 39, 58, 76 semidefinite, 38 sys, 86
local, 22, 33, 35, 39, 51, 55 power generation, 28 time, 86, 89
Pareto, 43 pre-processing, 69
strict, 39 precedence relation, 30 QCQP, 40, 53, 96
orbit, 51 primal, 103 feasibility, 40
ordered pair, 29 probability, 83 QP, 38, 41
origin, 82, 97 problem indefinite, 39
original problem, 42, 51 decision, 35 nonconvex, 38
orthant, 28 halting, 37 original, 41
 index 121

SDP relaxation, 41 resource allocation, 29 sensor network, 35

quadratic rest, 13 sequence, 35
equation, 40 restriction, 52 server, 94
quadratic form, 37–41 RHS, 23, 26, 78 set, 67
quadratic programming, 38 right, 13 convex, 33, 36
binary, 39 rLR, 38 edge, 71
convex, 37 RM, 14 feasible, 43
quadratically constrained, 40 Minsky, 37 finite, 54
quadruplet, 13 robust, 85 second-order conic, 42
robust optimization, 47 uncountable, 47
radius, 34, 83 row, 38, 72 vertex, 70
largest, 34 row generation, 29 set covering, 31
random projection, 105 RP, 76 set packing, 32
EDGP, 105 RR, 90, 91 set partitioning, 32
matrix, 105 rule, 13 shortest path
randomized algorithm, 76 weight, 95
approximation, 77 SAT, 55, 67 simplex
randomized rounding, 82–84, 90 sBB, 36, 37, 39, 40, 85 method, 27
phase, 86, 87 algorithm, 38 simplex method, 17, 27, 56
randomness, 76 node, 36 single level, 45
range scalability, 59 single-level, 45
discrete, 21 scalar, 34 single-objective, 43
rank, 98 scalar product, 83 single-user mode, 86
column, 80 scaled diagonally dominant, 103 SIP, 46, 47
constraint, 41 scheduling, 29, 30, 67 size
one, 41 Schur complement, 100 increasing, 20
row, 80 SCIP, 36, 37 slack variable, 52
rank constraint, 79 SDD, 103 SNOPT, 33, 35, 39, 40
ratio, 76 SDP, 37, 40, 41, 75, 81, 82, 90, 99–101, SOCP, 42, 103
realization, 35, 41, 96, 98, 99 103 solver, 86
function, 93 dual, 103 software
recursion formulation, 100 architecture, 61
primitive, 15 instance, 82 component, 61
reduction, 35 polytime, 82 system, 61
polytime, 20 relaxation, 79, 81–83, 101 software architecture, 61, 62
reformulation, 51, 56, 80, 95 small, 100 solution, 21, 53, 77
approximation, 51 solution, 82 acceptable, 105
automatic, 64 solver, 42, 85, 86, 100 approximate, 21, 42, 76
exact, 51, 57, 79, 80 variable, 100 aulity, 103
narrowing, 51 SDP relaxation, 53 candidate, 70, 71
polynomial time, 51 phase, 87 feasible, 19, 51, 54, 91
product, 85 search algorithm, 70 heuristic, 105
relaxation, 51 search quantifier, 15 improvement, 54
region search tree, 70, 71 rank 1, 82
feasible, 102 seconds SDP, 105
register, 14 CPU, 86 slightly infeasible, 85
register machine, 14 SeDuMi, 42 YES, 20
relation semantic interpretation, 64 solver, 17, 22, 23, 35, 59, 69
k-ary, 67 semantic relationship, 62 efficient, 51
relations, 69 semantics, 14 global, 35
relaxation, 51 semi-infinite optimization, 22 local, 35
convex, 38 semi-infinite programming, 46 local NLP, 33, 35, 39, 54, 55
EDGP, 42 semidefinite, 37 polytime, 51
SDP, 42, 79, 85 cone, 40 sBB, 85
tight, 100 positive, 38 SDP, 42
reliability, 61 semidefinite programming, 40 solver-dependent
representative, 51 semimetric, 93 property, 56
122 mathematical programming

sonar, 95 term binary, 53, 72, 85

source, 20 quadratic, 63 boolean, 75
space termination, 36 bounded, 55
Euclidean, 36, 37, 93 termination condition, 103 continuous, 28, 53
high-dimensional, 105 test, 16 continuous unbounded, 93
sphere packing, 27 threshold, 94, 103 control, 47
spin glass, 75 time decision, 21, 72
SQP, 36 difference, 94 discrete, 53
square, 34, 96 instant, 95 dual, 67
EDM, 96 time horizon, 28, 30 index, 67
unit, 34 time limit, 86 integer, 17, 28, 36, 63
square root, 86 TM, 13, 33, 37 integer bounded, 93
function, 88 universal, 37 linearization, 63
stability tolerance, 36, 37, 82 matrix, 102
numerical, 59 total cost, 29 non-negative, 52
stable total CPU, 86 product, 85
largest, 33 total energy, 47 relation between, 69
standard form, 30 tour, 29 slack, 52, 56, 72
standard deviation, 105 trade-off, 105 surplus, 52, 56
standard form, 27, 28, 30, 33, 34, transcendental, 33 symbol, 68
36–40, 42–44, 46 translation, 95, 97 unbounded, 95
LP, 52 transportation uncountable, 47
star, 61 cost, 18 variable domain
state, 13, 16 problem, 18 discrete bounded, 71
final, 13 random instance, 23 vector
initial, 13 transportation problem, 19, 23, 25, 27 binary variable, 53
step, 16 traveling salesman, 28 column, 80
sub-matrix, 72 triplet, 94 vertex, 47, 56
subarine, 95 TSP, 28 adjacent, 90
subdomain, 37 tuple, 24, 69 cover, 31
subgraph Turing complete, 37, 67, 68 incident, 75
densest, 62 Turing machine, 13 pair, 95
fully connected, 61 universal, 14 set, 78
subproblem, 35 vertex cover, 31
subspace, 34 unboundedness, 21 VLSI, 75
subtour, 29 uncountable, 47
subtour elimination, 29 undefined, 16 weight
sudoku, 67, 72 underwater vehicles maximum, 75
surplus variable, 52 unmanned, 95 weighted matching, 30
symbol, 13 undirected, 31 wire
sequence, 13 unit revenue, 45 intersection, 75
system universality, 14 write, 13
software, 61 unix, 24
system clock, 86 upper bound, 77
system CPU, 86 upper-level, 46 YALMIP, 42
user CPU, 86 YES, 69, 70
tabular form, 24
tape, 13, 16 variable, 69, 79 Zermelo-Fraenkel-Choice, 93
target, 20 additional, 41, 52–55, 58, 62, 72 zero, 88
task, 30 assignment, 63 ZFC, 93
tensor, 9 auxiliary, 52 zip, 26