Introduction to CENSO

A tutorial for downloading, installing and using CENSO ,

with example problems.

June 3, 2015
Contents
1 Introduction 3
1.1 Intent and reader prerequisites . . . . . . . . . . . . . . . . . 3
1.2 Types of problems solved . . . . . . . . . . . . . . . . . . . . 3
1.3 Supported constraint classes . . . . . . . . . . . . . . . . . . . 4
1.4 Mathematical background . . . . . . . . . . . . . . . . . . . . 6
1.4.1 The epigraph form . . . . . . . . . . . . . . . . . . . . 6
1.4.2 Convexity and convex relaxations . . . . . . . . . . . . 7
1.4.3 The B-spline . . . . . . . . . . . . . . . . . . . . . . . 8
1.4.4 Global optimization and the Branch-and-Bound algo-
rithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6.1 Ipopt . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.2 Eigen . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Installation 10
2.1 Getting system packages . . . . . . . . . . . . . . . . . . . . . 10
2.2 Getting the code . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 External code . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Compiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Using CENSO through C++/Qt 11

3.1 Code structure . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4 Formulation of an optimization problem 13

4.1 Type denitions . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 Smart pointers . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.3 Variable bounds . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.4 Variable types . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.5 Starting point . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.6 Branching variables . . . . . . . . . . . . . . . . . . . . . . . . 14
4.7 The objective function . . . . . . . . . . . . . . . . . . . . . . 14
4.8 The constraints . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.9 The Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5 Solvers 18
5.1 Ipopt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2 CENSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2.1 Basic algorithm parameters . . . . . . . . . . . . . . . 18
5.2.2 Branch-and-Bound specic parameters . . . . . . . . . 19
5.3 Bonmin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1
6 Example problems 22
6.1 Testing framework and the TestProblem class . . . . . . . . . 22
6.2 Farming problem . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.2.1 Linear programming case . . . . . . . . . . . . . . . . 23
6.2.2 Quadratic programming case . . . . . . . . . . . . . . 26
6.2.3 Integer variables . . . . . . . . . . . . . . . . . . . . . 30
6.3 Network ow problem . . . . . . . . . . . . . . . . . . . . . . 33
6.3.1 Simple maximum ow case . . . . . . . . . . . . . . . 33
6.3.2 Maximum ow with routing . . . . . . . . . . . . . . . 37
6.3.3 Pressure driven ow case - Linear ow model (INCOM-
PLETE) . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.4 The Six-Hump Camelback function . . . . . . . . . . . . . . . 44
6.4.1 Formulating the optimization problem . . . . . . . . . 45
6.4.2 Domain bounds, starting point, objective function and
constraint set. . . . . . . . . . . . . . . . . . . . . . . . 45
6.4.3 Local optimization using the ConstraintPolynomial
class . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.4.4 Local optimization using a custom constraint class . . 49
6.4.5 Global optimization using Alpha-Branch-and-Bound . 55
6.4.6 Global optimization using a B-spline approximation . 69

2
1 Introduction
1.1 Intent and reader prerequisites
This manual intends to give the reader a quick introduction to the CENSO
framework, so (s)he will be able set up and solve optimization problems.
It will focus on the practical aspects rather than the theory behind the
concepts used in the solver algorithm. Therefore, the reader should at least
be acquainted with the following topics within the eld of optimization:
• Mathematical formulation of mixed-integer nonlinear programming (MINLP)
optimization problems.

• Convexity, and the importance of convexity within optimization. A

good introduction to this topic can be found in chapters 2-4 of Boyd
and Vandenberghe's Convex Optimization [2].

• Interior-point algorithms. Ipopt is an interior-point algorithm, and

it is the default solver for local optimization problems in CENSO.
Good introductions on interior-point algorithms are found in chapter
11 of Convex Optimization [2] and chapters 14 and 19 of Nocedal and
Wright's Numerical Optimization [5].

• Branch-and-Bound algorithms. An introduction can be found in e.g.

[3].
TODO: Describe CENSO!!!!
1.2 Types of problems solved
CENSO is a framework for solving general mixed-integer nonlinear programs
(MINLPs) on the form

minimize f (x) = c> x, (1a)

subject to cL
i ≤ ci (x) ≤ cU
i , i = {1, . . . , m} (1b)
x ≤ x ≤ xU ,
L
(1c)
>
where x = x> x> . xc ∈ Rnc is a vector of continuous optimization

c i
variables and xi ∈ Z is a vector of integer optimization variables. The
ni

total number of optimization variables is n = nc + ni . The presence of

integer variables and nonlinear constraints makes this problem a MINLP.
• Equation (1a) shows the objective function f : Rnc × Zni 7→ R, which
is assumed to be linear in the optimization variables. This does not
lead to any loss of generality, since any optimization problem can be
written this way by converting it to its epigraph form (see section
1.4.1). c ∈ Rn is a vector of constants.

3
 • Equation (1b) shows the constraints of the MINLP. Each ci : Rnc ×
Zni 7→ R is a constraint function, cL
i ∈ (R ∪ {−∞}) is the lower bound
for the constraint function and ci ∈ (R ∪ {∞}) is the upper bound
U

for the constraint function. An equality constraint is specied by set-

ting cL
i = ci = ci (x). The number of constraints is m. No particular
U

assumptions are made about the constraint functions; they can be lin-
ear or nonlinear, convex or nonconvex. However, to ensure predictable
solver behaviour, they should be twice continuously dierentiable.

• Equation (1c) shows the domain bounds of each variable. xL ∈ (R ∪

{−∞})n is a vector of lower bounds and xU ∈ (R ∪ {∞})n is a vector
of upper bounds. Even though these constraints could be included in
(1b), they are stated explicitly here for the sake of simplicity.

1.3 Supported constraint classes

Even though any constraint can be implemented in CENSO, a few common
classes of constraints have been added to the framework for simple imple-
mentation in the MINLP. The variables used in these constraints can be any
of the optimization variables, continuous and/or integer. Some of the con-
straints specify relationships between a subset of the optimization variables.
In these cases we will use the notation x̃ ∈ Rq × Zr , q ≤ nc , r ≤ ni for a sub-
set of the optimization variables. The constraint classes which are currently
supported are:

• Linear constraints on the form

c̃L ≤ Ax̃ ≤ c̃U , (2)

Recall that a linear equality constraint on the form Ax̃ = b can be

specied by letting c̃L = c̃U = b. Here, c̃L ∈ (R ∪ {−∞})q+r and
c̃U ∈ (R∪{∞})q+r are vectors of upper and lower bounds corresponding
to the subset of the optimization variables.

• Quadratic constraints on the form

> >
cL U
i ≤ x̃ P x̃ + q x̃ + r ≤ ci , (3)

where P is a (q + r) × (q + r) matrix (not necessarily symmetric), and

q ∈ Rq+r and r ∈ R are constant vectors.

• One-dimensional quadratic constraints on the form

cL 2 U
i ≤ ax0 + bx0 + c − x1 ≤ ci , (4)

where a, b and c are constants, and x0 and x1 are any two optimization
variables.

4
 • Bilinear constraints on the form

cL U
i ≤ ax0 x1 − x2 ≤ ci , (5)

where a is a constant, and x0 , x1 and x2 are any three optimization

variables.
• Exponential constraints on the form

cL
i ≤ ae
bx0
− x1 ≤ cU
i , (6)

where a and b are constants, and x0 and x1 are any two optimization
variables.
• Polynomial constraints on the form

cL U
i ≤ p(x̃) ≤ ci . (7)

The polynomial p(x̃) consists of s monomials, and is specied by a

vector c ∈ Rs and an s × (q + r) matrix E = {ei,j }:
e e e
p(x̃) = c0 · x̃00,0 · x̃10,1 · ... · 0,q+r−1
x̃q+r−1
e e e1,q+r−1
+ c1 · x̃01,0 · x̃11,1 · ... · x̃q+r−1
(8)
+ ...
e e es−1,q+r−1
+ cs−1 · x̃0s−1,0 · x̃1s−1,1 · ... · x̃q+r−1 .
To clarify with an example, the polynomial dened by
   
  5 4 2 0
x1 3 1 0 8
x̃ = x3  , c = 
10 , E = 0 0
  ,
1
x8
1 1 0 0
with s = 4, would look like

p(x̃) = 5x41 x23 + 3x1 x88 + 10x8 + x1 .

• Sine function constraints on the form

cL U
i ≤ a sin(bx0 ) − x1 ≤ ci , (9)

where a and b are constants, and x0 and x1 are any two optimization
variables.
Constraints which do not t into any of the classes above, can be imple-
mented in one of two ways:
• Creating a new constraint class which matches the constraint exactly.
• Sampling the constraint in a grid and create a B-spline approximation
of the constraint (see section 1.4.3 for details).

5
1.4 Mathematical background
This section discusses a few of the mathematical concepts needed to better
understand how CENSO works. The epigraph form is a transformation of
an optimization problem to a form with a linear objective, which we need
in CENSO. Convexity and convex relaxations are important topics in op-
timization theory in general, and in Branch-and-Bound algorithms such as
CENSO, good convex relaxations are essential for a global solution to be
found. The B-spline is a powerful approximation and interpolation tech-
nique that can be used in CENSO to approximate nonlinear functions with
piecewise polynomials. Also, we can easily nd good convex relaxations of
B-splines, which is a very useful property. We also introduce global opti-
mization and the Branch-and-Bound algorithm.

1.4.1 The epigraph form

Before implementing the optimization problem in CENSO, we state the
problem in its epigraph form. This is done by taking the original minimiza-
tion problem (1), introducing an additional slack variable t, and minimizing
t subject to an additional constraint which says that t should be greater than
or equal to the original objective f (x):

minimize t, (10a)
subject to f (x) − t ≤ 0, (10b)
cL
i ≤ ci (x) ≤ cU
i , i = {1, . . . , m} (10c)
L
x ≤x≤x , U
(10d)

That is, we want to nd the smallest t that lies on or above the graph space of
f (x). This is illustrated in Figure 1. We now have a new objective function
which is linear (and convex). This conversion is always possible, meaning we
can assume that we only have to deal with nonconvexity and nonlinearity in
the constraints. From Figure 1 we see that (x∗ , t∗ ) is optimal for (10) if and
only if x∗ is optimal for (1) and t = f (x).

6
 (a) Standard form (b) Epigraph form. Note that (10b) con-
strains feasible points (x, t) to lie on or
above the graph space of f (x), which we
denote epi f .

Figure 1: Graphical representation of a standard form optimization problem

and an epigraph form optimization problem.

1.4.2 Convexity and convex relaxations

Convex optimization problems. It is well known that a solution to an
optimization problem obtained by a local solver such as Ipopt is not neces-
sarily the global solution. However, if the optimization problem is convex, we
can guarantee that the solution is the global solution. For an optimization
problem to be convex, the following conditions must be satised:

• The objective function f (x) must be convex. This is always the case
in (10), since the objective function (10a) is linear.

• The inequality constraints must be convex.

• The equality constraint must be linear.

Convex functions. A function f : Rn 7→ R is convex when the following

is satised:
f (αx1 + βx2 ) ≤ αf (x1 ) + βf (x2 ), (11)
for all x1 , x2 ∈ Rn and all α, β ∈ R with α + β = 1, α ≥ 0, β ≥ 0. This
means that a line joining two points x1 and x2 on f will lie on or above f
for all choices of x1 and x2 .

7
 (a) A non-convex function. The line (b) A convex function. No matter
joining(x1 , f (x1 )) and (x2 , f (x2 )) lies where we choose to place x1 , x2 and
below the graph of f , violating the x3 , the lines joining the points lie
convexity condition (11). This is in- above f .
dicated in red. The green line joining
(x2 , f (x2 )) and (x3 , f (x3 )) lies above
f , meaning f is convex in this inter-
val.

Figure 2: Convex and non-convex functions.

Linear functions are convex, because all points on a line joining two
points on the function would coincide with the function itself. Functions with
positive curvature in the entire function domain (positive denite Hessian)
are convex.

Convex sets.

Convex hulls.

Convex relaxations.

1.4.3 The B-spline

1.4.4 Global optimization and the Branch-and-Bound algorithm
1.5 Availability
CENSO is not publically available as of today.
1.6 Prerequisites
CENSO relies on a few open-source libraries to function. The most impor-
tant of these are listed below.

8
1.6.1 Ipopt
Ipopt is the default solver used in CENSO for solving optimization problems
locally. Ipopt uses an interior point algorithm to nd local solutions of (1).
For more information on Ipopt and installation instructions, the reader is
referred to the Ipopt home page. Details about the algorithm itself can be
found in [6].

1.6.2 Eigen
Eigen is a C++ template library for linear algebra. It includes functionality
for Matlab-like matrix and vector operations, solvers and other related algo-
rithms such as matrix decompositions. For more information on Eigen and
installation instructions, the reader is referred to the Eigen web site [4].

9
2 Installation
2.1 Getting system packages
2.2 Getting the code
2.3 External code
2.4 Compiling

10
3 Using CENSO through C++/Qt
CENSO is written in C++ 11, using the Qt IDE (Qt Creator), which is a
powerful and widely used IDE. Although the Qt framework oers a vast suite
of functions, these are not used in CENSO for the sake of easy portability.

3.1 Code structure

Figure 3: Folder structure in the
CENSO code, as shown in Qt
Creator:
BranchAndBound: Contains
the code and classes required
for the Branch-and-Bound algo-
rithm.
Interpolation: Contains inter-
polation functionality. This in-
cludes the InterpolationTable
class, the B-spline functionality,
and linear interpolation function-
ality.
OptimizationProblem: Con-
tains the classes required to for-
mulate the optimization problem,
that is, the classes which describe
constraints and objective func-
tions.
SolverInterface: Contains the
code which translates the op-
timization problem desribed by
the constraint and objective func-
tion classes into the format re-
quired by the dierent solvers
used (Ipopt, Bonmin, Nomad).
TestProblems: Contains a
framework for testing and some
testing problems (including the
examples in chapter 6).

In Qt Creator, the .cpp and header les are automatically placed in

separate folders project tree view, even though they are stored in the same
actual folder. When working on a le in the editor, pressing the F4 key
conveniently switches between the .cpp and header le.

11
 In addition to the folders described above, some additional les deserve
some explanation:

• censo.pro is the CENSO project le, and the project is opened in Qt

Creator by opening this le. What it actually contains is the informa-
tion required by qmake to build the application.

• The folder censo_libs contains the censo_libs.pri le which con-

tains library include paths to the various libraries used by CENSO. It
also species some qmake conguration and compile ags.

• generaldefinitions.h and generaldefinitions.cpp contain some

general functionality like typedefs, enums, printing functions and trans-
formations, which is reused throughout the code.

• The main.cpp le contains the main() function.

12
4 Formulation of an optimization problem
This section will introduce the C++ classes of the optimization framework.
An optimization problem is made up of one objective object, one constraint
object, three double vectors (variable lower bound, upper bound and solver
starting point). Branch-and-Bound solvers will in addition to these need two
int vectors, one for variable types (continuous, binary or integer) and one for
the indices of variables that can be divided in the branching procedure.

4.1 Type denitions

The framework makes use of type denitions for the most commonly used
data structures and types.
// Eigen vectors
typedef Eigen :: VectorXd DenseVector ;
typedef Eigen :: SparseVector < double > SparseVector ;

// Eigen matrices
typedef Eigen :: MatrixXd DenseMatrix ;
typedef Eigen :: SparseMatrix < double > SparseMatrix ; // declares a
column - major sparse matrix type of double

4.2 Smart pointers

Smart pointers are used for all classes dened in the framework. The smart
pointer implementation used is the stl shared_pointer via typedefs that
append Ptr to the class name, i.e.
typedef std :: shared_ptr < ClassName > ClassNamePtr ;.

4.3 Variable bounds

Variable bounds are represented as stl vectors of type double. A constant
INF is dened to represent positive innity and is used in the case on an
unbounded variable.
As an example take a problem with the three variables x0 , x1 , x2 and the
variable bounds

0 ≤ x0 ≤ 1
0 ≤ x1 ≤ 3
−∞ ≤ x2 ≤ ∞

The declaration of these would be

std :: vector < double > lb {0 , 0 , - INF };
std :: vector < double > ub {1 , 2 , INF };

13
4.4 Variable types
Variable types are represented as stl vectors of type int. The types are de-
clared as an enum. The available types are BINARY, INTEGER and CONTINUOUS.
The code snippet below shows how to create a vector of variable types.
std :: vector < int > variable_types { BINARY , INTEGER , CONTINUOUS
};

4.5 Starting point

Solvers require a starting point for the optimization. This is provided as a
double vector.
std :: vector < double > z0 {0 , 0 , 0};.

4.6 Branching variables

Branch and bound solvers must be supplied with a list of indices that indicate
which variables that should be used in the branching procedure. This is to
avoid unnecessary branching on variables that do not take part in nonconvex
terms. The indexes are given as an int vector. As an example; to allow
branching on x0 , x1 and x5 the code would be
std :: vector < int > branching_variables {0 , 1 , 5};.

4.7 The objective function

The objective is given by the Objective class. Objective is abstract and
denes the interface that solvers can use. The most important functions of
the interface are explained below.
void eval ( DenseVector & x , DenseVector & y )

evaluates the objective at x and stores the value in y.

void evalGradient ( DenseVector & x , DenseVector & dx )

evaluates the objective gradient at x and stores the value in dx.

void evalHessian ( DenseVector & x , DenseVector & ddx )

evaluates the objective Hessian at x and stores the value in ddx. The Hessian
is stored in the format indicated by
void structureHessian ( std :: vector < int >& iRow , std :: vector <
int >& jCol )

where iRow and jCol indicates the positions in the Hessian matrix.
void augmentDomain ( int dim )

14
will increase the dimension of the domain without altering the objective
function itself (this is only to allow solver to introduce additional variables
for instance when creating convex relaxations).
As an example; to create the linear objective function f (x) = cT x with
c = [0, 0, 1] we must rst create the vector cT . It is represented by a Eigen
T

matrix object v = cT . We can create this by writing

int numVars = 3;
DenseMatrix v (1 , numVars ) ;
v << 0 , 0 , 1;

An Objective pointer is then made by writing

ObjectivePtr obj ( new ObjectiveLinear ( v )) ;

Currently only linear objectives are avaiable. The global branch and
bound solver assumes a convex objective.

4.8 The constraints

Constraints are dened using the Constraint class. Most constraint class
implementations represents a type of constraint function, i.e. linear equa-
tions, polynomials or B-splines. The exception being the composite con-
straint. The composite constraint holds a collection of other constraint ob-
jects and represents them as if they were one unied object.
The most important functions in the constraint interface are described
below.
void eval ( DenseVector & x , DenseVector & y )

evaluates the constraint at x and stores the value in y.

void evalJacobian ( DenseVector & x , DenseVector & dx )

evaluates the constraint Jacobian at x and stores the value in dx using the
structure given by
void structureJacobian ( std :: vector < int >& iRow , std :: vector <
int >& jCol ) .

void evalHessian ( DenseVector & x , DenseVector & ddx )

evaluates the constraint Hessian at x and stores the value in ddx using the
structure given by
void structureHessian ( std :: vector < int >& eqnr , std :: vector <
int >& iRow , std :: vector < int >& jCol )

void setDomainBounds ( std :: vector < double > lb , std :: vector <
double > ub )

chages the domain bounds. The new bounds must be a subset of the current
constraint domain.

15
 The constraint composite is a special constraint object that contains a
list of other constraint objects. It is based on the composite pattern. The
composite has a add function that accepts a constraint object along with
a vector of indexes that indicates which variables that are related to the
constraint.
the following code snippet illustrates how to make a cubic B-spline con-
straint from a data table.
InterpolationTable data = ...
int splineDegree = 3;
int equality = true ;
ConstraintPtr cbspline ( new ConstraintBspline ( data , 3 , equality )
);

If we wish to use more than one constraint they must be collected in a

composite object. The composite is created by rst passing the number of
variables along with the variable bounds. The constraint objects are then
added in turn.
int numvars = ...;
ConstraintCompositePtr constraints ( new ConstraintComposite (
numvars , lb , ub ) ) ;

// add the bspline constraint using variables x1 , x3 and x5

std :: vector < int > idx {1 , 3 , 5};
constraints - > add ( cbspline , idx ) ;

// add more constraints ...

std :: vector < int > idx2 { i1 , i2 , ... , in };
ConstraintPrt c2 = new Constraint ....
constraints - > add ( c2 , idx2 ) ;

4.9 The Solver

The solvers are called by creating an Optimizer object, passing the objec-
tive, constraint, bounds, starting point and, if required, the variable types
and branching variables to the Optimizer constructor. The problem is then
solved by calling optimize() and the solution can be extracted using get
functions. Available solvers are Ipopt, Bonmin and a homemade global
branch and bound solver.
The optimization problem is solved by calling optimize(). The return
value from optimize indicates whether the solver was successful or not. Pos-
sible return values are 1 (success) and 0 (unsuccessful).
// Local solution using Ipopt ( ignoring integer restrictions )
OptimizerIpopt ip ( objective , constraints , z0 ) ;
int returnStatus = ip . optimize () ;

// Local , integer feasible solution using Bonmin

OptimizerBonmin ip ( objective , constraints ,

16
 z0 , variable_types , branching_variables ) ;
int returnStatus = ip . optimize () ;

// Global solution using branch and bound

BranchAndBound bnb ( objective , constraints ,
z0 , variable_types , branching_variables ) ;
int returnStatus = bnb . optimize () ;

17
5 Solvers
5.1 Ipopt
Ipopt solves convex NLPs and LPs. Refer to the Ipopt documentation.

5.2 CENSO
CENSO solves convex MINLPs and non-convex MINLPs with convex relax-
ations available.

5.2.1 Basic algorithm parameters

This section describes the parameters/settings of the CENSO Branch-and-
Bound algorithm. The corresponding variables that must be adjusted for
the various parameters are found in table 1. These variables are all members
in the BranchAndBound class. If adjustments are to be made, they must be
made in the instance of BranchAndBound used for optimization.

• Convergence limit ε. The value of ε is the termination criteria

for the Branch-and-Bound algorithm. When the optimality gap (the
dierence between the upper and lower bounds) becomes less than or
equal to ε, the Branch-and-Bound algorithm terminates.

• Maximum number of iterations. This parameter sets the maxi-

mum number of iterations before the algorithm terminates (regardless
of whether the optimal point has been found or not).

• Maximum number of infeasible iterations. This parameter sets

the maximum number of infeasible iterations before the algorithm ter-
minates. A high number of infeasible iterations can indicate that the
algorithm has ventured outside the domain in which the constraints
are well dened, and is struggling to nd feasible solutions.

• Maximum node tree depth. This parameter sets the maximum

depth in the Branch-and-Bound node tree. It the node tree gets very
deep, this can indicate that the convex relaxations do not get tight,
resulting in indenite branching.

Parameter Data type Variable name

Convergence limit ε double epsilon
Max. # iterations int maxIterations
Max. # infeasible iterations int maxInfeasibleIterations
Max. node tree depth int maxDepth

Table 1: Algorithm parameters and corresponding variable names

18
5.2.2 Branch-and-Bound specic parameters
This section describes a few more specialized parameters which can be used
to adjust the behaviour of the Branch-and-Bound algorithm. A summary of
these parameters and their corresponding variables is found in table 7. These
variables are all members in the BranchAndBound class. If adjustments are
to be made, they must be made in the instance of BranchAndBound used for
optimization.

• Node selection strategy. This parameter decides which node in the

node tree is selected as the next node for processing. The available
options are shown in table 2.

Variable name: nodeSelectionStrategy

BEST_FIRST Selects the node with the best (greatest)
lower bound.
DEPTH_FIRST Selects the nodes based on a standard
depth-rst search (DFS).
BREADTH_FIRST Selects the nodes based on a standard
breadth-rst search (BFS).
DEPTH_FIRST_BREADTH_AFTER Selects nodes based on depth-rst search
until a feasible solution is found, then
switches to breadth-rst search.
BREADTH_FIRST_DEPTH_AFTER Selects nodes based on breadth-rst
search until a feasible solution is found,
then switches to depth-rst search.
RANDOM_NODE Selects a random node from the tree.

Table 2: Node selection strategy options

• Node processing policy. This parameter decides the policy for node
processing. The available options are shown in table 3.

Variable name: nodeProcessingPolicy

EAGER When a node is selected, its child nodes are processed - nodes
are selected after they have been processed. The benet of this
approach is that it causes the algorithm to select better nodes
for processing, but it may process more nodes than necessary.
LAZY When a node is selected, the node itself is processed - nodes
are selected before they have been processed. Selecting nodes
before they are processed may cause the algorithm to select
poor nodes for processing, but it does not process more nodes
than necessary.

Table 3: Node selection strategy options

19
 • Branching rules for integer variables. This parameter decides
the rules for branching on integer varianles. The available options are
shown in table 4.

Variable name: branchingRuleInteger

MOST_FRACTIONAL Branches on the most fractional integer variable,
that is, the variable furthest from an integer value,
that is, the variable with fractional part closest to
0.5.
PSEUDOCOST This rule keeps track of how successful the algorithm
has been in branching on each variable which has
been branched on so far, and selects the variable
with the best record in increasing the lower bound.
This strategy has not yet been implemented.
STRONG_BRANCHING Strong branching tests which of the candidates will
give the best improvement before branching. This
strategy has not yet been implemented.
Table 4: Options for branching rules for integer variables

• Branching rules for continuous variables. This parameter decides

the rules for branching on continuous varianles. The available options
are shown in table 5.

Variable name: branchingRuleContinuous

MOST_PROMISING This rule keeps track of how successful the algorithm
has been in branching on each variable which has
been branched on so far, and selects the variable
with the best record in increasing the lower bound.
LONGEST_INTERVAL This rule selects the variable with the longest bound
interval, that is, the dierence between the upper
and lower bound for the variable.
RANDOM_VARIABLE This rule selects a random variable for branching.

Table 5: Options for branching rules for continuous variables

• Branching priority. This parameter decides how the dierent vari-

able types are prioritized for branching. The available options are
shown in table 6.

20
 Variable name: branchingRuleContinuous
BINARY_INTEGER_CONTINUOUS Starts with branching on all binary variables. When
nished with the binary variables, the algorithm
branches on the integer variables. Finally, contin-
uous variables are branched on.
RANDOM_MIX Selects a random mix of binary, integer and contin-
uous variables for branching.

Table 6: Options for branching priority

A summary of the parameters described above is found in the table

below:

Parameter Variable name Possible values

Node sel. nodeSelectionStrategy BEST_FIRST
strategy DEPTH_FIRST
BREADTH_FIRST
DEPTH_FIRST_BREADTH_AFTER
BREADTH_FIRST_DEPTH_AFTER
RANDON_NODE
Node proc. nodeProcessingPolicy EAGER
policy LAZY
Branching branchingRuleInteger MOST_FRACTIONAL
rule for PSEUDOCOST
int. vars. STRONG_BRANCHING
Branching branchingRuleContinuous MOST_PROMISING
rule for LONGEST_INTERVAL
cont. vars. RANDOM_VARIABLE
Branching branchingPriority BINARY_INTEGER_CONTINUOUS
priorities RANDOM_MIX

Table 7: Summary of Branch-and-Bound specic parameters, their corre-

sponding variable names and possible values.

5.3 Bonmin
Bonmin solves convex MINLPs. Refer to the Bonmin documentation.

21
6 Example problems
6.1 Testing framework and the TestProblem class
The TestProblem class is a framework for creating test problems. The code
that implements the test problem itself is in the solveProblem() function
of each class. The test problems described in the following section are sum-
marized in table 8. Each of these classes inherit from TestProblem.

Section Problem Class name

6.2.1 Farming LP case FarmingLP
6.2.2 Farming QP case FarmingQP
6.2.3 Farming QP case with integer FarmingInteger
variables
6.3.1 Network maximum ow case MaximumFlow
6.3.2 Network maximum ow with FlowWithRouting
routing
6.4.3 Six-Hump Camelback func- SixHumpCamelPolynomial
tion: Local optimization with
the ConstraintPolynomial
constraint class
6.4.4 Six-Hump Camelback func- SixHumpCamelCustom
tion: Local optimization with
a custom constraint class
6.4.5 Six-Hump Camelback func- SixHumpCamelABnB
tion: Global optimization
using an alpha-Branch-and-
Bound approach with custom
constraint and relaxed con-
straint classes
6.4.6 Six-Hump Camelback func- SixHumpCamelBSpline
tion: Global optimization us-
ing B-Spline approximation

Table 8: Test problems and their class names

22
6.2 Farming problem
In this section we will look at a simple optimization problem taken from the
lecture notes of the NTNU course TTK4135 Optimization and Control. We
will look at three cases:

1. Linear programming case

2. Quadratic programming case

3. Introduction of integer variables

6.2.1 Linear programming case

We will start with a simple LP case. A farmer wants to grow apples and
bananas. He has a eld of size 100000 m2 . Growing 1 tonne of apples
requires an area of 4000 m2 and 60 kg of fertilizer. Growing 1 tonne of
bananas requires an area of 3000 m2 and 80 kg of fertilizer. The prot for
apples is 7000 per tonne (including fertilizer cost), and the prot for bananas
is 6000 per tonne (including fertilizer cost). The farmer can legally use up
to 2000 kg of fertilizer. He wants to maximize his prots.

Optimization variables. We will introduce two optimization variables.

x0 is the number of tonnes of apples grown, and x1 is the number of tonnes
of bananas grown. We dene a vector of optimization variables x = [x0 , x1 ]> .

Objective function. The farmer wants to maximize his prots, that is,
he wants to
maximize 7000x0 + 6000x1 .
We want a minimization problem, so we negate the objective function and
write it on standard LP form:
 
  x0
minimize −7000 −6000 .
| {z } x1
v>
| {z }
x

Constraints. In this problem, we have two constraints. The farmer cannot

grow more than he has room for in the eld:

4000x0 + 3000x ≤ 100000 .

| {z } | {z }1 | {z }
Area used for apples Area used for bananas Available area

He cannot exceed the legal amount of fertilizer used:

60x0 + 80x ≤ 2000 .

| {z } | {z }1 |{z}
Fertilizer used for apples Fertilizer used for bananas Legal amount of fertilizer

23
We want a constraint on the form Ax ≤ b, so we write
    
4000 3000 x0 100000
≤
60 80 x1 2000
| {z } | {z } | {z }
A x b

Variable bounds. We also have nonnegativity constraints on the opti-

mization variables (he cannot grow negative amounts). These are included
in the variable bounds:
0 ≤ x ≤ ∞.

Optimization problem. We now have the optimization problem

minimize v > x (12a)
subject to Ax ≤ b (12b)
0≤x≤∞ (12c)

The problem is illustrated in g. 4.

25 Feasible area

20
x1 (Bananas [tonnes])

*
15 x =(14.2857,14.2857)

10

z0
0
0 5 10 15 20 25
x0 (Apples [tonnes])

Figure 4: Graphical representation of the farming LP. The optimal point at

x∗ = (14.2857, 14.2857) and starting point z0 = (0, 0) are marked with blue
circles.

24
 Setting up and solving the problem. We set up a new TestProblem
class called FarmingLP. The code below is implemented in the solveProblem()
function to set up and solve the farming LP.
1 void FarmingLP :: solveProblem ()
2 {
3 // Variable bounds
4 std :: vector < double > lb {0 , 0};
5 std :: vector < double > ub { INF , INF };
6 // Starting point
7 std :: vector < double > z0 {0 , 0};
8 // Objective function
9 DenseMatrix v (1 , 2) ;
10 v << -7000 , -6000;
11 ObjectivePtr objective ( new ObjectiveLinear ( v ) ) ;
12 // Constraints
13 DenseMatrix A (2 , 2) ;
14 A << 4000 , 3000 ,
15 60 , 80;
16 DenseVector b (2) ;
17 b << 100000 ,
18 2000;
19 ConstraintPtr cLinear ( new ConstraintLinear (A , b , false ) ) ;
20 // Constraint composite
21 ConstraintCompositePtr constraints ( new ConstraintComposite
(2 , lb , ub ) ) ;
22 constraints - > add ( cLinear );
23 // Optimize
24 OptimizerIpopt optIpopt ( objective , constraints , z0 ) ;
25 int status = optIpopt . optimize () ;
26 fopt_found = optIpopt . getObjectiveValue () ;
27 zopt_found = optIpopt . getOptimalSolution () ;
28 cout << " Optimal solution : f *= " << fopt_found << endl ;
29 cout << " Optimal point : x *=( " << zopt_found . at (0) << " ," <<
zopt_found . at (1) << " ) " << endl ;
30 }

Lines 4-7 declare and ll STL vectors with the lower variable bounds, upper
variable bounds and starting point. Lines 8-11 dene the objective function.
Lines 12-19 dene the linear constraints. Lines 20-22 create a constraint
composite and inserts the linear constraint. Lines 23-25 creates an Ipopt
optimizer object and solves the problem. Lines 26-29 extracts and prints
information about the solution. When used in the TestProblem framework,
this gives us the printout
Running Farming LP problem...

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit http://projects.coin-or.org/Ipopt
******************************************************************************

25
Optimal solution: f*=-185714
Optimal point: x*=(14.2857,14.2857)
Farming LP problem successfully solved in 0 (ms)
Press <RETURN> to close this window...
We see that the correct optimal solution is found, as expected. The maxi-
mum prot is 185714.

6.2.2 Quadratic programming case

Now consider a case where the price of apples and bananas depends on how
much is produced. Now the prot for apples is 7000 − 200x0 per tonne
(including fertilizer cost) and the prot for bananas is 4000 − 140x1 per
tonne (including fertilizer cost). All the other parameters of the problem
(eld size, fertilizer limit etc.) are the same as for the LP case. We want a
linear objective function, so we introduce a new variable x2 to represent the
prot:

x2 = (7000 − 200x0 )x0 + (4000 − 140x1 )x1

    
  −200 0 x0   x0
= x0 x1 + 7000 4000 .
0 −140 x1 x1

Optimization variables. We redene our vector of optimization variables

to be x = [x0 , x1 , x2 ]> . Here, x0 and x1 represent the same as before, and
the new variable x2 is the prot.

Objective function. Since x2 represents prot and we want to maximize

this, our new objective is
 
  x0
minimize 0 0 −1 x1  .
| {z } x
2
v> | {z }
x

Constraints. We keep the linear constraint from the LP case:

 
x
A 0 ≤ b.
x1

We also have to create a constraint to dene the relationship between the

three variables. We have
     
  −200 0 x0   x0
x2 − x0 x1 + 7000 4000 = 0,
0 −140 x1 x1

26
which we can write as
 
200 0 0
0 ≤ x>  0 140 0 x + −7000 −4000 1 x ≤ 0,
 

0 0 0 | {z }
| {z } q>
P

that is, we can use the ConstraintQuadratic class to dene this constraint.

Variable bounds. Both x0 and x1 have to be nonnegative as before, but

x2 can in principle also be negative (meaning the farmer is losing money).
Therefore, the bounds are

0 ≤ x0 ≤ ∞
0 ≤ x1 ≤ ∞
−∞ ≤ x2 ≤ ∞

Optimization problem. We now have the optimization problem

minimize v > x (13a)
 
x
subject to A 0 ≤ b, (13b)
x1
0 ≤ x> P x + q > x ≤ 0, (13c)
 
x
0 ≤ 0 ≤ ∞, (13d)
x1
−∞ ≤ x2 ≤ ∞. (13e)

The problem is illustrated in g. 5.

27
 25 Feasible area

20
x1 (Bananas [tonnes])

15

x*=(15.7178,12.3762,88676)

10

z
0
0
0 5 10 15 20 25
x0 (Apples [tonnes])

Figure 5: Graphical representation of the farming QP. The optimal point at

x∗ = (15.7178, 12.3762, 88676) and starting point z0 = (0, 0, 0) are marked
with blue circles.

Setting up and solving the problem. We set up a new TestProblem

class called FarmingQP. The code below is implemented in the solveProblem()
function to set up and solve the farming QP.
1 void FarmingQP :: solveProblem ()
2 {
3 // Variable bounds
4 std :: vector < double > lb {0 , 0 , - INF };
5 std :: vector < double > ub { INF , INF , INF };
6 // Starting point
7 std :: vector < double > z0 {0 , 0 , 0};
8 // Objective function
9 DenseMatrix v (1 , 3) ;
10 v << 0 , 0 , -1;
11 ObjectivePtr objective ( new ObjectiveLinear ( v ) ) ;
12 // Linear constraint
13 DenseMatrix A (2 , 2) ;
14 A << 4000 , 3000 ,
15 60 , 80;
16 DenseVector b (2) ;
17 b << 100000 ,
18 2000;
19 ConstraintPtr cLinear ( new ConstraintLinear (A , b , false ) ) ;

28
20 // Linear constraint variable mapping
21 std :: vector < int > varMapLinear {0 , 1};
22 // Quadratic constraint
23 DenseMatrix P (3 , 3) ;
24 P << 200 , 0, 0,
25 0 , 140 , 0 ,
26 0, 0 , 0;
27 DenseMatrix q (3 , 1) ;
28 q << -7000 ,
29 -4000 ,
30 1;
31 ConstraintPtr cQuadratic ( new ConstraintQuadratic (P , q , 0 ,
0 , 0) ) ;
32 // Constraint composite
33 ConstraintCompositePtr constraints ( new ConstraintComposite
(3 , lb , ub ) ) ;
34 constraints - > add ( cLinear , varMapLinear ) ;
35 constraints - > add ( cQuadratic ) ;
36 // Optimize
37 OptimizerIpopt optIpopt ( objective , constraints , z0 ) ;
38 int status = optIpopt . optimize () ;
39 fopt_found = optIpopt . getObjectiveValue () ;
40 zopt_found = optIpopt . getOptimalSolution () ;
41 cout << " Optimal solution : f *= " << fopt_found << endl ;
42 cout << " Optimal point : x *=( " << zopt_found . at (0) << " ," <<
zopt_found . at (1) << " ," << zopt_found . at (2) << " ) " <<
endl ;
43 }

Lines 3-7 declare and ll STL vectors with the lower variable bounds, upper
variable bounds and starting point. Lines 8-11 dene the objective func-
tion. Lines 12-19 dene the linear constraint. Line 21 denes an STL vector
of integers which denes the mapping between the variables in the linear
constraint (which has two variables) and the variables in the constraint com-
posite (which has three variables). Lines 22-31 dene the quadratic con-
straint. Lines 32-25 create a constraint composite and inserts the linear and
quadratic constraints. Lines 36-38 creates an Ipopt optimizer object and
solves the problem. Lines 39-42 extracts and prints information about the
solution. When used in the TestProblem framework, this gives us the print-
out
Running Farming QP problem...

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit http://projects.coin-or.org/Ipopt
******************************************************************************

Optimal solution: f*=-88675.7

29
 Optimal point: x*=(15.7178,12.3762,88675.7)
Farming QP problem successfully solved in 10 (ms)
Press <RETURN> to close this window...
We see that the correct optimal solution is found, as expected. The maxi-
mum prot is 88675.7.

6.2.3 Integer variables

We now say that the farmer is only allowed to produce integer amounts of
apples and bananas. That is, we impose the additional constraint that x0
and x1 are integers. Since the objective function is linear, and the orig-
inal constraints are convex, this becomes a convex MINLP. This is easily
formulated and solved in a Branch-and-Bound framework.

25 Feasible points

20
x1 (Bananas [tonnes])

15
x*=(16,12,88640)

10

z0
0
0 5 10 15 20 25
x (Apples [tonnes])
0

Figure 6: Graphical representation of the farming problem with integer

variables. The optimal point at x∗ = (16, 12, 88640) and starting point
z0 = (0, 0, 0) are marked with blue circles.

Setting up and solving the problem. We set up a new TestProblem

class called FarmingInteger. The code below is implemented in the solveProblem()
function to set up and solve the farming problem with integer variables.
1 void FarmingInteger :: solveProblem ()
2 {
3 // Variable bounds

30
 4 std :: vector < double > lb {0 , 0 , - INF };
5 std :: vector < double > ub {30 , 30 , INF };
6 // Starting point
7 std :: vector < double > z0 {0 , 0 , 0};
8 // Objective function
9 DenseMatrix v (1 , 3) ;
10 v << 0, 0 , -1;
11 ObjectivePtr objective ( new ObjectiveLinear ( v ) ) ;
12 // Linear constraint
13 DenseMatrix A (2 , 2) ;
14 A << 4000 , 3000 ,
15 60 , 80;
16 DenseVector b (2) ;
17 b << 100000 ,
18 2000;
19 ConstraintPtr cLinear ( new ConstraintLinear (A , b , false ) ) ;
20 // Linear constraint variable mapping
21 std :: vector < int > varMapLinear ;
22 varMapLinear . push_back (0) ;
23 varMapLinear . push_back (1) ;
24 // Quadratic constraint
25 DenseMatrix P (3 , 3) ;
26 P << 200 , 0, 0,
27 0 , 140 , 0 ,
28 0, 0 , 0;
29 DenseMatrix q (3 , 1) ;
30 q << -7000 ,
31 -4000 ,
32 1;
33 ConstraintPtr cQuadratic ( new ConstraintQuadratic (P , q , 0 ,
0 , 0) ) ;
34 // Constraint composite
35 ConstraintCompositePtr constraints ( new ConstraintComposite
(3 , lb , ub ) ) ;
36 constraints - > add ( cLinear , varMapLinear ) ;
37 constraints - > add ( cQuadratic ) ;
38 // Variable types
39 std :: vector < int > variable_types ;
40 variable_types . push_back ( INTEGER ) ; // x0
41 variable_types . push_back ( INTEGER ) ; // x1
42 variable_types . push_back ( CONTINUOUS ) ; // x2
43 // Branching variables
44 std :: vector < int > branching_variables ;
45 branching_variables . push_back (0) ; // x0
46 branching_variables . push_back (1) ; // x1
47 // Optimize
48 BranchAndBound bnb ( objective , constraints , z0 ,
variable_types , branching_variables ) ;
49 int status = bnb . optimize () ;
50 fopt_found = bnb . getObjectiveValue () ;
51 zopt_found = bnb . getOptimalSolution () ;
52 }

This is very similar to the QP case. The dierences are:

31
 • In line 5, the upper bounds of x0 and x1 are set to 30 instead of ∞.
This is because we are branching on these variables, so we must dene
an upper bound so that the Branch-and-Bound algorithm is able to
calculate where to split the variable when branching. 30 is a reasonable
choice as an upper bound, because there is not room to grow 30 tons
of either fruit.

• Lines 38-42 dene the variable types. Here, we have dened that x0
and x1 are integer variables, and x2 (the prot function) is continuous.

• Lines 43-46 dene which variables are to be branched on. Typically,

we choose our degrees of freedom as branching variables; in our case,
the amount of apples x0 and the amount of bananas x1 .

• In lines 47-49, we use the BranchAndBound class instead of the OptimizerIpopt

class. This is because we have integer variables and need to use branch-
ing to solve the problem.

When used in the TestProblem framework, this gives us the following out-
put:
Branch and Bound tree search finished after 19 iterations, using 0 sec.
Global solution upper bound = -88640
Global solution lower bound = -88640
Optimality gap = 0 <= 0.001 (epsilon)
Optimal point x* = ( 16, 12, 8.864e+04)

Farming QP problem successfully solved in 180 (ms)

Press <RETURN> to close this window...
We see that the correct optimal solution is found (quickly). The maximum
prot is 88640.

32
6.3 Network ow problem
Now we will look at some cases of network ow. Throughout this section we
will look at a simple ow network with one source vertex, one sink vertex,
four internal vertices and ten edges. The ow network is illustrated in g. 7.
We denote the vertices vi , i = {0 . . . 5} and the edges ej , j = {0 . . . 9}. For
context, we could say the ow network represents a routing network where
the edges represent pipelines, and the vertices represent connection points
between the pipelines.

Figure 7: Flow network. v0 is the source node and v5 is the sink node.

6.3.1 Simple maximum ow case

Consider a case where our objective is to maximize the ow though the
network, and each edge has a constant maximum capacity (see g. 8).

Figure 8: Flow network with constant capacities.

The notation we will use is the following:

• The ow through edge ej is xj . Since the ows are our degrees of
freedom, these are our optimization variables. We dene our vector of
optimization variables x = [x0 , . . . , x9 ]> .
• The maximum ow, or capacity, for edge ej is xj . This gives us a
vector of capacities x = [13, 7, 10, 5, 2, 3, 6, 3, 9, 12]> . We also dene
x = −x, meaning the minimum ow is the negated maximum ow.

33
Incidence matrix. To formulate the optimization problem, we will make
use of an incidence matrix A = {ai,j }. The rows of the incidence matrix
represent vertices, and the columns represent edges. If edge j leaves vertex
i, then ai,j = −1. If edge j enters vertex i, then ai,j = 1. The incidence
matrix for our ow network is shown below.
e0 e1 e2 e3 e4 e5 e6 e7 e8 e9
−1 −1 0
 
v0 0 0 0 0 0 0 0
v1 
 1 0 −1 −1 −1 0 0 0 0 0 

v2 
 0 1 1 0 0 −1 −1 0 0 0  =A

v3 
 0 0 0 1 0 1 0 −1 −1 0 

v4  0 0 0 0 1 0 1 1 0 −1 
v5 0 0 0 0 0 0 0 0 1 1

Note that this incidence matrix could be slightly dierent since our ow net-
work is undirected (liquid can ow both ways through a pipeline). However,
here we have assumed that the default ow direction is right → left and up
→ down.

Optimization variables. Our vector of optimization variables is x =

[x0 , . . . , x9 ]> , where xj is the ow through edge ej .

Objective function. We want to maximize the ow through the network.

Another way of saying this is that we want to maximize the ow into the
sink vertex v5 . This ow is the sum of the ows in edges e8 and e9 , in other
words, we want to maximize x8 + x9 . This gives the linear objective

x0
 
x1 
 
x2 
 
x3 
 
  x4 
minimize 0 0 0 0 0 0 0 0 −1 −1  .
}x5 
| {z 
x6 
v >  
x 
 7
x 
8
x9
| {z }
x

Note that we could just as well have maximized the ow out of the source
vertex, or the ow across any cut in the network.

Constraints. We have two constraints; the mass balance constraint and

the ow capacity constraint. We will implement the ow capacity constraint
later when we dene the variable bounds. The mass balance constraint says

34
 that for each vertex (except the source and the sink), the sum of inows
must equal the sum of outows:

For vertex v1 : x0 − x2 − x3 − x4 = 0,
For vertex v2 : x1 + x2 − x5 − x6 = 0,
For vertex v3 : x3 + x5 − x7 − x8 = 0,
For vertex v4 : x4 + x6 + x7 − x9 = 0.

To write this compactly, we dene the matrix Aint , which is the incidence
matrix A with the top and bottom rows removed (so that it represents only
the internal nodes):
 
1 0 −1 −1 −1 0 0 0 0 0
0 1 1 0 0 −1 −1 0 0 0
Aint =  .
0 0 0 1 0 1 0 −1 −1 0 
0 0 0 0 1 0 1 1 0 −1

Now we can state the mass balance constraint as a linear system of equations
Aint x = 0. The source and sink vertices are assumed to have innite capacity,
so these are not represented in any constraints.

Variable bounds. Since the optimization variables are the ows through
the edges, we can represent the ow capacity constraints as bounds on the
optimization variables:
x ≤ x ≤ x.

Optimization problem. We now have the optimization problem

minimize v > x (14a)
subject to Aint x = 0, (Mass balance) (14b)
x ≤ x ≤ x. (Flow capacity) (14c)

Setting up and solving the problem. We set up a new TestProblem

class called MaximumFlow. The code below is implemented in the solveProblem()
function to set up and solve the maximum ow problem.
1 void MaximumFlow :: solveProblem ()
2 {
3 // Starting point : zero flow
4 std :: vector < double > z0 {0 , 0 , 0 , 0 , 0 , 0 , 0 , 0, 0 , 0};
5 // Flow capacities ( c )
6 std :: vector < double > lb
7 { -13 , -7 , -10 , -5 , -3 , -2 , -6 , -3 , -9 , -12};
8 std :: vector < double > ub
9 { 13 , 7 , 10 , 5 , 3 , 2 , 6 , 3 , 9 , 12};
10 // Incidence matrix
11 DenseMatrix A (6 , 10) ;

35
12 A << -1 , -1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ,
13 1 , 0 , -1 , -1 , -1 , 0 , 0 , 0 , 0 , 0 ,
14 0 , 1 , 1 , 0 , 0 , -1 , -1 , 0 , 0 , 0 ,
15 0 , 0 , 0 , 1 , 0 , 1 , 0 , -1 , -1 , 0 ,
16 0 , 0 , 0 , 0 , 1 , 0 , 1 , 1 , 0 , -1 ,
17 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 1 , 1;
18 // Incidence matrix for internal nodes
19 DenseMatrix A_int = A . block (1 ,0 ,4 ,10) ;
20 // Objective function
21 DenseMatrix v (1 , 10) ;
22 v << 0, 0 , 0 , 0 , 0 , 0 , 0 , 0 , -1 , -1;
23 ObjectivePtr objective ( new ObjectiveLinear ( v ) ) ;
24 // Mass balance constraint
25 DenseVector zeros ; zeros . setZero (4 ,1) ;
26 ConstraintPtr cMassBalance ( new ConstraintLinear ( A_int ,
zeros , true ) ) ;
27 // Constraint composite ( with flow capacity as bounds )
28 ConstraintCompositePtr constraints ( new ConstraintComposite
(10 , lb , ub ) ) ;
29 constraints - > add ( cMassBalance ) ;
30 // Optimize
31 OptimizerIpopt optIpopt ( objective , constraints , z0 ) ;
32 int status = optIpopt . optimize () ;
33 fopt_found = optIpopt . getObjectiveValue () ;
34 zopt_found = optIpopt . getOptimalSolution () ;
35 cout << " Optimal solution : f *= " << fopt_found << endl ;
36 cout << " Optimal flows : " << endl ;
37 for ( int i = 0; i < 10; i ++) cout << " Edge " << i << " : "
<< zopt_found . at ( i ) << endl ;
38 }

Line 4 denes the starting point, which we have chosen to be zero ow. Lines
6-9 dene the variable bounds, which are also the edge capacities. Lines 10-
19 dene the incidence matrix and the internal node incidence matrix. Lines
20-23 dene the objective function. Lines 24-26 dene the mass balance con-
straint, and lines 27-29 dene the constraint composite and adds the mass
balance constraint. Lines 30-37 solves the problem and prints information
about the solution. When used in the TestProblem framework, this gives us
the following output:
Running Maximum Flow problem...

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit http://projects.coin-or.org/Ipopt
******************************************************************************

Optimal solution: f*=-16

Optimal flows:
Edge 0: 10.363

36
Edge 1: 5.63704
Edge 2: 2.36296
Edge 3: 5
Edge 4: 3
Edge 5: 2
Edge 6: 6
Edge 7: 0.196329
Edge 8: 6.80367
Edge 9: 9.19633
Maximum Flow problem successfully solved in 0 (ms)
Press <RETURN> to close this window...
We see that the four edges across the middle of the network are at full capac-
ity, and it is not possible to increase the ow across this cut further, meaning
this solution must be optimal (however, it is not the only optimal solution).

Figure 9: Optimal solution of maximum ow problem. Red edges indicate

that the edge is at its maximum capacity.

6.3.2 Maximum ow with routing

Now consider a case where we have to choose exactly two of the four edges
across the network (e3 - e6 ). To do this, we introduce four new binary
optimization variables b = [b3 , b4 , b5 , b6 ] (we also augment v with four zeros).
When bj = 1, this means we allow ow through edge ej , and when bj = 0,
the ow through edge ej is locked to xj = 0. That is,

[xj , xj ], bj = 1,
xj ∈
{0}, bj = 0.

We can accomplish this by collapsing the variable bounds based on the value
of the binary variables. Consider the following:

bj xj ≤ xj ≤ bj xj .

When bj = 1, we have xj ≤ xj ≤ xj as usual. However, when bj = 0, we

have 0 ≤ xj ≤ 0, meaning we must have xj = 0. This is called collapsing the

37
 bounds. We can split this into two equations and write it on matrix form:
  
bj xj − xj ≤ 0 −1 xj xj
⇒ ≤ 0. (15)
xj − bj xj ≤ 0 1 −xj bj

We want only two pipelines to be switched on, so we add the constraint

b3 + b4 + b5 + b6 = 2. (16)

Applying (15) to all four routing options and combining this with (16), we
get the linear inequality constraint

−1 0 0 0 x3 0 0 0 0
   
 0 −1 0
 
 0 0 x4 0 0  x3
0
 
0
 0 −1 0 0 0 x 5 0   4  0 
  x   
0
 0 0 −1 0 0 0 x6  x5  
  
0

1 0 0 0 −x3 0 0 0  x6   0 
   
 ≤ 
 
0
 1 0 0 0 −x4 0 0   b3   0 
0
 0 1 0 0 0 −x5 0   b4   0 
   
0
 0 0 1 0 0 0 −x6   b5  

0

0 0 0 0 1 1 1 1  b6 2
0 0 0 0 −1 −1 −1 −1 | {z } −2
| {z } x̃ | {z }
AR bR

Optimization problem. Our new optimization problem becomes

 
x>
minimize v (17a)
b
subject to Aint x = 0, (Mass balance) (17b)
AR x̃ ≤ bR , (Routing) (17c)
x ≤ x ≤ x. (Flow capacity) (17d)

Setting up and solving the problem. We set up a new TestProblem

class called FlowWithRouting. The code below is implemented in the solveProblem()
function to set up and solve the ow problem with routing constraints.
1 void FlowWithRouting :: solveProblem ()
2 {
3 // Starting point : zero flow
4 std :: vector < double > z0 {0 , 0 , 0 , 0 , 0 , 0 , 0 , 0, 0 , 0 , 0 ,
0 , 0 , 0};
5 // Flow capacities ( c )
6 std :: vector < double > lb { -13 , -7 , -10 , -5 , -3 , -2 , -6 , -3 ,
-9 , -12 , 0 , 0 , 0 , 0};
7 std :: vector < double > ub { 13 , 7 , 10 , 5 , 3 , 2 , 6 , 3 ,
9 , 12 , 1 , 1 , 1 , 1};
8 // Incidence matrix

38
 9 DenseMatrix A (6 , 10) ;
10 A << -1 , -1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ,
11 1 , 0 , -1 , -1 , -1 , 0 , 0 , 0 , 0 , 0 ,
12 0 , 1 , 1 , 0 , 0 , -1 , -1 , 0 , 0 , 0 ,
13 0 , 0 , 0 , 1 , 0 , 1 , 0 , -1 , -1 , 0 ,
14 0 , 0 , 0 , 0 , 1 , 0 , 1 , 1 , 0 , -1 ,
15 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 1 , 1;
16 // Incidence matrix for internal nodes
17 DenseMatrix A_int = A . block (1 ,0 ,4 ,10) ;
18 // Objective function
19 DenseMatrix v (1 , 14) ;
20 v << 0, 0 , 0 , 0 , 0 , 0 , 0 , 0 , -1 , -1 , 0 , 0 , 0 , 0;
21 ObjectivePtr objective ( new ObjectiveLinear ( v ) ) ;
22 // Mass balance constraints
23 std :: vector < int > massBalanceVars {0 ,1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9};
24 VecD zeros4 ; zeros4 . setZero (4 , 1) ;
25 ConstraintPtr cMassBalance ( new ConstraintLinear ( A_int ,
zeros4 , true ) ) ;
26 // Routing constraints
27 DenseMatrix AR (10 , 8) ;
28 AR << -1 , 0 , 0 , 0 , lb . at (3) , 0, 0, 0,
29 0 , -1 , 0 , 0 , 0 , lb . at (4) , 0, 0,
30 0 , 0 , -1 , 0 , 0, 0 , lb . at (5) , 0,
31 0 , 0 , 0 , -1 , 0, 0, 0 , lb . at (6) ,
32 1 , 0 , 0, 0 , - ub . at (3) , 0, 0, 0,
33 0 , 1 , 0, 0 , 0 , - ub . at (4) , 0, 0,
34 0 , 0 , 1, 0 , 0, 0 , - ub . at (5) , 0,
35 0 , 0 , 0, 1 , 0, 0, 0 , - ub . at (6) ,
36 0 , 0 , 0, 0 , 1, 1, 1, 1,
37 0 , 0 , 0, 0 , -1 , -1 , -1 , -1;
38 VecD bR ; bR . setZero (10 , 1) ;
39 bR (8) = 2;
40 bR (9) = -2;
41 // Variables x3 - x6 and b3 - b6
42 std :: vector < int > routingVars {3 ,4 ,5 ,6 ,10 ,11 ,12 ,13};
43 ConstraintPtr cRouting ( new ConstraintLinear ( AR , bR , false ) ) ;
44 // Constraint composite ( with flow capacity as bounds )
45 ConstraintCompositePtr constraints ( new ConstraintComposite
(14 , lb , ub ) ) ;
46 constraints - > add ( cMassBalance , massBalanceVars );
47 constraints - > add ( cRouting , routingVars ) ;
48 // Variable types
49 std :: vector < int > variable_types ;
50 for ( int i = 0; i < 10; i ++) variable_types . push_back (
CONTINUOUS ) ;
51 for ( int i = 0; i < 4; i ++) variable_types . push_back (
BINARY ) ;
52 // Branching variables ( routing options )
53 std :: vector < int > branching_variables {10 , 11 , 12 , 13};
54 // Optimize
55 BranchAndBound bnb ( objective , constraints , z0 ,
variable_types , branching_variables ) ;
56 int status = bnb . optimize () ;
57 fopt_found = bnb . getObjectiveValue () ;

39
58 zopt_found = bnb . getOptimalSolution () ;
59 cout << " Optimal solution : f *= " << fopt_found << endl ;
60 cout << " Optimal flows : " << endl ;
61 for ( int i = 0; i < 10; i ++) cout << " Edge " << i << " : "
<< zopt_found . at ( i ) << endl ;
62 cout << " Routing decision : " << endl ;
63 for ( int i = 10; i < 14; i ++) cout << " Edge " << i -7 << " :
" << zopt_found . at ( i ) << endl ;
64 }

The dierences between this and the previous example is the following:

• Lines 3-7: Starting point and bound vectors are augmented with four
elements to facilitate the new routing variables.

• Line 23: We have dened variable mapping vector for the mass balance
constraints, since this constraint no longer includes all the variables.

• Lines 27-42 is the implementation of equation (17c); that is, the routing
constraints.

• Lines 45-46 add both constraint pointers to the constraint composite

(in the previous example only one constraint pointer was added).

• Lines 47-50 dene the variable types; variables 0 to 9 (x0 -x9 ) are con-
tinuous, while variables 10 to 13 (b3 -b6 ) are binary.

• Line 52 dene the branching variables; these are the binary variables
b3 -b6 .

• Lines 54-55: We use the BranchAndBound solver in stead of OptimizerIpopt,

since we have a MILP with integer variables which cannot be solved
directly in Ipopt.

This code gives the output

Optimal solution: f*=-11
Optimal flows:
Edge 0: 7.526
Edge 1: 3.474
Edge 2: 2.526
Edge 3: 5
Edge 4: 0
Edge 5: 1.957e-08
Edge 6: 6
Edge 7: 0.2101
Edge 8: 4.79
Edge 9: 6.21
Routing decision:
Edge 3: 1

40
Edge 4: 0
Edge 5: 0
Edge 6: 1
Maximum Flow with Routing problem successfully solved in 370 (ms)
Press <RETURN> to close this window...
This solution is shown graphically in g. 10. As expected, the two edges
with the largest capacity are selected and the maximum ow is 11.

Figure 10: Optimal solution of maximum ow problem with routing con-
straints. Red edges indicate that the edge is at its maximum capacity and
dotted lines indicate that they have been swithced o.

Integer variables. If we want a solution with integer ows, this is eas-

ily achieved by specifying x0 -x9 as integer variables and adding them as
branching variables in lines 48-53:
// Variable types
std :: vector < int > variable_types ;
for ( int i = 0; i < 10; i ++) variable_types . push_back (
INTEGER ) ;
for ( int i = 0; i < 4; i ++) variable_types . push_back (
BINARY ) ;
// Branching variables ( routing options )
std :: vector < int > branching_variables
{0 ,1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10 ,11 ,12 ,13};

This gives the solution shown in g. 11.

41
Figure 11: Optimal solution of maximum ow problem with routing con-
straints and integer variables.

6.3.3 Pressure driven ow case - Linear ow model (INCOM-

PLETE)
Now we will augment the network model to include pressure/potential. We
assume a constant pressure in the source vertex, and a constant pressure in
the sink vertex. Given a set of equations which describe the pressure drop
across each edge/pipeline, we can formulate a maximum ow problem which
takes into account the pressure loss in the pipelines. For this example, we
assume the ow xj through edge ej is given by
xj = f (pA , pB ),
where pA is the pressure in the start vertex and pB is the pressure in the end
vertex (see g. 12)

Figure 12: Flow model.

In this section, we will assume a simple model where the ow is a linear
function of the pressure drop, that is,
f (pA , pB ) = k(pA − pB ).

Optimization variables. We have to add the pressures in each vertex

to the vector of optimization variables. Our new vector of optimization
variables is [x> , p> ]> = [x0 , . . . , x9 , p0 , . . . , p5 , b3 , . . . b6 ]> , where pi is the
pressure in vertex vi .

Objective function. The objective function is the same as in the previous

example, but we must augment v with six zeros to facilitate the new opti-
mization variables: v = [0, 0, 0, 0, 0, 0, 0, 0, −1, −1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]> .

42
Constraints. We keep the mass balance Aint x = 0, and add the ow
equation constraints
x0 = k0 (p0 − p1 )
x1 = k1 (p0 − p2 )
x2 = k2 (p1 − p2 )
x3 = k3 (p1 − p3 )
x4 = k4 (p1 − p4 )
x5 = k5 (p2 − p3 )
x6 = k6 (p2 − p4 )
x7 = k7 (p3 − p4 )
x8 = k8 (p3 − p5 )
x9 = k9 (p4 − p5 ).

We can make use of our incidence matrix and collect the kj 's in a diagonal
matrix K = diag {k0 , . . . , k9 }, and write this as −KA> p = x (we leave it to
the reader to verify this), which gives us the constraint
 
 >
 x
−I −KA = 0.
p

Variable bounds. We keep the ow capacity constraints as bounds on x

and let the pressures in the internal vertices be unbounded. However, we
want constant pressures in the source vertex and the sink vertex. We let the
pressure in the source be 10, and demand the pressure in the sink be greater
than 0. This is accomplished by using the upper and lower bounds for these
two pressures.

Optimization problem. We now have the optimization problem

 
x
minimize v >  p  (18a)
b
subject to Aint x = 0, (Mass balance) (18b)
AR x̃ ≤ bR , (Routing) (18c)
 
 x
> (18d)

−I −KA = 0, (Flow equations)
p
−c ≤ x ≤ c, (Flow capacity) (18e)
10 ≤ p0 ≤ 10, (Source pressure) (18f)
0 ≤ p5 ≤ ∞. (Sink pressure) (18g)

43
6.4 The Six-Hump Camelback function
In this section we will minimize the Six-Hump Camelback function locally
and globally, using various methods. The Six-Hump Camelback function is a
classical test function for optimization algorithms, and is found many places
in the literature. The function has two variables, and is given by

x40
 
2
fSH (x0 , x1 ) = 4 − 2.1x0 + x20 + x0 x1 + (−4 + 4x21 )x21 . (19)
3

The interesting area of this function is in the domain given by −3 ≤ x0 ≤ 3

and −2 ≤ x1 ≤ 2. Within this domain, there are six local minima, two
of which are global minima. The global minimum is fSH ∗ = f (x∗ , x∗ ) =
SH 0 1
−1.0316 at (x∗0 , x∗1 ) = (−0.0898, 0.7126) and (0.0898, −0.7126).

(a) −3 ≤ x0 ≤ 3 and −2 ≤ x1 ≤ 2

(b) A closer look: −2 ≤ x0 ≤ 2 and −1 ≤ x1 ≤ 1

Figure 13: The Six-Hump Camelback function.

44
6.4.1 Formulating the optimization problem
We want to nd the global minimum of the Six-Hump Camelback function,
that is, we want to solve the unconstrained optimization problem

x40
 
minimize x2 = 4 − 2.1x0 + 2
x20 + x0 x1 + (−4 + 4x21 )x21 (20)
3

Note that we have introduced the new variable x2 = fSH (x0 , x1 ). The reason
for this is that we want to state (20) in its epigraph form (see section 1.4.1).
We dene a vector of optimization variables x = [x0 , x1 , x2 ]> . Since we want
to minimize x2 , this gives the linear objective function
 
 x0
f (x) = 0 0 1 x1  = v > x. (21)


x2

We want to restrict x2 to the epigraph of fSH , so we add the constraint

x2 ≥ fSH (x0 , x1 ), which is equivalent to fSH (x0 , x1 ) − x2 ≤ 0. Since we want
both an upper and lower bound on our constraint function, we add the lower
bound −∞ and end up with the reformulated optimization problem

minimize f (x) = v > x (22a)

subject to − ∞ ≤ c1 (x0 , x1 , x2 ) = fSH (x0 , x1 ) − x2 ≤ 0. (22b)

Note that setting the lower bound equal to zero would give the same solution;
this would restrict x2 to lie on the graph of fSH as opposed to on or above
the graph of fSH .

6.4.2 Domain bounds, starting point, objective function and con-

straint set.
Dening the variable domain bounds, starting point, objective function and
constraint set will be the same for all the dierent approaches below, so we
will start with these.

Dene domain bounds. First, we dene our domain bounds. We have

restricted x0 to ±3 and x1 to ±2, while x2 is our free variable and should be
unrestricted (±∞). The domain bounds are dened using vectors of doubles
which are passed to the constraint composite (see section ??).
std :: vector < double > lb ; // Lower bound
std :: vector < double > ub ; // Upper bound

lb . push_back ( -3) ; // Lower bound for x0

lb . push_back ( -2) ; // Lower bound for x1
lb . push_back ( - IPOPT_UNBOUNDED ) ; // Lower bound for x2

45
ub . push_back (3) ; // Upper bound for x0
ub . push_back (2) ; // Upper bound for x1
ub . push_back ( IPOPT_UNBOUNDED ) ; // Upper bound for x2

Dene the starting point. The starting point for the optimization is
also dened using a vector of doubles. Since we are solving the problem
locally, the starting point will aect which solution we end up with. When
following this example, the reader should experiment with several dierent
starting points to see how the solver behaves.
std :: vector < double > z0 ; // Starting point

z0 . push_back (0) ; // Starting point , x0 coordinate

z0 . push_back (0) ; // Starting point , x1 coordinate
z0 . push_back (0) ; // Starting point , x2 coordinate

Dene the objective function. The objective function is dened using

the ObjectiveLinear class (since we have a linear objective function). The
constructor for this class takes an Eigen matrix as an input parameter, which
is the vector v from (21):
int numVars = 3; // Number of variables ( x0 , x1 and x2 )
DenseMatrix v (1 , numVars ) ; // 1 x 3 matrix ( row vector )
v << 0, 0 , 1; // Fill v with values
// Create the objective object
ObjectivePtr objective ( new ObjectiveLinear ( v ) ) ;

Create a ConstraintComposite object. To create the ConstraintComposite

object, we need the number of variables, and the upper and lower bounds for
these variables. These were dened above, so we can proceed with creating
a constraint composite to which we can add constraints later.
ConstraintCompositePtr constraints ( new ConstraintComposite (
numVars , lb , ub ) ) ;

6.4.3 Local optimization using the ConstraintPolynomial class

Note that fSH can be written as a sum of monomial terms:

x60
fSH (x0 , x1 ) = − 2.1x40 + 4x20 + x0 x1 + 4x41 − 4x21 .
3
This means we can write (22b) on the form (8) and implement the Six-Hump
Camelback function as a polynomial constraint. First, we need to nd the

46
vector c and the matrix E used in (8). We have that
x60
− 2.1x40 + 4x20 + x0 x1 + 4x41 − 4x21 − x2
3
1
= x60 x01 x02 − 2.1x40 x01 x02 + 4x20 x01 x02
3
+ x10 x11 x02 + 4x00 x41 x02 − 4x00 x21 x02 − x00 x01 x12 ,
so we get  1   
3 6 0 0
−2.1 4 0 0
   
 4  2 0 0
(23)
   
 1 
c= 1
and E =  1 0.

 4  0 4 0
   
 −4  0 2 0
−1 0 0 1
We are now ready to solve the optimization problem using CENSO. To do
this, we complete the following steps:
• Add the constraint header le
• Dene the Six-Hump Camelback constraint
• Add the constraint to the constraint composite
• Solve the optimization problem with a local solver

Add the constraint header le. The rst thing we must do is to include
the ConstraintPolynomial header le, since we will be using this class to
dene our constraint.
# include " OptimizationProblem / constraintpolynomial . h "

Dene the Six-Hump Camelback constraint. As mentioned above, we

will use the ConstraintPolynomial class to dene the Six-Hump Camelback
function using the vector c and the matrix E from (23). c must be dened
as an Eigen vector and E must be dened as an Eigen matrix:
DenseVector c (7) ; // 7 X 1 Eigen vector
DenseMatrix E (7 , 3) ; // 7 X 3 Eigen matrix

// Fill c and E with values using the << operator

c << (1/3.0) , -2.1 , 4 , 1 , 4 , -4 , -1;
E << 6, 0, 0,
4, 0, 0,
2, 0, 0,
1, 1, 0,
0, 4, 0,
0, 2, 0,
0, 0, 1;

47
Now we can create the ConstraintPolynomial object. The third and fourth
parameters passed to the constructor are the constraint's lower and upper
bound from (22b), which are −∞ and 0, respectively.
ConstraintPtr cSixHump ( new ConstraintPolynomial (c , E , -
IPOPT_UNBOUNDED , 0) ) ;

Add the constraint to the ConstraintComposite object. Now that we

have dened a constraint object, we need to add this to the ConstraintComposite
object that we created earlier. To do this, we have to dene a variable map-
ping to tell the ConstraintComposite object which variables are present in
the constraint being added. In this case, all the variables (x0 , x1 and x2 ) are
present, so we create a vector of integers which contains the indices of each
variable (0, 1 and 2):
std :: vector < int > variableMapping ;
variableMapping . push_back (0) ;
variableMapping . push_back (1) ;
variableMapping . push_back (2) ;

When this vector of integers is passed to the constraint composite object,

we are telling it that variable 0 in the constraint composite corresponds to
variable 0 in the polynomial constraint that is being added, and the same
goes for variables 1 and 2. Now that the variable mapping vector has been
created, we are ready to add our constraint to the constraint composite:
constraints - > add ( cSixHump , variableMapping ) ;

Solve the optimization problem with a local solver. Since we do not

yet have a convex relaxation of our constraint available, we are only able
to solve the optimization problem locally. Now that we have dened our
objective object, constraint composite object and starting point, CENSO
provides a sleek interface to Ipopt for solving optimization problems locally.
All that is required is to create an object of type OptimizerIpopt with our
objective, constraints and starting point, and call its optimize() function:
OptimizerIpopt optIpopt ( objective , constraints , z0 ) ;
int status = optIpopt . optimize () ;

We can also output some information about the solution obtained:

cout << " Solved ( locally ) using Ipopt ! " << endl ;
cout << " Status : " << status << endl ;
cout << " f *: " << optIpopt . getObjectiveValue () << endl ;
cout << " x *: " ;
printVec ( optIpopt . getOptimalSolution () ) ;

With the starting point z0 = (0, 0, 0), we should get the following output:
Solved (locally) using Ipopt!
Status: 1

48
f*: -9.99e-09
x*: ( 0, 0, 0)
This is not the global optimum; Ipopt is stuck in the saddle point located at
the origin. However, if we try dierent starting points, we will obtain better
solutions. For instance, if we give Ipopt the starting point z0 = (1, 0, 0),
it nds the global minimum at x∗ = (0.08984, −0.7127, −1.032), and the
starting point z0 = (−1, 0, 0) gives the other globally optimal point at x∗ =
(−0.08984, 0.7127, −1.032).

6.4.4 Local optimization using a custom constraint class

What if our constraint does not t into the ConstraintPolynomial class or
any of the other contstraint classes provided? In this case, we must create a
custom constraint class which inherits from the Constraint class. If we take
a look in the header le of this class, we see that quite a few of the public
functions are declared virtual. These functions must be implemented in
any class which inherits from the Constraint class. We will create a new
class which inherits from Constraint and call it ConstraintSixHumpCamel.
We start o with the header le constraintsixhumpcamel.h:
# ifndef CONSTRAINTSIXHUMPCAMEL_H
# define CONSTRAINTSIXHUMPCAMEL_H
# include " constraint . h "

class ConstraintSixHumpCamel : public Constraint

{
public :
ConstraintSixHumpCamel () ;
};
# endif // CONSTRAINTSIXHUMPCAMEL_H

We will now complete the following steps to implement this constraint class
and solve the optimization problem:
• Implement the constructor

• Implement the clone function

• Implement the function evaluation

• Implement the Jacobian evaluation

• Implement the Hessian evaluation

• Dene the structure of the Jacobian

• Dene the structure of the Hessian

• Create an instance of the constraint

• Add the constraint to the constraint composite

49
 • Run the optimization problem

Implementing the constructor. In the constructor, we must set some

constraint properties that are used for various checks. The most basic prop-
erties we must set are

• The number of variables (the dimension of the constraint's domain).

• The number of outputs. If there are several equations in one constraint

class this number may be dierent than 1, but in this case there is only
one equation.

• The domain bounds of the variables.

• The domain bounds of the output(s).

• Basic constraint properties:

 Is the Jacobian calculated?

 Is the Hessian calculated?
 Is the constraint linear?
 Is the constraint convex?
 Is there a convex relaxation available?

• The number of nonzero elements in the Jacobian.

• The number of nonzero elements in the Hessian.

We implement the following constructor:

ConstraintSixHumpCamel :: ConstraintSixHumpCamel ()
{
// Dimension of domain and codomain
dimensionDomainF = 3;
dimensionCodomainF = 1;
// Set the variable bounds to be unbounded
setDomainBoundsUnbounded () ;
// Set the output bounds between - Inf and 0
lowerBoundF . push_back ( - IPOPT_UNBOUNDED ) ;
upperBoundF . push_back (0) ;
// Gradient : true , Hessian : true , Linear : false , Convex :
false , Convex relaxation : false
setConstraintProperties ( true , true , false , false , false ) ;
// Number of nonzero elements in the Jacobian and Hessian
nnzGradient = 3;
nnzHessian = 3;
// Check settings for obvious mistakes
checkConstraintSanity () ;
}

50
Implementing the clone function. When a constraint is added to a
constraint composite, what actually happens is that a copy of the constraint
is created by calling the desired constraint class clone() function. Therefore,
we must implement the clone() function here. This function simply returns
a pointer to a clone of the constraint which is created by the copy constructor,
which is a built-in function that creates a new object and copies all the values
of each data member from the original to the copy. We implement it directly
in the header le:
virtual ConstraintSixHumpCamel * clone () const { return new
ConstraintSixHumpCamel (* this ) ;}

Implementing the function evaulation. In the function evaluation func-

tion, we simply implement the function given by (22b). We add the decla-
ration as a public function in the header le:
public :
virtual void evalF ( DenseVector & x , DenseVector & y ) ;

The parameters of the function are

• x, which is an Eigen vector of doubles with the values of each variable.

• y, which is also an Eigen vector of doubles where we write the function

value evaluated at x.
Recall that we wish to evaluate the function
x40
 
2
c1 (x0 , x1 , x2 ) = 4 − 2.1x0 + x20 + x0 x1 + (−4 + 4x21 )x21 − x2 .
3
We implement the function in the .cpp le:
void ConstraintSixHumpCamel :: evalF ( DenseVector & x , DenseVector
& y)
{
double t1 = (4.0 -2.1* pow ( x (0) ,2) + pow ( x (0) ,4) /3.0) ;
double t2 = -4+4* pow ( x (1) ,2) ;

y (0) = t1 * pow ( x (0) ,2) + x (0) * x (1) + t2 * pow ( x (1) ,2) -x (2) ;
}

Here, the doubles t1 and t2 are only used as temporary storage to make the
expression for y(0) a little bit tidier.

Implementing the Jacobian evaulation. To implement the Jacobian

evaulation, we must rst calculate the Jacobian. The Jacobian ∇c1 (x) is
h i
∂c1 ∂c1 ∂c1
∇c1 (x) = ∂x 0
, ∂x1 , ∂x2

= 8x0 − 8.4x30 + 2x50 + x1 , x0 − 8x1 + 16x31 , −1 .

 

51
When implementing the Jacobian evaluation function, the value of nnzGradient
denes the number of elements we must calculate. This value was set to
three because there are three nonzero elements in the Jacobian. We add the
declaration as a public function in the header le:
public :
virtual void evalGradF ( DenseVector & x , DenseVector & dx ) ;

The parameters of the function are

• x, which is an Eigen vector of doubles with the values of each variable.

• dx, which is also an Eigen vector of doubles where we write the value
of the Jacobian elements evaluated at x.

We implement the function in the .cpp le:

void ConstraintSixHumpCamel :: evalGradF ( DenseVector & x ,
DenseVector & dx )
{
dx (0) = 8* x (0) -8.4* pow ( x (0) ,3) +2* pow ( x (0) ,5) + x (1) ;
dx (1) = x (0) -8* x (1) +16* pow ( x (1) ,3) ;
dx (2) = -1;
}

An important remark here is that dx(0) does not necessarily mean the rst
element of the Jacobian; we will later implement the function structureGradF,
which maps each element of dx to the correct element in the Jacobian.

Implementing the Hessian evaluation. We calculate the Hessian ∇2 c1 (x):

8 − 25.2x20 + 10x40
 
1 0
∇2 c1 (x) =  1 48x21 − 8 0 (24)
0 0 0

This Hessian has three nonzero elements, which is the reason we set nnzHessian
to 3 in the constructor. Although there are technically four nonzero elements,
we do not count the symmetric elements of the Hessian, since these are re-
dundant. We add the declaration as a public function in the header le:
public :
virtual void evalHessianF ( DenseVector & x , DenseVector & ddx )
;

The parameters of the function are

• x, which is an Eigen vector of doubles with the values of each variable.

• ddx, which is also an Eigen vector of doubles where we write the value
of the Hessian elements evaluated at x.

We implement the function in the .cpp le:

52
void ConstraintSixHumpCamel :: evalHessianF ( DenseVector & x ,
DenseVector & ddx )
{
ddx (0) = 8 -25.2* pow ( x (0) ,2) +10* pow ( x (0) ,4) ;
ddx (1) = 1;
ddx (2) = 48* pow ( x (1) ,2) -8;
}

As with the Jacobian, the elements of ddx contain the values of the nonzero
elements, and we will dene the position of these elements later in the func-
tion structureHessianF.

Dening the structure of the Jacobian. Next, we will implement the

function structureGradF, which denes the position of the nonzero elements
in the Jacobian. We add the declaration as a public function in the header
le:
public :
virtual void structureGradF ( std :: vector < int >& iRow , std ::
vector < int >& jCol ) ;

The parameters of the function are

• iRow, which is an STL vector of integers with the row indices of the
elements of dx.

• jCol, which is an STL vector of integers with the column indices of

the elements of dx.

In our case, the calculated values assume the following positions in the Ja-
cobian:
Col 0 Col 1 Col 2
Row 0 dx(0) dx(1) dx(2)
That is, the value calculated for dx(0) should be placed in the (0,0) element
of the Jacobian, dx(1) should be placed in the (0,1) element and dx(2)
should be placed in the (0,2) element. This gives the following implementa-
tion in the .cpp le:
void ConstraintSixHumpCamel :: structureGradF ( std :: vector < int >
& iRow , std :: vector < int > & jCol )
{
// Position of dx (0) is (0 ,0)
iRow . push_back (0) ; jCol . push_back (0) ;
// Position of dx (1) is (0 ,1)
iRow . push_back (0) ; jCol . push_back (1) ;
// Position of dx (2) is (0 ,2)
iRow . push_back (0) ; jCol . push_back (2) ;
}

53
Dening the structure of the Hessian. The function structureHessianF
denes the position of the nonzero elements in the Hessian. We add the dec-
laration as a public function in the header le:
public :
virtual void structureHessianF ( std :: vector < int >& eqnr , std
:: vector < int >& iRow , std :: vector < int >& jCol ) ;

The parameters of this function are

• eqnr, which is an STL vector of integers which denes the equation

number associated with the position dened in iRow/jCol. This equa-
tion number is associated with the row number in the Jacobian from
which the Hessian is calculated; is the codomain of the constraint func-
tion has dimension 1, the equation number is always zero.

• iRow, which is an STL vector of integers with the row indices of the
elements of ddx.

• jCol, which is an STL vector of integers with the column indices of

the elements of ddx.

In our case, the calculated values assume the following positions in the Hes-
sian:
Col 0 Col 1 Col 2
Row 0 ddx(0) ddx(1) −
Row 1 ddx(1) ddx(2) −
Row 2 − − −
The implementation becomes
void ConstraintSixHumpCamel :: structureHessianF ( std :: vector < int
> & eqnr , std :: vector < int > & iRow , std :: vector < int > &
jCol )
{
// Position of ddx (0) is (0 ,0)
eqnr . push_back (0) ; iRow . push_back (0) ; jCol . push_back (0) ;
// Position of ddx (1) is (0 ,1) ( and (1 ,0) )
eqnr . push_back (0) ; iRow . push_back (0) ; jCol . push_back (1) ;
// Position of ddx (2) is (1 ,1)
eqnr . push_back (0) ; iRow . push_back (1) ; jCol . push_back (1) ;
}

Create an instance of the constraint. We use the following code to

create an instance of the constraint class we just created:
ConstraintPtr cSixHump ( new ConstraintSixHumpCamel () );

54
Add the constraint to the ConstraintComposite object. Now that we
have dened a constraint object, we need to add this to the ConstraintComposite
object that we created earlier. This is done the same way as for the polyno-
mial constraint:
constraints - > add ( cSixHump , variableMapping ) ;

Running the optimization problem. Now we are ready to solve the

optimization problem again, with the new constraint class:
OptimizerIpopt optIpopt ( objective , constraints , z0 ) ;
int status = optIpopt . optimize () ;

Of course we can display solution information here the same way as we could
when we used the polynomial constraint class.

6.4.5 Global optimization using Alpha-Branch-and-Bound

As explained in section 1.4.4, it is possible to nd the global minimum of
a function if we apply a convex relaxation of the function in a Branch-
and-Bound framework, as long as the relaxation improves as the domain
bounds shrink. In this section, we will implement a (pretty bad) convex
relaxation of the Six-Hump Camelback function using a technique presented
in [1] called Alpha-Branch-and-Bound (αBB). A detailed explanation of this
approach is beyond the scope of this manual, but in short, it nds a convex
underestimator of the (in our case) Six-Hump Camelback function which
depends on the domain bounds. The term convex underestimator refers to
a function that is convex, and that is less than or equal to the Six-Hump
Camelback function for all x within the domain bounds, that is

f SH (x) ≤ fSH (x), ∀ xL ≤ x ≤ xU ,

where f SH (x) is the convex underestimator of fSH (x). In this example we

will use a simple variant, where the convex underestimator is on the form
X
f SH (x) = fSH (x) + α (xL U
i − xi )(xi − xi ). (25)
i

It can be shown that this function is convex if and only if

 
1
α ≥ max 0, − min λi (x) , (26)
2 i,xL ≤x≤xU

where the λi (x)'s are the eigenvalues of Hf (x), the Hessian of f (x). To
nd the required value of α for our convex relaxation, we need to know the
minimum eigenvalue of the Hessian of f (x). This is not trivial to solve.
However, Thm. 3.2 in [1] gives us a neat trick that we can use to nd a

55
lower bound on the minimum eigenvalue of a matrix A = {aij }. It is given
by  
X
λmin ≥ min aii − max(|aij |, |aij |) (27)
i
j6=i

In other words, if we can nd upper and lower bounds on each element of
the Hessian, we can also get a lower bound on the minimum eigenvalue. The
Hessian of the Six-Hump Camelback function was given in (24). Note that
all the elements are constant except for the (0,0) element, which is a function
of x0 only, and the (1,1) element, which is a function of x1 only. We can
exploit this to nd upper and lower bounds. In g. 14 we see the (0,0)
element of the Hessian as a function of x0 . We know that both the minimum
and maximum of this function must be located either in the end points (xL 0
and xU 0 ) or in one of the stationary points, which are located at (0, 8) and
(±1.225, −7.876).

800 10

700

9.5
600

500
9
H(0,0)

H(0,0)

400

8.5
300

200
8

100

0 7.5
−3 −2 −1 0 1 2 3 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8
x0 x0

(a) −3 ≤ x0 ≤ 3 (b) −0.8 ≤ x0 ≤ 0.8

Figure 14: (0,0) element of Hessian dened in (24).

The same goes for the (1,1) element, except that this is even easier since
it is only a second degree polynomial, and it is convex (see g. 15). Here,
the minimum and and maximum must be located either in xL 1 , x1 , or the
U

stationary point at (0, −8).

200

150

100
H(1,1)

50

−50
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2
x1

Figure 15: (1,1) element of Hessian dened in (24) (−2 ≤ x1 ≤ 2).

56
 We create a new constraint class which we name ConstraintSixHumpCamelWithRelaxation,
which starts out the same as the ConstraintSixHumpCamel class. That is,
our starting point is the following header le:
# ifndef CONSTRAINTSIXHUMPCAMELWITHRELAXATION_H
# define CONSTRAINTSIXHUMPCAMELWITHRELAXATION_H
# include " constraint . h "

class ConstraintSixHumpCamelWithRelaxation : public Constraint

{
public :
ConstraintSixHumpCamelWithRelaxation () ;

// Clone function - uses copy constructor

virtual ConstraintSixHumpCamelWithRelaxation * clone () const
{ return new ConstraintSixHumpCamelWithRelaxation (* this
) ;}

virtual void evalF ( DenseVector & x , DenseVector & y ) ;

virtual void evalGradF ( DenseVector & x , DenseVector & dx ) ;

virtual void evalHessianF ( DenseVector & x , DenseVector & ddx )

;

virtual void structureGradF ( std :: vector < int >& iRow , std ::
vector < int >& jCol ) ;

virtual void structureHessianF ( std :: vector < int >& eqnr , std
:: vector < int >& iRow , std :: vector < int >& jCol ) ;
};

# endif // CONSTRAINTSIXHUMPCAMELWITHRELAXATION_H

We now go through the steps required to modify this class to allow global
optimization. These steps are:

• Add a pointer to a relaxed constraint as a private data member

• Modify the constructor

• Implement the copy constructor

• Implement the setDomainBounds() function

• Implement the computeConvexRelaxation() and getConvexRelaxation()

functions

• Create a new constraint class to represent the convex relaxation

• Solve the optimization problem in a Branch-and-Bound framework

57
Add a pointer to a relaxed constraint as a private data member.
We add the following in the header le:
private :
ConstraintPtr relaxedConstraint ;

This is a pointer to the relaxation of the constraint. In the Branch-and-

Bound framework, many instances of the constraint will be created, and
each instance maintains its own relaxed constraint.

Modify the constructor. We need to make some changes to the con-

structor:

• We have to set the constraint property convexRelaxationAvailable

to true, to tell the Branch-and-Bound framework that this constraint
has a convex relaxation available. The easiest way to accomplish this is
to change the corresponding parameter of the setConstraintProperties
function.

• We have to create a convex relaxation of the constraint along with the

constraint itself. To do this, we need to know the domain bounds.
Therefore, we add two parameters to the constructor lb and ub, which
are STL vectors of doubles. In the header le, the declaration becomes
public :
ConstraintSixHumpCamelWithRelaxation ( std :: vector <
double > lb , std :: vector < double > ub ) ;

We use these parameters to update the domainLowerBound and domainUpperBound

members, respectively. Now we can create some function computeConvexRelaxation()
that uses these domain bounds to calculate the convex relaxation, and
updates the relaxedConstraint member. We will dene the function
later, for now we just call it in the constructor.

The implementation in the .cpp le becomes:

ConstraintSixHumpCamelWithRelaxation ::
ConstraintSixHumpCamelWithRelaxation ( std :: vector < double >
lb , std :: vector < double > ub ) :
relaxedConstraint ( nullptr )
{
// Dimension of domain and codomain
dimensionDomainF = 3;
dimensionCodomainF = 1;
// Check consistency of domain bounds
assert ( lb . size () == ub . size () ) ;
assert ( lb . size () == dimensionDomainF ) ;
// Set domain bounds
domainLowerBound = lb ;
domainUpperBound = ub ;
// Set the output bounds between - Inf and 0

58
 lowerBoundF . push_back ( - IPOPT_UNBOUNDED );
upperBoundF . push_back (0) ;
// Gradient : true , Hessian : true , Linear : false , Convex :
false , Convex relaxation : true
setConstraintProperties ( true , true , false , false , true ) ;
// Number of nonzero elements in the Jacobian and Hessian
nnzGradient = 3;
nnzHessian = 3;
// Compute convex relaxation
computeConvexRelaxation () ;
// Check settings for obvious mistakes
checkConstraintSanity () ;
}

Implement the copy constructor. The default copy constructor only

copies the members of the object, not any objects they might be pointing
to. In this case, the smart pointer relaxedConstraint will be copied, but
the ConstraintSixHumpCamelConvexRelaxation object it points to will not.
This is illustrated in g. 16. This behaviour will cause problems when one
of the constraint objects makes changes to its relaxation, since these changes
then will also apply to the copy of the constraint (which has made no such
changes to its relaxation).

Figure 16: Default copy constructor behaviour.

The solution is to implement a copy constructor that clones the relaxed

constraint as well, so that the copy of the constraint gets its very own instance
of the relaxed constraint. This is illustrated in g. 17.

Figure 17: Desired copy constructor behaviour.

59
 To accomplish this, we add the declaration in the header le:
public :
ConstraintSixHumpCamelWithRelaxation (
ConstraintSixHumpCamelWithRelaxation const & copy ) ;

and the implementation in the .cpp le:

ConstraintSixHumpCamelWithRelaxation ::
ConstraintSixHumpCamelWithRelaxation (
ConstraintSixHumpCamelWithRelaxation const & copy )
: Constraint ( copy )
{
if ( copy . relaxedConstraint == nullptr )
{
relaxedConstraint = nullptr ;
}
else
{
relaxedConstraint = ConstraintPtr ( copy .
relaxedConstraint - > clone () ) ;
}
}

Note that we have to check whether the relaxation of the constraint we

are trying to copy exists or not, before we call the clone() function to
create a copy of the relaxation. Also, note that we also call the default copy
constructor of the Constraint class (: Constraint(copy)) to copy all the
basic constraint properties.

Implement the setDomainBounds() function. When the domain bounds

are changed, we must also calculate a new convex relaxation, since the re-
laxation changes when the bounds change. To do this, we overload the
setDomainBounds() function of the Constraint class to include the calcu-
lation of a new relaxation:
void ConstraintSixHumpCamelWithRelaxation :: setDomainBounds ( std
:: vector < double > lb , std :: vector < double > ub )
{
// Check parameters
assert ( lb . size () == dimensionDomainF ) ;
assert ( ub . size () == dimensionDomainF ) ;
// Avoid unnecessary updates
if ( compareVec ( lb , domainLowerBound ) && compareVec ( ub ,
domainUpperBound ) ) return ;
// Set domain bounds
Constraint :: setDomainBounds ( lb , ub ) ;
// Recalculate the convex relaxation
computeConvexRelaxation () ;
}

60
Implement the computeConvexRelaxation() and getConvexRelaxation()
functions. To implement these functions, we need equations (26) and (27).
The function computeConvexRelaxation() is declared a private function:
private :
void computeConvexRelaxation () ;

Its implementation looks like this:

void ConstraintSixHumpCamelWithRelaxation ::
computeConvexRelaxation ()
{
// Matrices to represent the interval Hessian
DenseMatrix H_lo , H_hi ;
// Compute interval Hessian ( depends on current domain
bounds )
intervalHessian ( H_lo , H_hi ) ;
// Calculate lower bound on minimum eigenvalue
double minEigenVal = minIntervalEigenValue ( H_lo , H_hi ) ;
// Calculate required value for alpha
double alpha = fmax (0 , -0.5* minEigenVal ) ;

// Create new relaxed constraint based on alpha

relaxedConstraint = ConstraintPtr ( new
ConstraintSixHumpCamelConvexRelaxation ( domainLowerBound
, domainUpperBound , alpha ) ) ;
}

The rst line:

// Matrices to represent the interval Hessian
DenseMatrix H_lo , H_hi ;

declares two matrices H_lo and H_hi, which represent the lower and upper
bounds on the elements of the Hessian (24). These bounds are calculated in
the second line:
// Compute interval Hessian ( depends on current domain bounds )
intervalHessian ( H_lo , H_hi ) ;

The function intervalHessian has the declaration

private :
void intervalHessian ( DenseMatrix & H_lo , DenseMatrix & H_hi );

and the implementation (refer to the comments to see exactly what the
function does):
void ConstraintSixHumpCamelWithRelaxation :: intervalHessian (
DenseMatrix & H_lo , DenseMatrix & H_hi )
{
// The Hessian of the six hump camelback function has the
following form :
//
// | 8 - 25.2 x0 ^2 + 10 x0 ^4 1 0 |
// H = | 1 48 x1 ^2 - 8 0 |
// | 0 0 0 |

61
//
// When the bounds on x0 and x1 are given , it is easy to
// find bounds on the elements of the Hessian matrix .
// The only non - constant elements are the (0 ,0) element
// and the (1 ,1) element . The (0 ,0) element has three
// stationary points at x0 = 0, x0 = +/ - 1.2249722
// where the function value is 8 and -7.876 , respectively .
// The (1 ,1) element has one stationary point at x1 = 0 ,
// where the function value is -8.

// x0 and x1 must be bounded for us to be able to find an

// interval Hessian .
// Upper bound on x0
assert ( domainUpperBound . at (0) < IPOPT_UNBOUNDED ) ;
// Upper bound on x1
assert ( domainUpperBound . at (1) < IPOPT_UNBOUNDED ) ;
// Lower bound on x0
assert ( domainLowerBound . at (0) > - IPOPT_UNBOUNDED ) ;
// Lower bound on x1
assert ( domainLowerBound . at (1) > - IPOPT_UNBOUNDED ) ;

// Max / min candidates

std :: vector < double > maxMinCandidates ;

// Find bounds on (0 ,0) element

double statPoint1 = -1.2249722; // Stationary points
double statPoint2 = 0;
double statPoint3 = 1.12249722;

// Extreme points ( at lower / upper bound )

double y0L , y0U , x0L , x0U ;
x0L = domainLowerBound . at (0) ;
x0U = domainUpperBound . at (0) ;
y0L = 8 - 25.2 * pow ( x0L , 2) + 10 * pow ( x0L , 4) ;
y0U = 8 - 25.2 * pow ( x0U , 2) + 10 * pow ( x0U , 4) ;
maxMinCandidates . push_back ( y0L ) ;
maxMinCandidates . push_back ( y0U ) ;

// Stationary points are added to candidate points

// if they are within the domain bounds
if ( valueWithinBounds ( statPoint1 , 0) ) maxMinCandidates .
push_back ( -7.876) ; // Value at x0 = -1.2249722
if ( valueWithinBounds ( statPoint2 , 0) ) maxMinCandidates .
push_back (0) ; // Value at x0 = 0
if ( valueWithinBounds ( statPoint3 , 0) ) maxMinCandidates .
push_back ( -7.876) ; // Value at x0 = 1.2249722

// Find the largest and smalles elements in the vector of

// max / min candidates
double maxElem00 = * std :: max_element ( maxMinCandidates . begin
() , maxMinCandidates . end () ) ;
double minElem00 = * std :: min_element ( maxMinCandidates . begin
() , maxMinCandidates . end () ) ;

62
 maxMinCandidates . clear () ;

// Find bounds on (1 ,1) element

double y1L , y1U , x1L , x1U ;
x1L = domainLowerBound . at (1) ;
x1U = domainUpperBound . at (1) ;
y1L = 48 * pow ( x1L , 2) - 8;
y1U = 48 * pow ( x1U , 2) - 8;
maxMinCandidates . push_back ( y1L ) ;
maxMinCandidates . push_back ( y1U ) ;
if ( valueWithinBounds (0 , 1) ) maxMinCandidates . push_back ( -8)
; // Value at x1 = 0

// Find the largest and smalles elements in the vector of

// max / min candidates
double maxElem11 = * std :: max_element ( maxMinCandidates . begin
() , maxMinCandidates . end () ) ;
double minElem11 = * std :: min_element ( maxMinCandidates . begin
() , maxMinCandidates . end () ) ;

// Fill out upper / lower bounds

H_lo . setZero (3 , 3) ;
H_lo (0 ,0) = minElem00 ;
H_lo (0 ,1) = 1;
H_lo (1 ,0) = 1;
H_lo (1 ,1) = minElem11 ;
H_hi . setZero (3 , 3) ;
H_hi (0 ,0) = maxElem00 ;
H_hi (0 ,1) = 1;
H_hi (1 ,0) = 1;
H_hi (1 ,1) = maxElem11 ;
}

This function uses the function valueWithinBounds(double value, int

varNo), which only checks whether value lies within the domain bounds
of variable number varNo. The declaration is
private :
bool valueWithinBounds ( double value , int varNo ) ;

and the implementation is:

bool ConstraintSixHumpCamelWithRelaxation :: valueWithinBounds (
double value , int varNo )
{
// Check that varNo is a valid variable index
assert ( varNo < dimensionDomainF ) ;
// Check if value is within bounds , return true / false
if ( domainLowerBound . at ( varNo ) <= value && domainUpperBound
. at ( varNo ) >= value )
{
return true ;
}
else
{

63
 return false ;
}
}

Now that we have calculated bounds on each element in the Hessian, we can
calculate a lower bound on the minimum eigenvalue of the Hessian. This is
done in the third line:
// Calculate lower bound on minimum eigenvalue
double minEigenVal = minIntervalEigenValue ( H_lo , H_hi ) ;

The function minIntervalEigenValue is nothing else than an application of

(27). The declaration is
private :
double minIntervalEigenValue ( DenseMatrix & H_lo , DenseMatrix
& H_hi ) ;

and the implementation is

double ConstraintSixHumpCamelWithRelaxation ::
minIntervalEigenValue ( DenseMatrix & H_lo , DenseMatrix & H_hi )
{
// Uses Thm . 3.2 in Adjiman et . al (1998) to find a lower
// bound on the minimum eigenvalue of the interval matrix
// given by H_lo and H_hi .

// Check that both matrices are square and that they are
// the same size
assert ( H_lo . rows () == H_lo . cols () ) ;
assert ( H_hi . rows () == H_hi . cols () ) ;
assert ( H_lo . rows () == H_hi . rows () ) ;
assert ( H_lo . cols () == H_hi . cols () ) ;

int dim = H_lo . rows () ;

double minEigenVal = IPOPT_UNBOUNDED ;

for ( int i = 0; i < dim ; i ++)
{
double minCandidate = H_lo (i , i ) ;
for ( int j = 0; j < dim ; j ++)
{
if ( i != j )
{
minCandidate -= fmax ( fabs ( H_lo (i , j ) ) , fabs (
H_hi (i , j ) ) ) ;
}
}
if ( minCandidate < minEigenVal )
{
minEigenVal = minCandidate ;
}
}
return minEigenVal ;
}

64
We use the obtained bound to calculate the required value of α with (26) in
line four:
// Calculate required value for alpha
double alpha = fmax (0 , -0.5* minEigenVal ) ;
and pass this value to the constructor of the relaxed constraint class in the
fth and nal line:
// Create new relaxed constraint based on alpha
relaxedConstraint = ConstraintPtr ( new
ConstraintSixHumpCamelConvexRelaxation ( domainLowerBound
, domainUpperBound , alpha ) ) ;
The function getConvexRelaxation() simply clones the relaxed constraint
and returns a pointer to the clone. It has the declaration
public :
virtual Constraint * getConvexRelaxation () const ;
and the implementation
Constraint * ConstraintSixHumpCamelWithRelaxation ::
getConvexRelaxation () const
{
assert ( relaxedConstraint != nullptr ) ;
return relaxedConstraint - > clone () ;
}

Create a new constraint class to represent the convex relaxation.

We need to create a new class to represent the convex relaxations. We will
call it ConstraintSixHumpCamelConvexRelaxation. The implementation of
this class will be very similar to the implementation of the ConstraintSixHumpCamel
class from section 6.4.4, with a few dierences:
• The function, Jacobian and Hessian evaluations are obviously slightly
dierent.
• The constructor takes the domain bounds as parameters as these are
used in the function evaluations.
• It contains the private data member alpha, which is also used in the
function evaluations.
• The constraint properties constraintConvex and convexRelaxationAvailable
are set to true.
Let the convex relaxation of c1 (x) be denoted c1 (x). Then we have
c1 (x) = fSH (x0 , x1 ) + α(xL U L U
0 − x0 )(x0 − x0 ) + α(x1 − x1 )(x1 − x1 ) − x2
x40
 
2
= 4 − 2.1x0 + x20 + x0 x1 + (−4 + 4x21 )x21
3
+ α(xL U L U
0 − x0 )(x0 − x0 ) + α(x1 − x1 )(x1 − x1 ) − x2

65
The gradient ∇c1 (x) is
>
8x0 − 8.4x30 + 2x50 + x1 + 2αx0 − α(xL U

0 + x0 )
∇c1 (x) =  x0 − 8x1 + 16x31 + 2αx1 − α(xL U
1 + x1 )
 .
−1

The Hessian ∇2 c1 (x) is

8 − 25.2x20 + 10x40 + 2α
 
1 0
∇2 c1 (x) =  1 48x21 − 8 + 2α 0 .
0 0 0

This gives us the header le

# ifndef CONSTRAINTSIXHUMPCAMELCONVEXRELAXATION_H
# define CONSTRAINTSIXHUMPCAMELCONVEXRELAXATION_H

# include " constraint . h "

class ConstraintSixHumpCamelConvexRelaxation : public

Constraint
{
public :
ConstraintSixHumpCamelConvexRelaxation ( std :: vector < double
> lb , std :: vector < double > ub , double alpha ) ;

// Clone function - uses copy constructor

virtual ConstraintSixHumpCamelConvexRelaxation * clone ()
const { return new
ConstraintSixHumpCamelConvexRelaxation (* this ) ;}

virtual void evalF ( DenseVector & x , DenseVector & y ) ;

virtual void evalGradF ( DenseVector & x , DenseVector & dx ) ;

virtual void evalHessianF ( DenseVector & x , DenseVector & ddx )

;

virtual void structureGradF ( std :: vector < int >& iRow , std ::
vector < int >& jCol ) ;

virtual void structureHessianF ( std :: vector < int >& eqnr , std
:: vector < int >& iRow , std :: vector < int >& jCol ) ;

private :
double alpha ;
};

# endif // CONSTRAINTSIXHUMPCAMELCONVEXRELAXATION_H

and the .cpp le

# include " constraintsixhumpcamelconvexrelaxation . h "

66
ConstraintSixHumpCamelConvexRelaxation ::
ConstraintSixHumpCamelConvexRelaxation ( std :: vector < double
> lb , std :: vector < double > ub , double alpha )
{
dimensionDomainF = 3;
dimensionCodomainF = 1;

gradientCalculatedF = true ;
hessianCalculatedF = true ;

assert ( lb . size () == ub . size () ) ;

assert ( lb . size () == dimensionDomainF ) ;

domainLowerBound = lb ;
domainUpperBound = ub ;

lowerBoundF . push_back ( - IPOPT_UNBOUNDED );

upperBoundF . push_back (0) ;

this - > alpha = alpha ;

// Gradient : true , Hessian : true , Linear : false , Convex :

true , Convex relaxation : true
setConstraintProperties ( true , true , false , true , true ) ;

nnzGradient = 3;
nnzHessian = 3;

checkConstraintSanity () ;
}

void ConstraintSixHumpCamelConvexRelaxation :: evalF ( DenseVector

& x , DenseVector & y )
{
double x0L = domainLowerBound . at (0) ;
double x0U = domainUpperBound . at (0) ;
double x1L = domainLowerBound . at (1) ;
double x1U = domainUpperBound . at (1) ;

double t1 = (4.0 -2.1* pow ( x (0) ,2) + pow ( x (0) ,4) /3.0) * pow ( x (0)
,2) ;
double t2 = x (0) * x (1) ;
double t3 = ( -4+4* pow ( x (1) ,2) ) * pow ( x (1) ,2) ;
double t4 = alpha *( x0L * x0U - x (0) *( x0L + x0U ) + pow ( x (0) ,2) ) ;
double t5 = alpha *( x1L * x1U - x (1) *( x1L + x1U ) + pow ( x (1) ,2) ) ;

y (0) = t1 + t2 + t3 + t4 + t5 - x (2) ;
}

void ConstraintSixHumpCamelConvexRelaxation :: evalGradF (

DenseVector & x , DenseVector & dx )
{
double x0L = domainLowerBound . at (0) ;

67
 double x0U = domainUpperBound . at (0) ;
double x1L = domainLowerBound . at (1) ;
double x1U = domainUpperBound . at (1) ;
dx (0) = 8* x (0) -8.4* pow ( x (0) ,3) +2* pow ( x (0) ,5) + x (1) +2* alpha * x
(0) - alpha *( x0L + x0U ) ;
dx (1) = x (0) -8* x (1) +16* pow ( x (1) ,3) +2* alpha * x (1) - alpha *( x1L +
x1U ) ;
dx (2) = -1;
}

void ConstraintSixHumpCamelConvexRelaxation :: evalHessianF (

DenseVector & x , DenseVector & ddx )
{
ddx (0) = 8 -25.2* pow ( x (0) ,2) +10* pow ( x (0) ,4) +2* alpha ;
ddx (1) = 1;
ddx (2) = -8+48* pow ( x (1) ,2) +2* alpha ;
}

void ConstraintSixHumpCamelConvexRelaxation :: structureGradF ( std

:: vector < int > & iRow , std :: vector < int > & jCol )
{
iRow . push_back (0) ; jCol . push_back (0) ;
iRow . push_back (0) ; jCol . push_back (1) ;
iRow . push_back (0) ; jCol . push_back (2) ;
}

void ConstraintSixHumpCamelConvexRelaxation :: structureHessianF (

std :: vector < int > & eqnr , std :: vector < int > & iRow , std ::
vector < int > & jCol )
{
eqnr . push_back (0) ; iRow . push_back (0) ; jCol . push_back (0) ;
eqnr . push_back (0) ; iRow . push_back (0) ; jCol . push_back (1) ;
eqnr . push_back (0) ; iRow . push_back (1) ; jCol . push_back (1) ;
}

Solve the optimization problem in a Branch-and-Bound frame-

work. We will now use the ConstraintSixHumpCamelWithRelaxation class
we created in the paragraphs above to solve the Six-Hump Camelback prob-
lem globally. First, we create an instance of the constraint and add it to the
constraint composite as we have done earlier:
ConstraintPtr cSixHump ( new ConstraintSixHumpCamelWithRelaxation
( lb , ub ) ) ;
constraints - > add ( cSixHump , variableMapping ) ;

In addition to the constraints and objective function, the Branch-and-Bound

framework needs to know the variable types (continuous/integer) and which
variables should be branced on. In our case, all variables are continuous and
x0 and x1 are the branching variables. We create two STL integer vectors
and insert the correct variable indices:
std :: vector < int > variable_types ;

68
variable_types . push_back ( CONTINUOUS ) ; // x0
variable_types . push_back ( CONTINUOUS ) ; // x1
variable_types . push_back ( CONTINUOUS ) ; // x2

std :: vector < int > branching_variables ;

branching_variables . push_back (0) ;
branching_variables . push_back (1) ;

Now we can create a BranchAndBound object, which takes the objective, con-
straints, starting point, variable types and branching variables as parameters,
and call its optimize() function to run the Branch-and-Bound algorithm:
BranchAndBound bnb ( objective , constraints , z0 , variable_types ,
branching_variables ) ;
bnb . optimize () ;

The optimal point and objective function value are available through the
getOptimalSolution() and getObjectiveValue() functions, respectively:
zopt_found = bnb . getOptimalSolution () ;
fopt_found = bnb . getObjectiveValue () ;

The Branch-and-Bound algorithm nds the global optimum, but achieving

epsilon-convergence takes a long time due to the poor quality of the convex
relaxations. For instance, with the starting point z0 = (0, 0, 0), it takes
2671 iterations to achieve epsilon-convergence (see g. 18(a)), and with the
starting point z0 = (1, 0, 0) it takes 4031 iterations (g. 18(b)). However,
the global optimal point is found within a few iterations; it is the shrinking
of the optimality gap (our certicate of a global solution) that takes a long
time.

0 0

−5 −5

−10 −10

−15 −15
Obj. fn. value

Obj. fn. value

−20 −20

−25 −25

−30 −30

−35 −35
Upper bound Upper bound
Lower bound Lower bound
−40 −40
0 500 1000 1500 2000 2500 0 500 1000 1500 2000 2500 3000 3500 4000
Iteration no. Iteration no.

(a) z0 = (0, 0, 0) (b) z0 = (1, 0, 0)

Figure 18: Branch-and-Bound algorithm progress with the αBB approach.

6.4.6 Global optimization using a B-spline approximation

The methods desribed in the paragraphs above are time consuming and
error-prone. An easier approach is to use a B-spline approximation of the Six-
Hump Camelback function to solve the problem globally. As mentioned in

69
section 1.4.3, an attractive property of the B-spline is that the control points
used to describe the B-spline can be used to construct a convex relaxation of
the B-spline. This property has been exploited in the ConstraintBspline
class, meaning that we can solve problems dened with B-splines globally.
To solve the Six-Hump Camelback problem with this approach, we have to
complete the following steps:
• Create an InterpolationTable object and ll it with samples
• Create a B-spline constraint using the interpolation table
• Add the constraint to the constraint composite
• Solve the optimization problem in a Branch-and-Bound framework

Create an InterpolationTable object and ll it with samples. The

B-spline constraint constructor takes three parameters; an interpolation ta-
ble lled with samples, the desired degree of the polynomial pieces from
which the B-spline is constructed, and a boolean value to indicate equality
or inequality. First, we create a function to evaluate the Six-Hump Camel-
back function itself:
DenseVector sixHumpCamelFunction ( DenseVector x )
{
assert ( x . rows () == 2) ;
DenseVector y ; y . setZero (1) ;
y (0) = (4 - 2.1* x (0) * x (0) + (1/3.) * x (0) * x (0) * x (0) * x (0) ) * x
(0) * x (0) + x (0) * x (1) + ( -4 + 4* x (1) * x (1) ) * x (1) * x (1) ;
return y ;
}
Now, we use this function to evaluate the Six-Hump Camelback function in
a grid (see section ?? for details on the InterpolationTable class).
InterpolationTable * data = new InterpolationTable (2 , 1 , false ) ;

double dx = 0.05;
for ( double x1 = lb . at (0) ; x1 <= ub . at (0) ; x1 += dx )
{
for ( double x2 = lb . at (1) ; x2 <= ub . at (1) ; x2 += dx )
{
std :: vector < double > x ;
x . push_back ( x1 ) ;
x . push_back ( x2 ) ;

DenseVector xd ; xd . setZero (2) ;

xd (0) = x1 ;
xd (1) = x2 ;
DenseVector yd = sixHumpCamelFunction ( xd ) ;

data - > addSample (x , yd (0) ) ;

}
}

70
Create a B-spline constraint using the interpolation table. We se-
lect a polynomial degree of 3, since this gives us a B-spline which is twice
continuously dierentiable. We set the third parameter to true to indicate
that we want an equality constraint.
ConstraintPtr cbspline ( new ConstraintBspline (* data , 3 , true ) ) ;

Add the constraint to the constraint composite. As before,

constraints - > add ( cbspline , variableMapping ) ;

Solve the optimization problem in a Branch-and-Bound frame-

work. This is done in the same way as described for the αBB approach;
we create two STL vectors with variable types and brancing variables, and
pass these to a BranchAndBound object along with the objective, constraints
and starting point:
std :: vector < int > variable_types ;
variable_types . push_back ( CONTINUOUS ) ; // x0
variable_types . push_back ( CONTINUOUS ) ; // x1
variable_types . push_back ( CONTINUOUS ) ; // x2
std :: vector < int > branching_variables ;
branching_variables . push_back (0) ;
branching_variables . push_back (1) ;

BranchAndBound bnb ( objective , constraints , z0 , variable_types ,

branching_variables ) ;
bnb . optimize () ;

zopt_found = bnb . getOptimalSolution () ;

fopt_found = bnb . getObjectiveValue () ;

This time, the problem is solved in only 49 iterations. The large improvement
in performance is due to the improved quality of the convex relaxations. The
progress of the algorithm is shown in g. 19. Note that the initial lower
bound is very close to the upper bound (< 0.01 as opposed to > 25-30 for
the αBB approach).

71
 −1.03

−1.032

−1.034
Obj. fn. value

−1.036

−1.038

−1.04

Upper bound
Lower bound
−1.042
0 5 10 15 20 25 30 35 40
Iteration no.

Figure 19: Branch-and-Bound algorithm progress with the B-spline ap-

proach.

Since we are using an approximation, the optimal solution will have an

error. However, this error becomes smaller when we increase the number of
samples.

72
References
[1] C.S. Adjiman, S. Dallwig, C.A. Floudas, and A. Neumaier. A global
optimization method, αbb, for general twice-dierentiable constrained
{NLPs} — i. theoretical advances. Computers & Chemical Engineering,
22(9):1137  1158, 1998.

[2] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cam-

bridge University Press, New York, NY, USA, 2004.

[3] Jens Clausen. Branch and bound algorithms-principles and examples.

Department of Computer Science, University of Copenhagen, pages 1
30, 1999.

[4] Gaël Guennebaud, Benoît Jacob, et al. Eigen v3.

http://eigen.tuxfamily.org, 2010.

[5] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer,

1999.

[6] Andreas Wächter and Lorenz T. Biegler. On the implementation of

an interior-point lter line-search algorithm for large-scale nonlinear pro-
gramming. Mathematical Programming, 106(1):2557, 2006.

73