Tutorial: Operations Research in

Constraint Programming

John Hooker
Carnegie Mellon University
May 2009
Revised June 2009

CPAIOR tutorial
May 2009 Slide 1

Why Integrate OR and CP?

Complementary strengths
Computational advantages
Outline of the Tutorial

CPAIOR tutorial
May 2009 Slide 2

Complementary Strengths

CP:
Inference methods
Modeling
Exploits local structure
OR:
Relaxation methods
Duality theory
Exploits global structure

CPAIOR tutorial
May 2009 Slide 3

Lets bring them

together!

Computational Advantage of
Integrating CP and OR
Using CP + relaxation from MILP
Problem

Speedup

Focacci, Lodi,
Milano (1999)

Lesson
timetabling

2 to 50 times faster
than CP

Refalo (1999)

Piecewise linear
costs

2 to 200 times
faster than MILP

Hooker & Osorio

(1999)

Flow shop
scheduling, etc.

4 to 150 times
faster than MILP.

Thorsteinsson &
Ottosson (2001)

Product
configuration

30 to 40 times
faster than CP,
MILP

CPAIOR tutorial
May 2009 Slide 4

Computational Advantage of
Integrating CP and MILP
Using CP + relaxation from MILP
Problem

Speedup

Sellmann &
Fahle (2001)

Automatic
recording

1 to 10 times faster
than CP, MILP

Van Hoeve
(2001)

Stable set
problem

Better than CP in
less time

Bollapragada,
Ghattas &
Hooker (2001)

Structural design
(nonlinear)

Up to 600 times
faster than MILP.
2 problems: <6 min
vs >20 hrs for MILP

Beck & Refalo

(2003)

Scheduling with
earliness &
tardiness costs

Solved 67 of 90, CP
solved only 12

CPAIOR tutorial
May 2009 Slide 5

Computational Advantage of
Integrating CP and MILP
Using CP-based Branch and Price

Yunes, Moura &

de Souza (1999)

Easton,
Nemhauser &
Trick (2002)

CPAIOR tutorial
May 2009 Slide 6

Problem

Speedup

Urban transit
crew scheduling

Optimal schedule
for 210 trips, vs.
120 for traditional
branch and price

Traveling
tournament
scheduling

First to solve
8-team instance

Computational Advantage of
Integrating CP and MILP
Using CP/MILP Benders methods

Problem

Speedup

Jain &
Grossmann
(2001)

Min-cost planning
& scheduing

20 to 1000 times
faster than CP,
MILP

Thorsteinsson
(2001)

Min-cost planning
& scheduling

10 times faster
than Jain &
Grossmann

Timpe (2002)

Polypropylene
batch scheduling
at BASF

Solved previously
insoluble problem
in 10 min

CPAIOR tutorial
May 2009 Slide 7

Computational Advantage of
Integrating CP and MILP
Using CP/MILP Benders methods
Problem

Speedup

Benoist, Gaudin,
Rottembourg
(2002)

Call center scheduling

Solved twice as
many instances
as traditional
Benders

Hooker (2004)

Min-cost,
min-makespan
planning & cumulative
scheduling

100-1000 times
faster than CP,
MILP

Hooker (2005)

Min tardiness
planning & cumulative
scheduling

10-1000 times
faster than CP,
MILP

CPAIOR tutorial
May 2009 Slide 8

Software for Integrated Methods

ECLiPSe
Exchanges information between ECLiPSEe solver, Xpress-MP
OPL Studio
Combines CPLEX and ILOG CP Optimizer with script language
Mosel
Combines Xpress-MP, Xpress-Kalis with low-level modeling
BARON
Global optimization with relaxation + domain reduction
SIMPL
Full integration with high-level modeling (prototype)
SCIP
Combines MILP and CP-based propagation

CPAIOR tutorial
May 2009 Slide 9

Outline of the Tutorial

Why Integrate OR and CP?

A Glimpse at CP
Initial Example: Integrated Methods
CP Concepts
CP Filtering Algorithms
Linear Relaxation and CP
Mixed Integer/Linear Modeling
Cutting Planes
Lagrangean Relaxation and CP
Dynamic Programming in CP
CP-based Branch and Price
CP-based Benders Decomposition

CPAIOR tutorial
May 2009 Slide 10

Detailed Outline

Why Integrate OR and CP?

Complementary strengths
Computational advantages
Outline of the tutorial
A Glimpse at CP
Early successes
Advantages and disadvantages
Initial Example: Integrated Methods
Freight Transfer
Bounds Propagation
Cutting Planes
Branch-infer-and-relax Tree

CPAIOR tutorial
May 2009 Slide 11

Detailed Outline

CP Concepts
Consistency
Hyperarc Consistency
Modeling Examples
CP Filtering Algorithms
Element
Alldiff
Disjunctive Scheduling
Cumulative Scheduling
Linear Relaxation and CP
Why relax?
Algebraic Analysis of LP
Linear Programming Duality
LP-Based Domain Filtering
Example: Single-Vehicle Routing
Disjunctions of Linear Systems

CPAIOR tutorial
May 2009 Slide 12

Detailed Outline

Mixed Integer/Linear Modeling

MILP Representability
4.2 Disjunctive Modeling
4.3 Knapsack Modeling
Cutting Planes
0-1 Knapsack Cuts
Gomory Cuts
Mixed Integer Rounding Cuts
Example: Product Configuration
Lagrangean Relaxation and CP
Lagrangean Duality
Properties of the Lagrangean Dual
Example: Fast Linear Programming
Domain Filtering
Example: Continuous Global Optimization

CPAIOR tutorial
May 2009 Slide 13

Detailed Outline
Dynamic Programming in CP
Example: Capital Budgeting
Domain Filtering
Recursive Optimization
CP-based Branch and Price
Basic Idea
Example: Airline Crew Scheduling
CP-based Benders Decomposition
Benders Decomposition in the Abstract
Classical Benders Decomposition
Example: Machine Scheduling

CPAIOR tutorial
May 2009 Slide 14

Background Reading

This tutorial is based on:

J. N. Hooker, Integrated Methods for Optimization, Springer
(2007). Contains 295 exercises.
J. N. Hooker, Operations research methods in constraint
programming, in F. Rossi, P. van Beek and T. Walsh, eds.,
Handbook of Constraint Programming, Elsevier (2006), pp.
527-570.

CPAIOR tutorial
May 2009 Slide 15

A Glimpse at Constraint Programming

Early Successes
Advantages and Disadvantages

CPAIOR tutorial
May 2009 Slide 16

What is constraint programming?

It is a relatively new technology developed in the computer

science and artificial intelligence communities.
It has found an important role in scheduling, logistics and supply
chain management.

CPAIOR tutorial
May 2009 Slide 17

Early commercial successes

Circuit design (Siemens)

Real-time control
(Siemens, Xerox)

CPAIOR tutorial
May 2009 Slide 18

Container port scheduling

(Hong Kong and Singapore)

Applications
Job shop scheduling
Assembly line smoothing
and balancing
Cellular frequency
assignment
Nurse scheduling
Shift planning
Maintenance planning
Airline crew rostering and scheduling
Airport gate allocation and stand planning

CPAIOR tutorial
May 2009 Slide 19

Applications
Production scheduling
chemicals
aviation
oil refining
steel
lumber
photographic plates
tires
Transport scheduling (food,
nuclear fuel)
Warehouse management
Course timetabling
CPAIOR tutorial
May 2009 Slide 20

Advantages and Disadvantages

CP vs. Mathematical Programming
MP

CP

Numerical calculation

Logic processing

Relaxation

Inference (filtering,
constraint propagation)

Atomistic modeling
(linear inequalities)

High-level modeling
(global constraints)

Branching

Branching

Independence of model
and algorithm

Constraint-based
processing

CPAIOR tutorial
May 2009 Slide 21

Programming programming

In constraint programming:
programming = a form of computer programming
(constraint-based processing)

In mathematical programming:
programming = logistics planning (historically)

CPAIOR tutorial
May 2009 Slide 22

CP vs. MP

In mathematical programming, equations

(constraints) describe the problem but dont tell how to
solve it.

In constraint programming, each constraint invokes a

procedure that screens out unacceptable solutions.
Much as each line of a computer program invokes
an operation.

CPAIOR tutorial
May 2009 Slide 23

Advantages of CP
Better at sequencing and scheduling
where MP methods have weak relaxations.
Adding messy constraints makes the problem easier.
The more constraints, the better.
More powerful modeling language.
Global constraints lead to succinct models.
Constraints convey problem structure to the solver.
Better at highly-constrained problems
Misleading better when constraints propagate well, or
when constraints have few variables.
CPAIOR tutorial
May 2009 Slide 24

Disdvantages of CP
Weaker for continuous variables.
Due to lack of numerical techniques
May fail when constraints contain many variables.
These constraints dont propagate well.
Often not good for finding optimal solutions.
Due to lack of relaxation technology.
May not scale up
Discrete combinatorial methods
Software is not robust
Younger field
CPAIOR tutorial
May 2009 Slide 25

Obvious solution
Integrate CP and MP.

CPAIOR tutorial
May 2009 Slide 26

Trends
CP is better known in continental Europe, Asia.
Less known in North America, seen as threat to OR.
CP/MP integration is growing
Eclipse, Mozart, OPL Studio, SIMPL, SCIP, BARON
Heuristic methods increasingly important in CP
Discrete combinatorial methods
MP/CP/heuristics may become a single technology.

CPAIOR tutorial
May 2009 Slide 27

Initial Example: Integrated Methods

Freight Transfer
Bounds Propagation
Cutting Planes
Branch-infer-and-relax Tree

CPAIOR tutorial
May 2009 Slide 28

Example: Freight Transfer

Transport 42 tons of freight using 8 trucks, which come in

4 sizes

Truck
size

CPAIOR tutorial
May 2009 Slide 29

Number Capacity
available
(tons)

Cost
per
truck

90

60

50

40

Number of trucks of type 1

Knapsack
packing
constraint

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
xi {0,1,2,3}
Truck
type

CPAIOR tutorial
May 2009 Slide 30

Number Capacity
available
(tons)

Cost
per
truck

90

60

50

40

Knapsack
covering
constraint

Bounds propagation
min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
xi {0,1,2,3}

42 5 3 4 3 3 3
x1
=1

CPAIOR tutorial
May 2009 Slide 31

Bounds propagation

Reduced
domain

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
x1 {1,2,3}, x2 , x3 , x 4 {0,1,2,3}

42 5 3 4 3 3 3
x1
=1

CPAIOR tutorial
May 2009 Slide 32

Bounds consistency
Let {Lj, , Uj} be the domain of xj
A constraint set is bounds consistent if for each j :
xj = Lj in some feasible solution and
xj = Uj in some feasible solution.
Bounds consistency we will not set xj to any infeasible
values during branching.
Bounds propagation achieves bounds consistency for a
single inequality.
7x1 + 5x2 + 4x3 + 3x4 42 is bounds consistent when the
domains are x1 {1,2,3} and x2, x3, x4 {0,1,2,3}.
But not necessarily for a set of inequalities.
CPAIOR tutorial
May 2009 Slide 33

Bounds consistency

Bounds propagation may not achieve bounds consistency
for a set of constraints.

Consider set of inequalities

x1 + x2 1
x1 x2 0

with domains x1, x2 {0,1}, solutions (x1,x2) = (1,0), (1,1).

Bounds propagation has no effect on the domains.

But constraint set is not bounds consistent because x1 = 0
in no feasible solution.

CPAIOR tutorial
May 2009 Slide 34

Cutting Planes
Begin with continuous relaxation
min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
Replace domains
0 xi 3, x1 1
with bounds

This is a linear programming problem, which is easy to

solve.
Its optimal value provides a lower bound on optimal
value of original problem.

CPAIOR tutorial
May 2009 Slide 35

Cutting planes (valid inequalities)

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
0 xi 3, x1 1
We can create a tighter relaxation (larger minimum
value) with the addition of cutting planes.

CPAIOR tutorial
May 2009 Slide 36

Cutting planes (valid inequalities)

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
Cutting
plane
x1 + x2 + x3 + x 4 8
0 xi 3, x1 1

Continuous
relaxation

All feasible solutions of the

original problem satisfy a
cutting plane (i.e., it is valid).
But a cutting plane may
exclude (cut off) solutions of
the continuous relaxation.
Feasible solutions
CPAIOR tutorial
May 2009 Slide 37

Cutting planes (valid inequalities)

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
0 xi 3, x1 1
{1,2} is a packing
because 7x1 + 5x2 alone cannot satisfy the inequality,
even with x1 = x2 = 3.

CPAIOR tutorial
May 2009 Slide 38

Cutting planes (valid inequalities)

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
0 xi 3, x1 1
{1,2} is a packing
So,

4 x3 + 3 x 4 42 (7 3 + 5 3)

Knapsack cut

which implies

42 (7 3 + 5 3)
x3 + x 4
=2

max {4,3}
CPAIOR tutorial
May 2009 Slide 39

Cutting planes (valid inequalities)

Let xi have domain [Li,Ui] and let a 0.
In general, a packing P for ax a0 satisfies

a x
i P

a0 ai Ui
i P

and generates a knapsack cut

a0 ai U i
i P

xi

max {ai }
i P

i P

CPAIOR tutorial
May 2009 Slide 40

Cutting planes (valid inequalities)

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x 4 8
0 xi 3, x1 1
Maximal Packings

Knapsack cuts

{1,2}

x3 + x4 2

{1,3}

x2 + x4 2

{1,4}

x2 + x3 3

CPAIOR tutorial
May 2009 Slide 41

Knapsack cuts corresponding to nonmaximal

packings can be nonredundant.

Continuous relaxation with cuts

min 90 x1 + 60 x2 + 50 x3 + 40 x 4
7 x1 + 5 x2 + 4 x3 + 3 x 4 42
x1 + x2 + x3 + x4 8
0 xi 3, x1 1
x3 + x 4 2
x2 + x4 2
x 2 + x3 3

Knapsack cuts

Optimal value of 523.3 is a lower bound on optimal value

of original problem.

CPAIOR tutorial
May 2009 Slide 42

Branchinfer-andrelax tree
Propagate bounds
and solve
relaxation of
original problem.

CPAIOR tutorial
May 2009 Slide 43

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523

Branch-inferand-relax tree
Branch on a
variable with
nonintegral value
in the relaxation.

CPAIOR tutorial
May 2009 Slide 44

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523
x1 {1,2}

x1 = 3

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523

Branch-inferand-relax tree
Propagate bounds
and solve
relaxation.
Since relaxation
is infeasible,
backtrack.

CPAIOR tutorial
May 2009 Slide 45

x1 { 12 }
x2 { 23}
x3 { 123}
x4 { 123}
infeasible
relaxation

x1 {1,2}

x1 = 3

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523

Branch-inferand-relax tree
Propagate bounds
and solve
relaxation.
Branch on
nonintegral
variable.

CPAIOR tutorial
May 2009 Slide 46

x1 { 12 }
x2 { 23}
x3 { 123}
x4 { 123}
infeasible
relaxation

x1 {1,2}

x1 = 3

x1 {
3}
x2 {0123}
x3 {0123}
x4 {0123}
x = (3,2.6,2,0)
value = 526
x2 = 3
x2 {0,1,2}

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523

Branch-inferand-relax tree
Branch again.

x1 { 12 }
x2 { 23}
x3 { 123}
x4 { 123}
infeasible
relaxation

x1 {1,2}

x1 = 3

x1 {
3}
x2 {0123}
x3 {0123}
x4 {0123}
x = (3,2.6,2,0)
value = 526

x2 = 3
x1 {
3}
x2 {0,1,2}
x2 {012 }
x3 { 123}
x4 {0123}
x = (3,2,2,0)
value = 527
x3 {1,2}

CPAIOR tutorial
May 2009 Slide 47

x3 = 3

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523

Branch-inferand-relax tree
Solution of
relaxation
is integral and
therefore feasible
in the original
problem.
This becomes the
incumbent
solution.

CPAIOR tutorial
May 2009 Slide 48

x1 { 12 }
x2 { 23}
x3 { 123}
x4 { 123}
infeasible
relaxation

x1 {1,2}

x1 = 3

x1 {
3}
x2 {0123}
x3 {0123}
x4 {0123}
x = (3,2.6,2,0)
value = 526

x2 = 3
x1 {
3}
x2 {0,1,2}
x2 {012 }
x3 { 123}
x4 {0123}
x = (3,2,2,0)
value = 527
x1 {
3} x3 {1,2}
x2 { 12 }
x3 { 12 }
x4 { 123}
x = (3,2,2,1)
value = 530
feasible solution

x3 = 3

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523

Branch-inferand-relax tree
Solution is
nonintegral, but
we can backtrack
because value of
relaxation is
no better than
incumbent solution.

CPAIOR tutorial
May 2009 Slide 49

x1 { 12 }
x2 { 23}
x3 { 123}
x4 { 123}
infeasible
relaxation

x1 {1,2}

x1 = 3

x1 {
3}
x2 {0123}
x3 {0123}
x4 {0123}
x = (3,2.6,2,0)
value = 526

x2 = 3
x1 {
3}
x2 {0,1,2}
x2 {012 }
x3 { 123}
x4 {0123}
x = (3,2,2,0)
value = 527
x1 {
3} x3 {1,2}
x2 { 12 }
x3 { 12 }
x4 { 123}
x = (3,2,2,1)
value = 530
feasible solution

x3 = 3

x1 {
3}
x2 {012 }
x3 {
3}
x4 {012 }
x = (3,1,3,)
value = 530
backtrack
due to bound

x1 { 123}
x2 {0123}
x3 {0123}
x4 {0123}
x = (2,3,2,0)
value = 523

Branch-inferand-relax tree
Another feasible
solution found.
No better than
incumbent solution,
which is optimal
because search
has finished.

CPAIOR tutorial
May 2009 Slide 50

x1 { 12 }
x2 { 23}
x3 { 123}
x4 { 123}
infeasible
relaxation

x1 {1,2}

x1 = 3

x1 {
3}
x2 {0123}
x3 {0123}
x4 {0123}
x = (3,2.6,2,0)
value = 526

x2 = 3
x1 {
3}
x2 {0,1,2}
x2 {012 }
x3 { 123}
x4 {0123}
x = (3,2,2,0)
value = 527
x1 {
3} x3 {1,2}
x2 { 12 }
x3 { 12 }
x4 { 123}
x = (3,2,2,1)
value = 530
feasible solution

x3 = 3

x1 {
3}
x2 {012 }
x3 {
3}
x4 {012 }
x = (3,1,3,)
value = 530
backtrack
due to bound

x1 {
3}
x2 {
3}
x3 {012 }
x4 {012 }
x = (3,3,0,2)
value = 530
feasible solution

Two optimal solutions

x = (3,2,2,1)

x = (3,3,0,2)

CPAIOR tutorial
May 2009 Slide 51

Constraint Programming Concepts

Consistency
Generalized Arc Consistency
Modeling Examples

CPAIOR tutorial
May 2009 Slide 52

Consistency
A constraint set is consistent if every partial assignment to the
variables that violates no constraint is feasible.
i.e., can be extended to a feasible solution.
Consistency feasibility
Consistency means that any infeasible partial assignment is
explicitly ruled out by a constraint.
Fully consistent constraint sets can be solved without
backtracking.

CPAIOR tutorial
May 2009 Slide 53

Consistency
Consider the constraint set

x1 + x100 1
x1 x100 0
x j {0,1}

It is not consistent, because x1 = 0 violates no constraint

and yet is infeasible (no solution has x1 = 0).
Adding the constraint x1 = 1 makes the set consistent.

CPAIOR tutorial
May 2009 Slide 54

x1 + x100 1
x1 x100 1
other constraints
x j {0,1}

x1 = 0

x1 = 1

subtree with 299 nodes

but no feasible solution
By adding the constraint
x1 = 1, the left subtree is
eliminated

CPAIOR tutorial
May 2009 Slide 55

Generalized Arc Consistency (GAC)

Also known as hyperarc consistency.

A constraint set is GAC if every value in every variable
domain is part of some feasible solution.
That is, the domains are reduced as much as
possible.
If all constraints are binary (contain 2 variables),
GAC = arc consistency.
Domain reduction is CPs biggest engine.

CPAIOR tutorial
May 2009 Slide 56

Graph coloring problem that can be solved by arc

consistency maintenance alone. Color nodes with red,
green, blue with no two adjacent nodes having the same
color.

CPAIOR tutorial
May 2009 Slide 57

Graph coloring problem that can be solved by arc

consistency maintenance alone. Color nodes with red,
green, blue with no two adjacent nodes having the same
color.

CPAIOR tutorial
May 2009 Slide 58

Graph coloring problem that can be solved by arc

consistency maintenance alone. Color nodes with red,
green, blue with no two adjacent nodes having the same
color.

CPAIOR tutorial
May 2009 Slide 59

Graph coloring problem that can be solved by arc

consistency maintenance alone. Color nodes with red,
green, blue with no two adjacent nodes having the same
color.

CPAIOR tutorial
May 2009 Slide 60

Graph coloring problem that can be solved by arc

consistency maintenance alone. Color nodes with red,
green, blue with no two adjacent nodes having the same
color.

CPAIOR tutorial
May 2009 Slide 61

Graph coloring problem that can be solved by arc

consistency maintenance alone. Color nodes with red,
green, blue with no two adjacent nodes having the same
color.

CPAIOR tutorial
May 2009 Slide 62

Graph coloring problem that can be solved by arc

consistency maintenance alone. Color nodes with red,
green, blue with no two adjacent nodes having the same
color.

CPAIOR tutorial
May 2009 Slide 63

Modeling Examples with Global Constraints

Traveling Salesman
Traveling salesman problem:
Let cij = distance from city i to city j.
Find the shortest route that visits each of n cities exactly
once.

CPAIOR tutorial
May 2009 Slide 64

Popular 0-1 model

Let xij = 1 if city i immediately precedes city j, 0 otherwise

min

c x
ij

ij

ij

s.t.

x
x

ij

= 1, all j

ij

= 1, all i

x
i V jW

ij

1, all disjoint V ,W {1,, n}

xij {0,1}
Subtour elimination constraints

CPAIOR tutorial
May 2009 Slide 65

A CP model

Let yk = the kth city visited.

The model would be written in a specific constraint programming
language but would essentially say:
Variable indices

min

c
k

s.t.

y k y k +1

alldiff( y1,, y n )
y k {1,, n}

CPAIOR tutorial
May 2009 Slide 66

Global constraint

An alternate CP model

Let yk = the city visited after city k.

min

ky k

s.t.

circuit( y1,, y n )
y k {1,, n}

CPAIOR tutorial
May 2009 Slide 67

Hamiltonian circuit
constraint

Element constraint

The constraint cy 5 can be implemented:

z5

element ( y ,(c1,, cn ), z )

Assign z the yth

value in the list

The constraint xy 5 can be implemented

z5
element ( y ,( x1,, xn ), z )
(this is a slightly different constraint)

CPAIOR tutorial
May 2009 Slide 68

Add the
constraint
z = xy

Modeling example: Lot sizing and scheduling

Day:

Product

At most one product manufactured on each day.

Demands for each product on each day.
Minimize setup + holding cost.
CPAIOR tutorial
May 2009 Slide 69

min hit sit + qij ijt

t ,i
j t

s.t. si ,t 1 + xit = d it + sit , all i , t

Integer
programming
model
(Wolsey)

zit y it y i ,t 1, all i , t
zit y it , all i , t
zit 1 y i ,t 1, all i , t

ijt y i ,t 1 + y jt 1, all i , j , t
ijt y i ,t 1, all i , j , t
ijt y jt , all i , j , t
xit Cy it , all i , t
y it = 1, all t
i

y it , zit , ijt {0,1}

xit , sit 0
CPAIOR tutorial
May 2009 Slide 70

Many variables

CP model
Minimize holding and setup costs

min qy t 1y t + hi sit
t
i

s.t. si ,t 1 + xit = d it + sit , all i , t

0 xit C, sit 0, all i , t
( y t i ) ( xit = 0 ) , all i , t

CPAIOR tutorial
May 2009 Slide 71

Inventory balance
Production capacity

CP model
Minimize holding and setup costs
Variable indices

min qy t 1y t + hi sit
t
i

s.t. si ,t 1 + xit = d it + sit , all i , t

0 xit C, sit 0, all i , t
( y t i ) ( xit = 0 ) , all i , t

Inventory balance
Production capacity

Production level of product i in period t

Product manufactured in period t
CPAIOR tutorial
May 2009 Slide 72

Cumulative scheduling constraint

Used for resource-constrained scheduling.
Total resources consumed by jobs at any one time must
not exceed L.

cumulative ( (t1,, tn ),( p1,, pn ),(c1,, cn ), L )

Job start times
(variables)
Job processing times
Job resource
requirements

CPAIOR tutorial
May 2009 Slide 73

Cumulative scheduling constraint

Minimize makespan (no deadlines, all release times = 0):
L
4
1

resources

Min makespan = 8

3
time
L

min z
s.t.

cumulative ( (t1,, t5 ),(3,3,3,5,5),(3,3,3,2,2),7 )

z t1 + 3

z t5 + 2

CPAIOR tutorial
May 2009 Slide 74

Resources used
Processing times
Job start times

Modeling example: Ship loading

Will use ILOGs OPL Studio modeling language.

Example is from OPL manual.
The problem
Load 34 items on the ship in minimum time (min makespan)
Each item requires a certain time and certain number of
workers.
Total of 8 workers available.

CPAIOR tutorial
May 2009 Slide 75

Item

Duration

Labor

Item

Duration

Labor

18

19

20

21

22

23

24

25

26

10

27

11

28

12

29

13

30

14

31

15

32

16

33

17

34

CPAIOR tutorial
May 2009 Slide 76

Problem data

Precedence constraints
1 2,4
2 3
3 5,7
4 5
5 6
6 8
7 8
8 9
9 10
9 14
10 11
10 12

CPAIOR tutorial
May 2009 Slide 77

11 13
12 13
13 15,16
14 15
15 18
16 17
17 18
18 19
18 20,21
19 23
20 23
21 22

22 23
23 24
24 25
25 26,30,31,32
26 27
27 28
28 29
30 28
31 28
32 33
33 34

Use the cumulative scheduling constraint.

min z
s.t.

z t1 + 3, z t2 + 4, etc.
cumulative ( (t1,, t34 ),(3,4,,2),(4,4,,3),8 )
t2 t1 + 3, t 4 t1 + 3, etc.

CPAIOR tutorial
May 2009 Slide 78

OPL model
int capacity = 8;
int nbTasks = 34;
range Tasks 1..nbTasks;
int duration[Tasks] = [3,4,4,6,,2];
int totalDuration =
sum(t in Tasks) duration[t];
int demand[Tasks] = [4,4,3,4,,3];
struct Precedences {
int before;
int after;
}
{Precedences} setOfPrecedences = {
<1,2>, <1,4>, , <33,34> };

CPAIOR tutorial
May 2009 Slide 79

scheduleHorizon = totalDuration;
Activity a[t in Tasks](duration[t]);
DiscreteResource res(8);
Activity makespan(0);
minimize
makespan.end
subject to
forall(t in Tasks)
a[t] precedes makespan;
forall(p in setOfPrecedences)
a[p.before] precedes a[p.after];
forall(t in Tasks)
a[t] requires(demand[t]) res;
};

CPAIOR tutorial
May 2009 Slide 80

Modeling example: Production scheduling with

intermediate storage
Storage
Tanks
Capacity
C1
Manufacturing
Unit
Capacity
C2

Capacity
C3
CPAIOR tutorial
May 2009 Slide 81

Packing
Units

Filling of storage tank

Level

t
Filling starts
Packing starts
CPAIOR tutorial
May 2009 Slide 82

Need to enforce
capacity constraint
here only

t + (b/r)

Manufacturing rate

Batch size
Filling ends

u + (b/s)
Packing rate
Packing ends

Makespan

min T
s.t.

T uj +

bj
sj

, all j
Job release time

t j R j , all j
cumulative ( t ,v , e, m )
b
v i = ui + i t i , all i
si

m storage tanks
Job duration

si
bi 1
ri

+ si ui Ci , all i

b1

bn
cumulative u, ,, , e, p

sn
s1

uj tj 0
e = (1,,1)
CPAIOR tutorial
May 2009 Slide 83

Tank capacity

p packing units

Modeling example: Employee scheduling

Schedule four nurses in 8-hour shifts.
A nurse works at most one shift a day, at least 5 days a week.
Same schedule every week.
No shift staffed by more than two different nurses in a week.
A nurse cannot work different shifts on two consecutive days.
A nurse who works shift 2 or 3 must do so at least two days in
a row.

CPAIOR tutorial
May 2009 Slide 84

Two ways to view the problem

Assign nurses to shifts
Sun

Mon

Tue

Wed

Thu

Fri

Sat

Shift 1

Shift 2

Shift 3

Assign shifts to nurses

Sun

Mon

Tue

Wed

Thu

Fri

Sat

Nurse A

Nurse B

Nurse C

Nurse D

CPAIOR tutorial
May 2009 Slide 85

0 = day off

Use both formulations in the same model!

First, assign nurses to shifts.
Let wsd = nurse assigned to shift s on day d

alldiff(w1d ,w 2d ,w 3d ), all d

The variables w1d, w2d,

w3d take different values
That is, schedule 3
different nurses on each
day

CPAIOR tutorial
May 2009 Slide 86

Use both formulations in the same model!

First, assign nurses to shifts.
Let wsd = nurse assigned to shift s on day d

alldiff(w1d ,w 2d ,w 3d ), all d

cardinality (w | ( A, B,C, D ),(5,5,5,5),(6,6,6,6) )

A occurs at least 5 and at most 6
times in the array w, and similarly
for B, C, D.
That is, each nurse works at least
5 and at most 6 days a week

CPAIOR tutorial
May 2009 Slide 87

Use both formulations in the same model!

First, assign nurses to shifts.
Let wsd = nurse assigned to shift s on day d

alldiff (w1d ,w 2d ,w 3d ) , all d

cardinality (w | ( A, B,C, D ),(5,5,5,5),(6,6,6,6) )

nvalues (w s,Sun ,...,w s,Sat | 1,2 ) , all s
The variables ws,Sun, , ws,Sat take
at least 1 and at most 2 different
values.
That is, at least 1 and at most 2
nurses work any given shift.
CPAIOR tutorial
May 2009 Slide 88

Remaining constraints are not easily expressed in this

notation.
So, assign shifts to nurses.
Let yid = shift assigned to nurse i on day d

alldiff ( y1d , y 2d , y 3d ) , all d

Assign a different nurse to each
shift on each day.
This constraint is redundant of
previous constraints, but
redundant constraints speed
solution.
CPAIOR tutorial
May 2009 Slide 89

Remaining constraints are not easily expressed in this

notation.
So, assign shifts to nurses.
Let yid = shift assigned to nurse i on day d

alldiff ( y1d , y 2d , y 3d ) , all d

stretch ( y i ,Sun ,, y i ,Sat | (2,3),(2,2),(6,6), P ) , all i
Every stretch of 2s has length between 2 and 6.
Every stretch of 3s has length between 2 and 6.
So a nurse who works shift 2 or 3 must do so at least
two days in a row.
CPAIOR tutorial
May 2009 Slide 90

Remaining constraints are not easily expressed in this

notation.
So, assign shifts to nurses.
Let yid = shift assigned to nurse i on day d

alldiff ( y1d , y 2d , y 3d ) , all d

stretch ( y i ,Sun ,, y i ,Sat | (2,3),(2,2),(6,6), P ) , all i
Here P = {(s,0),(0,s) | s = 1,2,3}
Whenever a stretch of as immediately precedes a stretch of bs,
(a,b) must be one of the pairs in P.
So a nurse cannot switch shifts without taking at least one day off.
CPAIOR tutorial
May 2009 Slide 91

Now we must connect the wsd variables to the yid variables.

Use channeling constraints:

w y id d = i , all i , d
y w sd d = s, all s, d

Channeling constraints increase propagation and make the

problem easier to solve.

CPAIOR tutorial
May 2009 Slide 92

The complete model is:

alldiff (w1d ,w 2d ,w 3d ) , all d

cardinality (w | ( A, B,C, D ),(5,5,5,5),(6,6,6,6) )

nvalues (w s,Sun ,...,w s,Sat | 1,2 ) , all s
alldiff ( y1d , y 2d , y 3d ) , all d
stretch ( y i ,Sun ,, y i ,Sat | (2,3),(2,2),(6,6), P ) , all i

w y id d = i , all i , d
y w sd d = s, all s, d
CPAIOR tutorial
May 2009 Slide 93

CP Filtering Algorithms
Element
Alldiff
Disjunctive Scheduling
Cumulative Scheduling

CPAIOR tutorial
May 2009 Slide 94

Filtering for element

element ( y ,( x1,, xn ), z )
Variable domains can be easily filtered to maintain GAC.

Dz Dz

j Dy

Domain of z

xj

Dz if Dy = { j }
Dx j

D
otherwise
x j

CPAIOR tutorial
May 2009 Slide 95

Dy Dy j | Dz Dx j

Filtering for element

Example...

element ( y ,( x1, x2 , x3 , x4 ), z )

The initial domains are:

The reduced domains are:

Dz = {20,30,60,80,90}
Dy = {1,3,4}

Dz = {80,90}
Dy = {3}

Dx2 = {10,20}

Dx2 = {10,20}

Dx1 = {10,50}

Dx3 = {40,50,80,90}
Dx4 = {40,50,70}

CPAIOR tutorial
May 2009 Slide 96

Dx1 = {10,50}

Dx3 = {80,90}

Dx4 = {40,50,70}

Filtering for alldiff

alldiff ( y1,, y n )
Domains can be filtered with an algorithm based on maximum
cardinality bipartite matching and a theorem of Berge.
It is a special case of optimality conditions for max flow.

CPAIOR tutorial
May 2009 Slide 97

Filtering for alldiff

Consider the domains

y1 {1}

y 2 {2,3,5}
y 3 {1,2,3,5}
y 4 {1,5}
y 5 {1,2,3,4,5,6}

CPAIOR tutorial
May 2009 Slide 98

Indicate domains with edges

y1

1
2

y2

y3
4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 99

Indicate domains with edges

y1

1
2

y2

y3
4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 100

Find maximum cardinality bipartite

matching.

Indicate domains with edges

y1

1
2

y2

y3
4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 101

Find maximum cardinality bipartite

matching.

Indicate domains with edges

y1

1
2

y2

y3
4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 102

Find maximum cardinality bipartite

matching.
Mark edges in alternating paths
that start at an uncovered vertex.

Indicate domains with edges

y1

1
2

y2

y3
4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 103

Find maximum cardinality bipartite

matching.
Mark edges in alternating paths
that start at an uncovered vertex.

Indicate domains with edges

y1

1
2

y2

y3

Find maximum cardinality bipartite

matching.
Mark edges in alternating paths
that start at an uncovered vertex.

4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 104

Mark edges in alternating cycles.

Indicate domains with edges

y1

1
2

y2

y3

Find maximum cardinality bipartite

matching.
Mark edges in alternating paths
that start at an uncovered vertex.

4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 105

Mark edges in alternating cycles.

Remove unmarked edges not in

matching.

Indicate domains with edges

y1

1
2

y2

y3

Find maximum cardinality bipartite

matching.
Mark edges in alternating paths
that start at an uncovered vertex.

4
y4
5
y5

CPAIOR tutorial
May 2009 Slide 106

Mark edges in alternating cycles.

Remove unmarked edges not in

matching.

Filtering for alldiff

Domains have been filtered:

y1 {1}

y 2 {2,3,5}
y 3 {1,2,3,5}
y 4 {1,5}
y 5 {1,2,3,4,5,6}
GAC achieved.

CPAIOR tutorial
May 2009 Slide 107

y1 {1}

y 2 {2,3}
y 3 {2,3}
y 4 {5}
y 5 {4,6}

Disjunctive scheduling
Consider a disjunctive scheduling constraint:

disjunctive ( (s1, s2 , s3 , s5 ),( p1, p2 , p3 , p5 ) )

Start time variables

CPAIOR tutorial
May 2009 Slide 108

Edge finding for disjunctive scheduling

Consider a disjunctive scheduling constraint:

disjunctive ( (s1, s2 , s3 , s5 ),( p1, p2 , p3 , p5 ) )

Processing times

CPAIOR tutorial
May 2009 Slide 109

Edge finding for disjunctive scheduling

Consider a disjunctive scheduling constraint:

disjunctive ( (s1, s2 , s3 , s5 ),( p1, p2 , p3 , p5 ) )

Variable domains defined by time
windows and processing times
s1 [0,10 1]
s2 [0,10 3]
s3 [2,7 3]
s5 [4,7 2]

CPAIOR tutorial
May 2009 Slide 110

Edge finding for disjunctive scheduling

Consider a disjunctive scheduling constraint:

disjunctive ( (s1, s2 , s3 , s5 ),( p1, p2 , p3 , p5 ) )

A feasible (min makespan) solution:

Time window
CPAIOR tutorial
May 2009 Slide 111

Edge finding for disjunctive scheduling

But lets reduce 2 of the deadlines to 9:

CPAIOR tutorial
May 2009 Slide 112

Edge finding for disjunctive scheduling

But lets reduce 2 of the deadlines to 9:
We will use edge finding
to prove that there is no
feasible schedule.

CPAIOR tutorial
May 2009 Slide 113

Edge finding for disjunctive scheduling

We can deduce that job 2 must precede jobs 3 and 5: 2 {3,5}
Because if job 2 is not first, there is not enough time for all 3
jobs within the time windows:

L{2,3,5} E{3,5} < p{2,3,5}

CPAIOR tutorial
May 2009 Slide 114

E{3,5}

7<3+3+2

L{2,3,5}

Edge finding for disjunctive scheduling

We can deduce that job 2 must precede jobs 3 and 5: 2 {3,5}
Because if job 2 is not first, there is not enough time for all 3
jobs within the time windows:

L{2,3,5} E{3,5} < p{2,3,5}

Latest deadline

CPAIOR tutorial
May 2009 Slide 115

E{3,5}

7<3+3+2

L{2,3,5}

Edge finding for disjunctive scheduling

We can deduce that job 2 must precede jobs 3 and 5: 2 {3,5}
Because if job 2 is not first, there is not enough time for all 3
jobs within the time windows:

L{2,3,5} E{3,5} < p{2,3,5}

Earliest release time

CPAIOR tutorial
May 2009 Slide 116

E{3,5}

7<3+3+2

L{2,3,5}

Edge finding for disjunctive scheduling

We can deduce that job 2 must precede jobs 3 and 5: 2 {3,5}
Because if job 2 is not first, there is not enough time for all 3
jobs within the time windows:

L{2,3,5} E{3,5} < p{2,3,5}

Total processing time

CPAIOR tutorial
May 2009 Slide 117

E{3,5}

7<3+3+2

L{2,3,5}

Edge finding for disjunctive scheduling

We can deduce that job 2 must precede jobs 3 and 5: 2 {3,5}
So we can tighten deadline of job 2 to minimum of

L{3} p{3} = 4

L{5} p{5} = 5

L{3,5} p{3,5} = 2

Since time window of job 2 is now too narrow, there is no

feasible schedule.

CPAIOR tutorial
May 2009 Slide 118

E{3,5}

7<3+3+2

L{2,3,5}

Edge finding for disjunctive scheduling

In general, we can deduce that job k must precede all the jobs
in set J: k J
If there is not enough time for all the jobs after the earliest
release time of the jobs in J

LJ { k } EJ < pJ { k }

CPAIOR tutorial
May 2009 Slide 119

L{2,3,5} E{3,5} < p{2,3,5}

Edge finding for disjunctive scheduling

In general, we can deduce that job k must precede all the jobs
in set J: k J
If there is not enough time for all the jobs after the earliest
release time of the jobs in J

LJ { k } EJ < pJ { k }

L{2,3,5} E{3,5} < p{2,3,5}

Now we can tighten the deadline for job k to:

min {LJ pJ }
J J

CPAIOR tutorial
May 2009 Slide 120

L{3,5} p{3,5} = 2

Edge finding for disjunctive scheduling

There is a symmetric rule:

k J

If there is not enough time for all the jobs before the latest
deadline of the jobs in J:

LJ EJ { k } < pJ { k }
Now we can tighten the release date for job k to:

max {EJ + pJ }
J J

CPAIOR tutorial
May 2009 Slide 121

Edge finding for disjunctive scheduling

Problem: how can we avoid enumerating all subsets J of jobs

to find edges?

LJ { k } EJ < pJ { k }
and all subsets J of J to tighten the bounds?

min {LJ pJ }
J J

CPAIOR tutorial
May 2009 Slide 122

Edge finding for disjunctive scheduling

Key result: We only have to consider sets J whose time

windows lie within some interval.

min {LJ pJ }
J J

e.g., J = {3,5}

CPAIOR tutorial
May 2009 Slide 123

Edge finding for disjunctive scheduling

Key result: We only have to consider sets J whose time

windows lie within some interval.

min {LJ pJ }
J J

e.g., J = {3,5}

Removing a job from those within an interval only weakens the

test
LJ { k } EJ < pJ { k }
CPAIOR tutorial
May 2009 Slide 124

There are a polynomial number of intervals

defined by release times and deadlines.

Edge finding for disjunctive scheduling

Key result: We only have to consider sets J whose time

windows lie within some interval.

min {LJ pJ }
J J

e.g., J = {3,5}

Note: Edge finding does not achieve bounds consistency,

which is an NP-hard problem.

CPAIOR tutorial
May 2009 Slide 125

Edge finding for disjunctive scheduling

One O(n2) algorithm is based on the Jackson pre-emptive
schedule (JPS). Using a different example, the JPS is:

CPAIOR tutorial
May 2009 Slide 126

Edge finding for disjunctive scheduling

One O(n2) algorithm is based on the Jackson pre-emptive
schedule (JPS). Using a different example, the JPS is:

Jobs unfinished at time Ei in JPS

For each job i
Scan jobs k Ji in decreasing order of Lk
Total remaining processing
Select first k for which Lk Ei < pi + pJik
time in JPS of jobs in Jik
Conclude that i Jik
Jobs j i in Ji with Lj Lk
Update Ei to JPS(i , k )
CPAIOR tutorial
May 2009 Slide 127

Latest completion time in JPS of jobs in Jik

Not-first/not-last rules
We can deduce that job 4 cannot precede jobs 1 and 2:

( 4 {1,2} )

Because if job 4 is first, there is too little time to complete the

jobs before the later deadline of jobs 1 and 2:

L{1,2} E 4 < p1 + p2 + p4

E4
CPAIOR tutorial
May 2009 Slide 128

6<1+3+3

L{1,2}

Not-first/not-last rules
We can deduce that job 4 cannot precede jobs 1 and 2:

( 4 {1,2} )

Now we can tighten the release time of job 4 to minimum of:

E1 + p1 = 3

E4
CPAIOR tutorial
May 2009 Slide 129

6<1+3+3

E2 + p2 = 4

L{1,2}

Not-first/not-last rules
In general, we can deduce that job k cannot precede all the
jobs in J:
(k J )
if there is too little time after release time of job k to complete
all jobs before the latest deadline in J:

LJ Ek < pJ
Now we can update Ei to

min {E j + p j }
j J

CPAIOR tutorial
May 2009 Slide 130

Not-first/not-last rules
In general, we can deduce that job k cannot precede all the
jobs in J:
(k J )
if there is too little time after release time of job k to complete
all jobs before the latest deadline in J:

LJ Ek < pJ
Now we can update Ei to

min {E j + p j }
j J

There is a symmetric not-last rule.

The rules can be applied in polynomial time, although an
efficient algorithm is quite complicated.

CPAIOR tutorial
May 2009 Slide 131

Cumulative scheduling
Consider a cumulative scheduling constraint:

cumulative ( (s1, s2 , s3 ),( p1, p2 , p3 ),(c1, c2 , c3 ),C )

A feasible solution:

CPAIOR tutorial
May 2009 Slide 132

Edge finding for cumulative scheduling

We can deduce that job 3 must finish after the others finish: 3 > {1,2}
Because the total energy required exceeds the area between
the earliest release time and the later deadline of jobs 1,2:

e3 + e{1,2} > C L{1,2} E {1,2,3}

CPAIOR tutorial
May 2009 Slide 133

Edge finding for cumulative scheduling

We can deduce that job 3 must finish after the others finish: 3 > {1,2}
Because the total energy required exceeds the area between
the earliest release time and the later deadline of jobs 1,2:

e3 + e{1,2} > C L{1,2} E {1,2,3}

Total energy
required = 22

9
8
5

CPAIOR tutorial
May 2009 Slide 134

Edge finding for cumulative scheduling

We can deduce that job 3 must finish after the others finish: 3 > {1,2}
Because the total energy required exceeds the area between
the earliest release time and the later deadline of jobs 1,2:

e3 + e{1,2} > C L{1,2} E {1,2,3}

Total energy
required = 22

9
8

Area available
= 20
5

CPAIOR tutorial
May 2009 Slide 135

Edge finding for cumulative scheduling

We can deduce that job 3 must finish after the others finish: 3 > {1,2}
We can update the release time of job 3 to

E{1,2} +

eJ (C c3 )(L{1,2} E{1,2} )
c3

Energy available
for jobs 1,2 if
space is left for job
3 to start anytime
= 10
10

CPAIOR tutorial
May 2009 Slide 136

Edge finding for cumulative scheduling

We can deduce that job 3 must finish after the others finish: 3 > {1,2}
We can update the release time of job 3 to

E{1,2} +

eJ (C c3 )(L{1,2} E{1,2} )

Energy available
for jobs 1,2 if
space is left for job
3 to start anytime
= 10
Excess energy
required by jobs
1,2 = 4
CPAIOR tutorial
May 2009 Slide 137

c3

10

Edge finding for cumulative scheduling

We can deduce that job 3 must finish after the others finish: 3 > {1,2}
We can update the release time of job 3 to

E{1,2} +

eJ (C c3 )(L{1,2} E{1,2} )

Energy available
for jobs 1,2 if
space is left for job
3 to start anytime
= 10
Excess energy
required by jobs
1,2 = 4
CPAIOR tutorial
May 2009 Slide 138

c3

10
E3

Move up job 3
release time
4/2 = 2 units
beyond E{1,2}

Edge finding for cumulative scheduling

In general, if

eJ {k } > C LJ EJ { k }

then k > J, and update Ek to

eJ (C ck )(LJ EJ )
max
E J +

J J
c

e (C c )( L E )>0
J

In general, if

eJ {k } > C LJ { k } EJ

then k < J, and update Lk to

eJ (C ck )(LJ EJ )
min
LJ

J J
c

eJ (C ck )( LJ EJ )>0
CPAIOR tutorial
May 2009 Slide 139

Edge finding for cumulative scheduling

There is an O(n2) algorithm that finds all applications of the
edge finding rules.

CPAIOR tutorial
May 2009 Slide 140

Other propagation rules for cumulative

scheduling
Extended edge finding.
Timetabling.
Not-first/not-last rules.
Energetic reasoning.

CPAIOR tutorial
May 2009 Slide 141

Linear Relaxation
Why Relax?
Algebraic Analysis of LP
Linear Programming Duality
LP-Based Domain Filtering
Example: Single-Vehicle Routing
Disjunctions of Linear Systems
CPAIOR tutorial
May 2009 Slide 142

Why Relax?
Solving a relaxation of a problem can:

Tighten variable bounds.

Possibly solve original problem.
Guide the search in a promising direction.
Filter domains using reduced costs or Lagrange multipliers.
Prune the search tree using a bound on the optimal value.
Provide a more global view, because a single OR relaxation
can pool relaxations of several constraints.

CPAIOR tutorial
May 2009 Slide 143

Some OR models that can provide relaxations:

Linear programming (LP).

Mixed integer linear programming (MILP)
Can itself be relaxed as an LP.
LP relaxation can be strengthened with cutting planes.
Lagrangean relaxation.
Specialized relaxations.
For particular problem classes.
For global constraints.

CPAIOR tutorial
May 2009 Slide 144

Motivation
Linear programming is remarkably versatile for representing
real-world problems.
LP is by far the most widely used tool for relaxation.
LP relaxations can be strengthened by cutting planes.
- Based on polyhedral analysis.
LP has an elegant and powerful duality theory.
- Useful for domain filtering, and much else.
The LP problem is extremely well solved.

CPAIOR tutorial
May 2009 Slide 145

Algebraic Analysis of LP
An example

min 4 x1 + 7 x2
2 x1 + 3 x2 6
2 x1 + x2 4

2x1 + x2 4
Optimal solution
x = (3,0)

x1, x2 0
4x1 + 7x2 = 12

2x1 + 3x2 6

CPAIOR tutorial
May 2009 Slide 146

Algebraic Analysis of LP
Rewrite
min 4 x1 + 7 x2

as

2 x1 + 3 x2 6
2 x1 + x2 4
x1, x2 0
In general an LP has the form

min 4 x1 + 7 x2
2 x1 + 3 x2 x3 = 6
2 x1 + x2 x4 = 4
x1, x2 , x3 , x4 0

min cx
Ax = b
x0

CPAIOR tutorial
May 2009 Slide 147

Algebraic analysis of LP
Write

min cx

as

BxB + NxN = b
xB , xN 0

Ax = b
x0
m n matrix

min cB xB + cN xN

Basic
variables

Nonbasic
variables

where

A = [B N ]
Any set of
m linearly
independent
columns of A.
These form a
basis for the
space spanned
by the columns.

CPAIOR tutorial
May 2009 Slide 148

Algebraic analysis of LP
Write

min cx

as

Ax = b
x0

min cB xB + cN xN

where

BxB + NxN = b
xB , xN 0

Solve constraint equation for xB:

xB = B 1b B 1NxN

All solutions can be obtained by setting xN to some value.

The solution is basic if xN = 0.
It is a basic feasible solution if xN = 0 and xB 0.

CPAIOR tutorial
May 2009 Slide 149

A = [B N ]

Example

x2

min 4 x1 + 7 x2
2 x1 + 3 x2 x3 = 6

= basic feasible
solution

x2, x3 basic
2x1 + x2 4

2 x1 + x2 x4 = 4
x1, x2 , x3 , x4 0
x2, x4 basic

x1, x2 basic
2x1 + 3x2 6

x3, x4 basic
CPAIOR tutorial
May 2009 Slide 150

x1, x4 basic

x1, x3 basic

x1

Algebraic analysis of LP
Write

min cx

as

Ax = b
x0

min cB xB + cN xN

where

A = [B N ]

BxB + NxN = b
xB , xN 0

Solve constraint equation for xB:

xB = B 1b B 1NxN

Express cost in terms of nonbasic variables:

cB B 1b + (cN cB B 1N )xN
Vector of reduced costs
CPAIOR tutorial
May 2009 Slide 151

Since xN 0,
basic solution (xB,0)
is optimal if
reduced costs are
nonnegative.

Example

x2

min 4 x1 + 7 x2
2x1 + 3 x2 x3 = 6
2x1 + x2 x4 = 4
x1, x2 , x3 , x4 0

Consider this
basic feasible
solution

x1, x4 basic
x1
CPAIOR tutorial
May 2009 Slide 152

Example

Write

min 4 x1 + 7 x2
2x1 + 3 x2 x3 = 6
2x1 + x2 x4 = 4
x1, x2 , x3 , x4 0

CPAIOR tutorial
May 2009 Slide 153

as

cBxB

c Nx N

x2
x1
min [ 4 0 ] + [7 0]
x4
x3
2 0 x1 3 1 x1 6
+
=
BxB

2 1 x 4 1 0 x 4 4
NxN
b
x1 x1 0
x , x 0
4 4

Example

cBxB

c Nx N

x2
x1
min [ 4 0 ] + [7 0]
x4
x3
2 0 x1 3 1 x1 6
+
=
BxB

2 1 x 4 1 0 x 4 4
b
NxN
x1 x1 0
x , x 0
4 4

CPAIOR tutorial
May 2009 Slide 154

Example
Basic solution is

xB = B 1b B 1NxN = B 1b
cBxB

c Nx N

x2
x1
min [ 4 0 ] + [7 0]
x4
x3
2 0 x1 3 1 x1 6
+
=
BxB

2 1 x 4 1 0 x 4 4
b
NxN
x1 x1 0
x , x 0
4 4

x1 1/ 2 0 6 3
= =
=

x4 1 1 4 2

x2

x1, x4 basic

CPAIOR tutorial
May 2009 Slide 155

x1

Example
Basic solution is

xB = B 1b B 1NxN = B 1b
cBxB

c Nx N

x2
x1
min [ 4 0 ] + [7 0]
x4
x3
2 0 x1 3 1 x1 6
+
=
BxB

2 1 x 4 1 0 x 4 4
NxN
x1 x1 0
x , x 0
4 4

CPAIOR tutorial
May 2009 Slide 156

x1 1/ 2 0 6 3
= =
=

x4 1 1 4 2

Reduced costs are

cN cB B 1N
1/ 2 0 3 1
= [7 0] [ 4 0]

1 1 1 0
= [1 2] [0 0]
Solution is
optimal

Linear Programming Duality

An LP can be viewed as an inference problem

min cx
Ax b
x0

max v
x 0

Ax b cx v
implies

Dual problem: Find the tightest lower bound on the

objective function that is implied by the constraints.

CPAIOR tutorial
May 2009 Slide 157

An LP can be viewed as an inference problem

min cx

max v

Ax b
x0

x 0

Ax b cx v

That is, some surrogate

(nonnegative linear
combination) of
Ax b dominates cx v

From Farkas Lemma: If Ax b, x 0 is feasible,

x 0

Ax b cx v

iff

Ax b dominates cx v
for some 0
A c and b v

CPAIOR tutorial
May 2009 Slide 158

An LP can be viewed as an inference problem

min cx

max v

Ax b
x0

x 0

Ax b cx v

max b
A c
0

This is the
classical
LP dual

From Farkas Lemma: If Ax b, x 0 is feasible,

x 0

Ax b cx v

iff

Ax b dominates cx v
for some 0
A c and b v

CPAIOR tutorial
May 2009 Slide 159

This equality is called strong duality.

min cx
Ax b
x0

max b
A c
0

This is the
classical
LP dual

If Ax b, x 0 is feasible

Note that the dual of the dual is the primal

(i.e., the original LP).

CPAIOR tutorial
May 2009 Slide 160

Example
Primal

min 4 x1 + 7 x2

Dual

2 x1 + 3 x2 6

(1 )

2 x1 + x2 4
x1, x2 0

(1 )

max 61 + 42
= 12
21 + 22 4
( x1 )
31 + 2 7
( x2 )
1, 2 0

A dual solution is (1,2) = (2,0)

2 x1 + 3 x2 6 (1 = 2)
2 x1 + x2 4 (2 = 0)

Dual multipliers

4 x1 + 6 x2 12

Surrogate

dominates
CPAIOR tutorial
May 2009 Slide 161

4 x1 + 7 x2 12

Tightest bound on cost

Weak Duality

If x* is feasible in the
primal problem

and * is feasible in the

dual problem

min cx

max b

Ax b
x0

A c
0

then cx* *b.

This is because
cx* *Ax* *b
* is dual
feasible
and x* 0

CPAIOR tutorial
May 2009 Slide 162

x* is primal
feasible
and * 0

Dual multipliers as marginal costs

Suppose we perturb the RHS of an LP
(i.e., change the requirement levels):

min cx
Ax b + b
x0

The dual of the perturbed LP has the

same constraints at the original LP:

max ( b + b )
A c
0

So an optimal solution * of the original dual is feasible in the

perturbed dual.

CPAIOR tutorial
May 2009 Slide 163

Dual multipliers as marginal costs

Suppose we perturb the RHS of an LP
(i.e., change the requirement levels):

min cx
Ax b + b
x0

By weak duality, the optimal value of the perturbed LP is at least

*(b + b) = *b + *b.
Optimal value of original LP, by strong duality.
So i* is a lower bound on the marginal cost of increasing the
i-th requirement by one unit (bi = 1).
If i* > 0, the i-th constraint must be tight (complementary slackness).
CPAIOR tutorial
May 2009 Slide 164

Dual of an LP in equality form

Primal

Dual

min cB xB + cN xN

max b
B cB
( xB )
( xB )
N cN
unrestricted

BxB + NxN = b
xB , xN 0

CPAIOR tutorial
May 2009 Slide 165

( )

Dual of an LP in equality form

Primal

Dual

min cB xB + cN xN

max b
B cB
( xB )
( xB )
N cN
unrestricted

BxB + NxN = b
xB , xN 0

( )

Recall that reduced cost vector is

cN cB B 1N = cN N

this solves the dual

if (xB,0) solves the primal

CPAIOR tutorial
May 2009 Slide 166

Dual of an LP in equality form

Primal
min cB xB + cN xN
BxB + NxN = b
xB , xN 0

Dual
max b
B cB
( xB )
( xB )
N cN
unrestricted

( )

Recall that reduced cost vector is

Check:

B = cB B B = cB
N = cB B 1N cN
1

cN cB B 1N = cN N

this solves the dual

if (xB,0) solves the primal
Because reduced cost is nonnegative
at optimal solution (xB,0).

CPAIOR tutorial
May 2009 Slide 167

Dual of an LP in equality form

Primal
min cB xB + cN xN
BxB + NxN = b
xB , xN 0

Dual
( )

Recall that reduced cost vector is

max b
B cB
( xB )
( xB )
N cN
unrestricted

cN cB B 1N = cN N

this solves the dual

if (xB,0) solves the primal
In the example,

1/ 2 0
= cB B = [ 4 0 ]
= [ 2 0]

1 1
1

CPAIOR tutorial
May 2009 Slide 168

Dual of an LP in equality form

Primal
min cB xB + cN xN
BxB + NxN = b
xB , xN 0

Dual
( )

Recall that reduced cost vector is

max b
B cB
( xB )
( xB )
N cN
unrestricted

cN cB B 1N = cN N

Note that the reduced cost of an individual variable xj is

r j = c j Aj

Column j of A
CPAIOR tutorial
May 2009 Slide 169

LP-based Domain Filtering

min cx

Let

Ax b

be an LP relaxation of a CP problem.

x0

 One way to filter the domain of xj is to minimize and maximize xj

subject to Ax b, x 0.
- This is time consuming.
 A faster method is to use dual multipliers to derive valid
inequalities.
- A special case of this method uses reduced costs to bound or
fix variables.
- Reduced-cost variable fixing is a widely used technique in OR.
CPAIOR tutorial
May 2009 Slide 170

Suppose:
min cx
Ax b
x0

has optimal solution x*, optimal value v*, and

optimal dual solution *.

and i* > 0, which means the i-th constraint is tight

(complementary slackness);
and the LP is a relaxation of a CP problem;
and we have a feasible solution of the CP problem with value
U, so that U is an upper bound on the optimal value.

CPAIOR tutorial
May 2009 Slide 171

min cx

Supposing

Ax b
x0

has optimal solution x*, optimal value v*, and

optimal dual solution *:

If x were to change to a value other than x*, the LHS of i-th constraint
Aix bi would change by some amount bi.
Since the constraint is tight, this would increase the optimal value
as much as changing the constraint to Aix bi + bi.
So it would increase the optimal value at least i*bi.

CPAIOR tutorial
May 2009 Slide 172

min cx

Supposing

Ax b
x0

has optimal solution x*, optimal value v*, and

optimal dual solution *:

We have found: a change in x that changes Aix by bi increases

the optimal value of LP at least i*bi.
Since

optimal value of the LP optimal value of the CP U,

we have i*bi U v*, or
U v*

bi

CPAIOR tutorial
May 2009 Slide 173

i*

min cx

Supposing

Ax b
x0

has optimal solution x*, optimal value v*, and

optimal dual solution *:

We have found: a change in x that changes Aix by bi increases

the optimal value of LP at least i*bi.
Since

optimal value of the LP optimal value of the CP U,

we have i*bi U v*, or
U v*

bi

i*

Since bi = Aix Aix* = Aix bi, this implies the inequality

Ai x bi +
CPAIOR tutorial
May 2009 Slide 174

U v*

i*

which can be propagated.

Example
min 4 x1 + 7 x2
2 x1 + 3 x2 6 (1 = 2)
2 x1 + x2 4
(1 = 0)
x1, x2 0

Suppose we have a feasible solution

of the original CP with value U = 13.

Since the first constraint is tight, we can propagate

the inequality

A1x b1 +

or

CPAIOR tutorial
May 2009 Slide 175

2 x1 + 3 x2 6 +

U v*

1*

13 12
= 6.5
2

Reduced-cost domain filtering

Suppose xj* = 0, which means the constraint xj 0 is tight.

The inequality

Ai x bi +

U v*

*
i

becomes

U v*
xj
rj

The dual multiplier for xj 0 is the reduced cost

rj of xj, because increasing xj (currently 0) by 1
increases optimal cost by rj.
Similar reasoning can bound a variable below when it is at its
upper bound.
CPAIOR tutorial
May 2009 Slide 176

Example
min 4 x1 + 7 x2
2 x1 + 3 x2 6 (1 = 2)
2 x1 + x2 4
(1 = 0)
x1, x2 0

Suppose we have a feasible solution

of the original CP with value U = 13.

U v*
Since x2* = 0, we have x2
r2

13 12
or x2
= 0.5
2
If x2 is required to be integer, we can fix it to zero.
This is reduced-cost variable fixing.
CPAIOR tutorial
May 2009 Slide 177

Example: Single-Vehicle Routing

A vehicle must make several stops and return home, perhaps subject
to time windows.
The objective is to find the order of stops that minimizes travel time.
This is also known as the traveling salesman problem (with time
windows).
Stop j
Travel time cij
Stop i

CPAIOR tutorial
May 2009 Slide 178

Assignment Relaxation

min

c x
ij

= 1 if stop i immediately precedes stop j

ij

ij

xij = x ji = 1, all i
j

xij {0,1}, all i , j

CPAIOR tutorial
May 2009 Slide 179

Stop i is preceded and

followed by exactly one stop.

Assignment Relaxation

min

c x
ij

= 1 if stop i immediately precedes stop j

ij

ij

xij = x ji = 1, all i
j

Stop i is preceded and

followed by exactly one stop.

0 xij 1, all i , j
Because this problem is totally unimodular, it can be solved as an LP.
The relaxation provides a very weak lower bound on the optimal value.
But reduced-cost variable fixing can be very useful in a CP context.

CPAIOR tutorial
May 2009 Slide 180

Disjunctions of linear systems

Disjunctions of linear systems often occur naturally in problems
and can be given a convex hull relaxation.
A disjunction of linear systems
represents a union of polyhedra.

min cx

(A x b )
k

CPAIOR tutorial
May 2009 Slide 181

Relaxing a disjunction of linear systems

Disjunctions of linear systems often occur naturally in problems
and can be given a convex hull relaxation.
A disjunction of linear systems
represents a union of polyhedra.
We want a convex hull relaxation
(tightest linear relaxation).

CPAIOR tutorial
May 2009 Slide 182

min cx

(A x b )
k

Relaxing a disjunction of linear systems

Disjunctions of linear systems often occur naturally in problems
and can be given a convex hull relaxation.
The closure of the convex hull of

min cx

(A x b )
k

is described by

min cx
Ak x k b k y k , all k

y =1
x = x
k

CPAIOR tutorial
May 2009 Slide 183

0 yk 1

Why?
To derive convex hull
relaxation of a disjunction

Write each
solution as a
convex
combination
of points in
the
polyhedron

min cx
Ak x k b k , all k

y =1
x = y x
k

0 yk 1

x2

Convex hull relaxation

(tightest linear relaxation)
CPAIOR tutorial
May 2009 Slide 184

Why?
To derive convex hull
relaxation of a disjunction

Write each
solution as a
convex
combination
of points in
the
polyhedron

min cx
Ak x k b k , all k

y =1
x = y x

min cx
Ak x k b k y k , all k

y =1
x = x

Change of
variable

x = yk x k

0 yk 1

0 yk 1

x2

Convex hull relaxation

(tightest linear relaxation)
CPAIOR tutorial
May 2009 Slide 185

Mixed Integer/Linear Modeling

MILP Representability
Disjunctive Modeling
Knapsack Modeling

CPAIOR tutorial
May 2009 Slide 186

Motivation
A mixed integer/linear programming
(MILP) problem has the form

min cx + dy
Ax + by b
x, y 0
y integer

We can relax a CP problem by modeling some constraints with an MILP.

If desired, we can then relax the MILP by dropping the integrality constraint,
to obtain an LP.
The LP relaxation can be strengthened with cutting planes.
The first step is to learn how to write MILP models.

CPAIOR tutorial
May 2009 Slide 187

MILP Representability
A subset S of R n is MILP representable if it is the projection onto x
of some MILP constraint set of the form

Ax + Bu + Dy b
x, y 0

x R n , u R m , y k {0,1}

CPAIOR tutorial
May 2009 Slide 188

MILP Representability
A subset S of R n is MILP representable if it is the projection onto x
of some MILP constraint set of the form

Ax + Bu + Dy b
x, y 0

x R n , u R m , y k {0,1}
Theorem. S R is MILP
representable if and only if
S is the union of finitely
many polyhedra having the
same recession cone.

Recession cone
of polyhedron

CPAIOR tutorial
May 2009 Slide 189

Polyhedron

Example: Fixed charge function

Minimize a fixed charge function:

min x2
if x1 = 0
0
x2

f + cx1 if x1 > 0
x1 0

x2

x1
CPAIOR tutorial
May 2009 Slide 190

Example
Minimize a fixed charge function:

min x2
if x1 = 0
0
x2

f + cx1 if x1 > 0
x1 0

x2
Feasible set

x1
CPAIOR tutorial
May 2009 Slide 191

Example
Minimize a fixed charge function:

min x2
if x1 = 0
0
x2

f + cx1 if x1 > 0
x1 0

x2
Union of two
polyhedra
P1, P2
P1
x1
CPAIOR tutorial
May 2009 Slide 192

Example
Minimize a fixed charge function:

min x2
if x1 = 0
0
x2

f + cx1 if x1 > 0
x1 0

x2
Union of two
polyhedra
P1, P2

P2

P1
x1
CPAIOR tutorial
May 2009 Slide 193

Example
Minimize a fixed charge function:

min x2
if x1 = 0
0
x2

f + cx1 if x1 > 0
x1 0

x2
The
polyhedra
have
different
recession
cones.

P2

P1
x1

CPAIOR tutorial
May 2009 Slide 194

P1
recession
cone

P2
recession
cone

Example
Minimize a fixed charge function:
Add an upper bound on x1

min x2
if x1 = 0
0
x2

f + cx1 if x1 > 0
0 x1 M

x2
The
polyhedra
have the
same
recession
cone.

P2

P1
x1

CPAIOR tutorial
May 2009 Slide 195

P1
recession
cone

P2
recession
cone

Modeling a union of polyhedra

Start with a disjunction of linear
systems to represent the union
of polyhedra.
The kth polyhedron is {x |

Akx

b}

Introduce a 0-1 variable yk that is

1 when x is in polyhedron k.
Disaggregate x to create an xk for
each k.

min cx

(A
k

x bk )

min cx
Ak x k b k y k , all k
yk = 1
k

x = xk
k

y k {0,1}
CPAIOR tutorial
May 2009 Slide 196

Example
Start with a disjunction of
linear systems to represent
the union of polyhedra

min x2
x1 = 0 0 x1 M
x 0 x f + cx
2
2
1

x2
P2

P1
x1
CPAIOR tutorial
May 2009 Slide 197

Example
Start with a disjunction of
linear systems to represent
the union of polyhedra

Introduce a 0-1 variable yk

that is 1 when x is in
polyhedron k.
Disaggregate x to create an
xk for each k.

CPAIOR tutorial
May 2009 Slide 198

min x2
x1 = 0 0 x1 M
x 0 x f + cx
2
2
1

min cx
x11 = 0, x21 0
0 x12 My 2 , cx12 + x22 fy 2
y1 + y 2 = 1, y k {0,1}
x = x1 + x 2

Example
To simplify:
Replace x12 with x1.
Replace x22 with x2.
Replace y2 with y.

This yields

min x2
0 x1 My
x2 fy + cx1

y {0,1}
CPAIOR tutorial
May 2009 Slide 199

min x2
x11 = 0, x21 0
0 x12 My 2 , cx12 + x22 fy 2
y1 + y 2 = 1, y k {0,1}
x = x1 + x 2
or

min fy + cx
0 x My
y {0,1}

Big M

Disjunctive Modeling
Disjunctions often occur naturally in problems and can be given
an MILP model.
Recall that a disjunction of linear
systems (representing polyhedra
with the same recession cone)
has the MILP model

min cx

(A x b )
k

min cx
Ak x k b k y k , all k
yk = 1
k

x = xk
k

CPAIOR tutorial
May 2009 Slide 200

y k {0,1}

Example: Uncapacitated facility location

m possible
factory
locations

n markets

Locate factories to serve

markets so as to minimize
total fixed cost and
transport cost.
No limit on production
capacity of each factory.

fi

cij
i

Fixed
cost
CPAIOR tutorial
May 2009 Slide 201

j
Transport
cost

Fraction of
market js demand
satisfied from
location i

Uncapacitated facility location

m possible
factory
locations

n markets

Disjunctive model:
min

z + c x
i

ij

ij

ij

xij = 0, all j 0 xij 1, all j

z =0
, all i
z

i
i
i
xij = 1, all j
i

fi

cij
i

Fixed
cost

j
Transport
cost

CPAIOR tutorial
May 2009 Slide 202

No factory
at location i

Factory
at location i

Uncapacitated facility location

MILP formulation:
min

fi y i + cij xij
i

ij

0 xij y i , all i , j
y i {0,1}

Disjunctive model:
min

z + c x
i

ij

ij

ij

xij = 0, all j 0 xij 1, all j

z =0
, all i
z

i
i
i
xij = 1, all j
i

No factory
at location i

CPAIOR tutorial
May 2009 Slide 203

Factory
at location i

Uncapacitated facility location

MILP formulation:
min

Beginners model:

f y + c x
i

ij

ij

f y + c x

min

ij

0 xij y i , all i , j
y i {0,1}

Maximum output
from location i

ij

ij

ij

ij

ny i , all i , j

y i {0,1}

Based on capacitated location model.

It has a weaker continuous relaxation
(obtained by replacing yi {0,1} with 0 yi 1).
This beginners mistake can be avoided by
starting with disjunctive formulation.
CPAIOR tutorial
May 2009 Slide 204

Knapsack Modeling
Knapsack models consist of knapsack covering and
knapsack packing constraints.
The freight transfer model presented earlier is an example.
We will consider a similar example that combines disjunctive
and knapsack modeling.
Most OR professionals are unlikely to write a model as good
as the one presented here.

CPAIOR tutorial
May 2009 Slide 205

Note on tightness of knapsack models

The continuous relaxation of a knapsack model is not in general
a convex hull relaxation.
- A disjunctive formulation would provide a convex hull
relaxation, but there are exponentially many disjuncts.
Knapsack cuts can significantly tighten the relaxation.

CPAIOR tutorial
May 2009 Slide 206

Example: Package transport

Disjunctive model
Each package j
has size aj

Knapsack
constraints

min

Q y a ; x
i

Each truck i has

capacity Qi and
costs ci to
operate
Truck i used

CPAIOR tutorial
May 2009 Slide 207

1 if truck i carries
package j

ij

= 1, all j

yi = 1

yi = 0
zi = ci

z
=
0
a j xij Qi i
, all i
j
x = 0

0 x 1, all j ij
ij

Truck i not used

xij , y i {0,1}
1 if truck i is used

Example: Package transport

MILP model

Disjunctive model

c y
Q y a ; x

min

a x
j

ij

Qi y i , all i

xij y i , all i , j
xij , y i {0,1}

CPAIOR tutorial
May 2009 Slide 208

min

ij

= 1, all j

Q y a ; x
i

ij

= 1, all j

yi = 1

yi = 0
zi = ci

z
=
0
a j xij Qi i
, all i
j
x = 0

0 x 1, all j ij
ij

xij , y i {0,1}

Example: Package transport

Most OR professionals
would omit this constraint,
since it is the sum over i
of the next constraint.
But it generates very
effective knapsack cuts.

MILP model

c y
Q y a ; x

min

a x
j

ij

xij y i , all i , j
xij , y i {0,1}

CPAIOR tutorial
May 2009 Slide 209

= 1, all j

Qi y i , all i

ij

Modeling trick;
unobvious without
disjunctive approach

Cutting Planes
0-1 Knapsack Cuts
Gomory Cuts
Mixed Integer Rounding Cuts
Example: Product Configuration

CPAIOR tutorial
May 2009 Slide 210

To review
A cutting plane (cut, valid inequality) for
an MILP model:
is valid

Cutting
plane

Continuous
relaxation

- It is satisfied by all feasible solutions

of the model.
cuts off solutions of the continuous
relaxation.
- This makes the relaxation tighter.

Feasible solutions

CPAIOR tutorial
May 2009 Slide 211

Motivation
Cutting planes (cuts) tighten the continuous relaxation of an
MILP model.
Knapsack cuts
- Generated for individual knapsack constraints.
- We saw general integer knapsack cuts earlier.
- 0-1 knapsack cuts and lifting techniques are well studied
and widely used.
Rounding cuts
- Generated for the entire MILP, they are widely used.
- Gomory cuts for integer variables only.
- Mixed integer rounding cuts for any MILP.
CPAIOR tutorial
May 2009 Slide 212

0-1 Knapsack Cuts

0-1 knapsack cuts are designed for knapsack constraints with 0-1
variables.
The analysis is different from that of general knapsack constraints,
to exploit the special structure of 0-1 inequalities.

CPAIOR tutorial
May 2009 Slide 213

0-1 Knapsack Cuts

0-1 knapsack cuts are designed for knapsack constraints with 0-1
variables.
The analysis is different from that of general knapsack constraints,
to exploit the special structure of 0-1 inequalities.
Consider a 0-1 knapsack packing constraint ax a0. (Knapsack
covering constraints are similarly analyzed.)
Index set J is a cover if

a
j J

The cover inequality

ax a0

CPAIOR tutorial
May 2009 Slide 214

x
j J

> a0

J 1 is a 0-1 knapsack cut for

Only minimal covers need be considered.

Example
J = {1,2,3,4} is a cover for

6 x1 + 5 x2 + 5 x3 + 5 x4 + 8 x5 + 3 x6 17
This gives rise to the cover inequality

x1 + x2 + x3 + x 4 3

Index set J is a cover if

a
j J

The cover inequality

ax a0

CPAIOR tutorial
May 2009 Slide 215

x
j J

> a0

J 1 is a 0-1 knapsack cut for

Only minimal covers need be considered.

Sequential lifting
A cover inequality can often be strengthened by lifting it into a
higher dimensional space.
That is, by adding variables.
Sequential lifting adds one variable at a time.
Sequence-independent lifting adds several variables at once.

CPAIOR tutorial
May 2009 Slide 216

Sequential lifting
To lift a cover inequality

x
j J

J 1

add a term to the left-hand side

x
j J

+ k xk J 1

where k is the largest coefficient for which the inequality is still valid.

So, k = J 1 max x j
x j {0,1}
jJ
for j J

a j x j a0 ak

j J

This can be done repeatedly (by dynamic programming).

CPAIOR tutorial
May 2009 Slide 217

Example
Given

6 x1 + 5 x2 + 5 x3 + 5 x4 + 8 x5 + 3 x6 17

To lift

x1 + x2 + x3 + x 4 3

add a term to the left-hand side

where

5 = 3 max

x j {0,1}

x1 + x2 + x3 + x 4 + 5 x5 3

{x1 + x2 + x3 + x4 6 x1 + 5 x2 + 5 x3 + 5 x4 17 8}

for j{1,2,3,4}

This yields

x1 + x2 + x3 + x 4 + 2 x5 3

Further lifting leaves the cut unchanged.

But if the variables are added in the order x6, x5, the result is different:

x1 + x2 + x3 + x 4 + x5 + x6 3
CPAIOR tutorial
May 2009 Slide 218

Sequence-independent lifting
Sequence-independent lifting usually yields a weaker cut than
sequential lifting.
But it adds all the variables at once and is much faster.
Commonly used in commercial MILP solvers.

CPAIOR tutorial
May 2009 Slide 219

Sequence-independent lifting
To lift a cover inequality

x
j J

J 1

add terms to the left-hand side

x + (a ) x
j J

j J

J 1

j
if Aj u A j +1 and j {0,, p 1}

where (u ) = j + (u A j ) / if Aj u < A j and j {1,, p 1}

p + (u A ) / if A u
p
p

with

= a j a0

A j = ak

J = {1,, p}

A0 = 0

j J

CPAIOR tutorial
May 2009 Slide 220

k =1

Example
Given

6 x1 + 5 x2 + 5 x3 + 5 x4 + 8 x5 + 3 x6 17

x1 + x2 + x3 + x 4 3
Add terms x1 + x2 + x3 + x 4 + (8) x5 + (3) x6 3

To lift

where (u) is given by

This yields the lifted cut

x1 + x2 + x3 + x 4 + (5 / 4)x5 + (1/ 4)x6 3

CPAIOR tutorial
May 2009 Slide 221

Gomory Cuts
When an integer programming
problem has a nonintegral solution,
we can generate at least one Gomory
cut to cut off that solution.

Separating
cut

Solution of
continuous
relaxation

- This is a special case of a

separating cut, because it
separates the current solution of
the relaxation from the feasible
set.
Gomory cuts are widely used and
very effective in MILP solvers.
Feasible solutions

CPAIOR tutorial
May 2009 Slide 222

Gomory cuts
Given an integer programming
problem

min cx
Ax = b
x 0 and integral

Let (xB,0) be an optimal solution

of the continuous relaxation,
where

N
xB = b Nx
b = B 1b, N = B 1N

Then if xi is nonintegral in this solution, the following Gomory cut is

violated by (xB,0):

xi + N i xN bi

CPAIOR tutorial
May 2009 Slide 223

Example
min 2 x1 + 3 x2

or

min 2 x1 + 3 x2

x1 + 3 x2 3

x1 + 3 x2 x3 = 3

4 x1 + 3 x2 6
x1, x2 0 and integral

4 x1 + 3 x2 x 4 = 6
x j 0 and integral

Optimal solution of
the continuous
relaxation has
x 1
xB = 1 =

x2 2 / 3

1/ 3 1/ 3
N =

4 / 9 1/ 9

1
b =

2 / 3

CPAIOR tutorial
May 2009 Slide 224

Example
min 2 x1 + 3 x2

or

min 2 x1 + 3 x2

x1 + 3 x2 3

x1 + 3 x2 x3 = 3

4 x1 + 3 x2 6
x1, x2 0 and integral

4 x1 + 3 x2 x 4 = 6
x j 0 and integral

Optimal solution of
the continuous
relaxation has
x 1
xB = 1 =

x2 2 / 3

The Gomory cut xi + N i xN bi

1/ 3 1/ 3
N =

4 / 9 1/ 9

x3
is x2 + [ 4 / 9 1/ 9] 2 / 3
x4

1
b =

2 / 3

or

x 2 x3 0

CPAIOR tutorial
May 2009 Slide 225

In x1,x2 space this is

x1 + 2 x2 3

Example
min 2 x1 + 3 x2

or

min 2 x1 + 3 x2

x1 + 3 x2 3

x1 + 3 x2 x3 = 3

4 x1 + 3 x2 6
x1, x2 0 and integral

4 x1 + 3 x2 x 4 = 6
x j 0 and integral

Optimal solution of
the continuous
relaxation has
x 1
xB = 1 =

x2 2 / 3

1/ 3 1/ 3
N =

4 / 9 1/ 9

1
b =

2 / 3
1
b =

2 / 3

Gomory cut x1 + 2x2 3

Gomory cut after re-solving LP with
previous cut.

CPAIOR tutorial
May 2009 Slide 226

Mixed Integer Rounding Cuts

Mixed integer rounding (MIR) cuts can be generated for solutions
of any relaxed MILP in which one or more integer variables has a
fractional value.
Like Gomory cuts, they are separating cuts.
MIR cuts are widely used in commercial solvers.

CPAIOR tutorial
May 2009 Slide 227

MIR cuts
Given an MILP problem

In an optimal solution of the

continuous relaxation, let
J = { j | yj is nonbasic}

min cx + dy
Ax + Dy = b

K = { j | xj is nonbasic}

x, y 0 and y integral

N = nonbasic cols of [A D]

Then if yi is nonintegral in this solution, the following MIR cut is

violated by the solution of the relaxation:

frac(N ij )
1
y i + N ij y j + N ij +
+
N ij+ x j N ij bi

frac( bi ) frac(bi ) jK
j J1
j J 2

where

J1 = j J frac(N ij ) frac(b j )

CPAIOR tutorial
May 2009 Slide 228

J 2 = J \ J1

Example
3 x1 + 4 x2 6 y1 4 y 2 = 1
x1 + 2 x2 y1 y 2 = 3
x j , y j 0, y j integer

The MIR cut is y1 + 1/ 3 +

Take basic solution (x1,y1) = (8/3,17/3).

Then

1/ 3 2 / 3
N =

2 / 3 8 / 3

J = {2}, K = {2}, J1 = , J2 = {2}

1/ 3
1
y
+
(2 / 3)+ x2 8 / 3
2
2/3
2/3

or y1 + (1/ 2)y 2 + x2 3

CPAIOR tutorial
May 2009 Slide 229

8/3
b =

17 / 3

Example: Product Configuration

This example illustrates:
Combination of propagation and relaxation.
Processing of variable indices.
Continuous relaxation of element constraint.

CPAIOR tutorial
May 2009 Slide 230

The problem
Choose what type of each component, and how many
Personal computer
Memory
Memory
Memory

Memory

Disk
drive

Power
supply

Memory

Power
supply
CPAIOR tutorial
May 2009 Slide 231

Disk
drive

Memory

Power
supply

Disk
drive
Disk
drive
Disk
drive

Power
supply

Model of the problem

Unit cost of producing
attribute j

Amount of attribute j
produced
(< 0 if consumed):
memory, heat, power,
weight, etc.

min

Amount of attribute j
produced by type ti
of component i

c v
j

v j = qi Aijti , all j
ik

L j v j U j , all j
Quantity of
component i
installed

CPAIOR tutorial
May 2009 Slide 232

ti is a variable
index

To solve it:

Branch on domains of ti and qi.

Propagate element constraints and bounds on vj.
Variable index is converted to specially structured
element constraint.
Valid knapsack cuts are derived and propagated.
Use linear continuous relaxations.
Special purpose MILP relaxation for element.

CPAIOR tutorial
May 2009 Slide 233

Propagation

min

c v
j

v j = qi Aijti , all j
ik

L j v j U j , all j

CPAIOR tutorial
May 2009 Slide 234

This is propagated
in the usual way

Propagation
v j = zi , all j
i

element ( t i ,(qi , Aij 1,, qi Aijn ), zi ) , all i , j

min

c v
j

v j = qi Aijti , all j

This is rewritten as

ik

L j v j U j , all j

CPAIOR tutorial
May 2009 Slide 235

This is propagated
in the usual way

Propagation
v j = zi , all j
i

element ( t i ,(qi , Aij 1,, qi Aijn ), zi ) , all i , j

This can be propagated by
(a) using specialized filters for element constraints of this form

CPAIOR tutorial
May 2009 Slide 236

Propagation
v j = zi , all j
i

element ( t i ,(qi , Aij 1,, qi Aijn ), zi ) , all i , j

This is propagated by
(a) using specialized filters for element constraints of this form,
(b) adding knapsack cuts for the valid inequalities:

max {A } q v , all j
min{A } q v , all j
i

k Dti

k Dti

ijk

ijk

and (c) propagating the knapsack cuts.

[v j ,v j ] is current
domain of vj

CPAIOR tutorial
May 2009 Slide 237

Relaxation

min

c v
j

v j = qi Aijti , all j
ik

L j v j U j , all j

CPAIOR tutorial
May 2009 Slide 238

This is relaxed as

v j vj vj

Relaxation
v j = zi , all j
i

element ( t i ,(qi , Aij 1,, qi Aijn ), zi ) , all i , j

min

c v
j

v j = qi Aijti , all j
ik

L j v j U j , all j

CPAIOR tutorial
May 2009 Slide 239

This is relaxed by relaxing this

and adding the knapsack cuts.
This is relaxed as

v j vj vj

Relaxation
v j = zi , all j
i

element ( t i ,(qi , Aij 1,, qi Aijn ), zi ) , all i , j

This is relaxed by replacing each element constraint

with a disjunctive convex hull relaxation:

zi =

CPAIOR tutorial
May 2009 Slide 240

kDti

ijk

qik , qi =

kDti

ik

Relaxation
So the following LP relaxation is solved at each node
of the search tree to obtain a lower bound:
min

c v
j

vj =
i

qj =

k Dti

kDti

ik

ijk

qik , all j

, all i

v j v j v j , all j
qi qi qi , all i

max {A } q v , all j
knapsack cuts for min{ A } q v , all j
knapsack cuts for

qik 0, all i , k
CPAIOR tutorial
May 2009 Slide 241

kDti

k Dti

ijk

ijk

Computational Results
1000

Seconds

100

10

CPLEX
CLP
Hybrid

1
8x10

16x20

20x24

0.1

0.01
Problem

CPAIOR tutorial
May 2009 Slide 242

20x30

Lagrangean Relaxation
Lagrangean Duality
Properties of the Lagrangean Dual
Example: Fast Linear Programming
Domain Filtering
Example: Continuous Global Optimization

CPAIOR tutorial
May 2009 Slide 243

Motivation
Lagrangean relaxation can provide better bounds than LP
relaxation.
The Lagrangean dual generalizes LP duality.
It provides domain filtering analogous to that based on LP
duality.
- This is a key technique in continuous global optimization.
Lagrangean relaxation gets rid of troublesome constraints by
dualizing them.
- That is, moving them into the objective function.
- The Lagrangean relaxation may decouple.

CPAIOR tutorial
May 2009 Slide 244

Lagrangean Duality
Consider an
inequality-constrained
problem

min f ( x )
g(x ) 0

Hard constraints

x S

Easy constraints
The object is to get rid of (dualize) the hard constraints
by moving them into the objective function.

CPAIOR tutorial
May 2009 Slide 245

Lagrangean Duality
Consider an
inequality-constrained
problem

It is related to an
inference problem

min f ( x )

max v

g(x ) 0
x S

sS

g( x ) b f (x ) v
implies

Lagrangean Dual problem: Find the tightest lower bound

on the objective function that is implied by the constraints.
CPAIOR tutorial
May 2009 Slide 246

Primal

Dual

min f ( x )
g( x ) 0

max v
sS

g( x ) b f (x ) v

x S

Surrogate
Let us say that
xS

g( x ) 0 f ( x ) v

iff

g ( x ) 0 dominates f (x ) v 0
for some 0
g(x) f(x) v for all x S
That is, v f(x) g(x) for all x S

CPAIOR tutorial
May 2009 Slide 247

Primal

Dual

min f ( x )
g( x ) 0

max v
sS

g( x ) b f (x ) v

x S

Surrogate
Let us say that
xS

g( x ) 0 f ( x ) v

iff

g ( x ) 0 dominates f (x ) v 0
for some 0
g(x) f(x) v for all x S
That is, v f(x) g(x) for all x S
Or v min {f ( x ) g ( x )}
xS

CPAIOR tutorial
May 2009 Slide 248

Primal

Dual

min f ( x )
g( x ) 0

max v
sS

g( x ) b f (x ) v

x S

Surrogate
Let us say that
xS

g( x ) 0 f ( x ) v

iff

g ( x ) 0 dominates f (x ) v 0
for some 0
g(x) f(x) v for all x S
That is, v f(x) g(x) for all x S
Or v min {f ( x ) g ( x )}
xS

So the dual becomes

max v
CPAIOR tutorial
May 2009 Slide 249

v min{f ( x ) g ( x )} for some 0

xS

Now we have
Primal

Dual

min f ( x )
g( x ) 0

max v
v min {f ( x ) g ( x )} for some 0
xS

x S
These constraints
are dualized

or

where

max ( )

( ) = min {f ( x ) g ( x )}

Lagrangean
relaxation

CPAIOR tutorial
May 2009 Slide 250

xS

Vector of
Lagrange
multipliers

The Lagrangean dual can be viewed as the problem

of finding the Lagrangean relaxation that gives the
tightest bound.

Example
min 3 x1 + 4 x2

The Lagrangean relaxation is

x1 + 3 x2 0

(1, 2 ) = min {3 x1 + 4 x2 1( x1 + 3 x2 ) 2 (2 x1 + x2 5)}

x {0,,3}

2 x1 + x2 5 0
x1, x2 {0,1,2,3}

= min

x j {0,,3}

{(3 + 1 22 )x1 + (4 31 2 )x2 + 52 }

The Lagrangean relaxation is easy to solve
for any given 1, 2:
0 if 3 + 1 22 0
x1 =
3 otherwise

Strongest
surrogate

0 if 4 31 2 0
x2 =
3 otherwise
Optimal solution (2,1)

CPAIOR tutorial
May 2009 Slide 251

Example
min 3 x1 + 4 x2
x1 + 3 x2 0
2 x1 + x2 5 0
x1, x2 {0,1,2,3}

(1,2) is piecewise linear and concave.

2

()=5

()=9 2/7
()=0
1
()=0
()=7.5

Solution of Lagrangean dual:

(1,2) = (5/7, 13/7), () = 9 2/7
Optimal solution (2,1)
Value = 10
CPAIOR tutorial
May 2009

Slide 252

Note duality gap between 10 and 9 2/7

(no strong duality).

Example
x1 + 3 x2 0

Note: in this example, the Lagrangean dual

provides the same bound (9 2/7) as the
continuous relaxation of the IP.

2 x1 + x2 5 0
x1, x2 {0,1,2,3}

This is because the Lagrangean relaxation

can be solved as an LP:

min 3 x1 + 4 x2

(1, 2 ) = min {(3 + 1 22 )x1 + (4 31 2 )x2 + 52 }

x {0,,3}
j

= min {(3 + 1 22 )x1 + (4 31 2 )x2 + 52 }

0 x j 3

Lagrangean duality is useful when the

Lagrangean relaxation is tighter than an LP
but nonetheless easy to solve.

CPAIOR tutorial
May 2009 Slide 253

Properties of the Lagrangean dual

Weak duality: For any feasible x* and any * 0, f(x*) (*).
In particular, min f ( x )

g( x ) 0
x S

max ( )
0

Concavity: () is concave. It can therefore be maximized by

local search methods.
Complementary slackness: If x* and * are optimal, and there
is no duality gap, then *g(x*) = 0.

CPAIOR tutorial
May 2009 Slide 254

Solving the Lagrangean dual

Let k be the kth iterate, and let

k +1 = k + k k
Subgradient of () at = k

If xk solves the Lagrangean relaxation for = k, then k = g(xk).

This is because () = f(xk) + g(xk) at = k.
The stepsize k must be adjusted so that the sequence
converges but not before reaching a maximum.

CPAIOR tutorial
May 2009 Slide 255

Example: Fast Linear Programming

In CP contexts, it is best to process each node of the search tree
very rapidly.
Lagrangean relaxation may allow very fast calculation of a lower
bound on the optimal value of the LP relaxation at each node.
The idea is to solve the Lagrangean dual at the root node (which
is an LP) and use the same Lagrange multipliers to get an LP
bound at other nodes.

CPAIOR tutorial
May 2009 Slide 256

At root node, solve min cx

Dualize
Special structure,
e.g. variable bounds

Ax b

( )

Dx d
x 0

The (partial) LP dual solution *

solves the Lagrangean dual in which

( ) = min
{cx ( Ax b )}
Dx d
x 0

CPAIOR tutorial
May 2009 Slide 257

At root node, solve min cx

Dualize
Special structure,
e.g. variable bounds

Ax b

( )

Dx d
x 0

The (partial) LP dual solution *

solves the Lagrangean dual in which

( ) = min
{cx ( Ax b )}
Dx d
x 0

min cx
Ax b
At another node, the LP is Dx d

Here (*) is still a lower bound on the optimal

value of the LP and can be quickly calculated
by solving a specially structured LP.
CPAIOR tutorial
May 2009 Slide 258

Hx h
x 0

( )
Branching
constraints,
etc.

Domain Filtering
Suppose:
min f ( x )
g(x ) 0
x S

has optimal solution x*, optimal value v*, and

optimal Lagrangean dual solution *.

and i* > 0, which means the i-th constraint is tight

(complementary slackness);
and the problem is a relaxation of a CP problem;
and we have a feasible solution of the CP problem with value
U, so that U is an upper bound on the optimal value.

CPAIOR tutorial
May 2009 Slide 259

min f ( x )

Supposing g ( x ) 0
x S

has optimal solution x*, optimal value v*, and

optimal Lagrangean dual solution *:

If x were to change to a value other than x*, the LHS of i-th constraint
gi(x) 0 would change by some amount i.
Since the constraint is tight, this would increase the optimal value
as much as changing the constraint to gi(x) i 0.
So it would increase the optimal value at least i*i.
(It is easily shown that Lagrange multipliers are marginal costs. Dual
multipliers for LP are a special case of Lagrange multipliers.)

CPAIOR tutorial
May 2009 Slide 260

min f ( x )

Supposing g ( x ) 0
x S

has optimal solution x*, optimal value v*, and

optimal Lagrangean dual solution *:

We have found: a change in x that changes gi(x) by i increases

the optimal value at least i*i.
Since

optimal value of this problem optimal value of the CP U,

we have i*i U v*, or
U v*

CPAIOR tutorial
May 2009 Slide 261

i*

min f ( x )

Supposing g ( x ) 0
x S

has optimal solution x*, optimal value v*, and

optimal Lagrangean dual solution *:

We have found: a change in x that changes gi(x) by i increases

the optimal value at least i*i.
Since

optimal value of this problem optimal value of the CP U,

we have i*i U v*, or
U v*

i*

Since i = gi(x) gi(x*) = gi(x), this implies the inequality

gi ( x )
CPAIOR tutorial
May 2009 Slide 262

U v*

i*

which can be propagated.

Example: Continuous Global Optimization

Some of the best continuous global solvers (e.g., BARON)
combine OR-style relaxation with CP-style interval arithmetic and
domain filtering.
These methods can be combined with domain filtering based on
Lagrange multipliers.

CPAIOR tutorial
May 2009 Slide 263

Continuous Global Optimization

x2

max x1 + x2
4 x1x2 = 1
2 x1 + x2 2

Global optimum

x1 [0,1], x2 [0,2]
Local optimum

Feasible set

x1
CPAIOR tutorial
May 2009 Slide 264

To solve it:

Search: split interval domains of x1, x2.

Each node of search tree is a problem restriction.
Propagation: Interval propagation, domain filtering.
Use Lagrange multipliers to infer valid inequality for
propagation.
Reduced-cost variable fixing is a special case.
Relaxation: Use function factorization to obtain linear
continuous relaxation.

CPAIOR tutorial
May 2009 Slide 265

Interval propagation
x2
Propagate intervals
[0,1], [0,2]
through constraints
to obtain
[1/8,7/8], [1/4,7/4]

x1
CPAIOR tutorial
May 2009 Slide 266

Relaxation (function factorization)

Factor complex functions into elementary functions that have
known linear relaxations.
Write 4x1x2 = 1 as 4y = 1 where y = x1x2.
This factors 4x1x2 into linear function 4y and bilinear function x1x2.
Linear function 4y is its own linear relaxation.

CPAIOR tutorial
May 2009 Slide 267

Relaxation (function factorization)

Factor complex functions into elementary functions that have
known linear relaxations.
Write 4x1x2 = 1 as 4y = 1 where y = x1x2.
This factors 4x1x2 into linear function 4y and bilinear function x1x2.
Linear function 4y is its own linear relaxation.
Bilinear function y = x1x2 has relaxation:

x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2

x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2
where domain of xj is [ x j , x j ]

CPAIOR tutorial
May 2009 Slide 268

Relaxation (function factorization)

The linear relaxation becomes:

min x1 + x2
4y = 1
2 x1 + x2 2
x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2
x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2
x j x j x j , j = 1,2

CPAIOR tutorial
May 2009 Slide 269

Relaxation (function factorization)

x2

Solve linear relaxation.

x1

CPAIOR tutorial
May 2009 Slide 270

Relaxation (function factorization)

x2

x2 [1,1.75]
Solve linear relaxation.

Since solution is infeasible,

split an interval and branch.

x2 [0.25,1]

CPAIOR tutorial
May 2009 Slide 271

x1

x2 [0.25,1]

x2 [1,1.75]

x2

x2

x1
CPAIOR tutorial
May 2009 Slide 272

x1

x2 [0.25,1]

x2 [1,1.75]

x2

Solution of
relaxation is
feasible,
value = 1.25

x2

This becomes
incumbent
solution

x1
CPAIOR tutorial
May 2009 Slide 273

x1

x2 [0.25,1]

x2 [1,1.75]

x2

Solution of
relaxation is
feasible,
value = 1.25
This becomes
incumbent
solution

x1
CPAIOR tutorial
May 2009 Slide 274

x2
Solution of
relaxation is
not quite
feasible,
value = 1.854
Also use
Lagrange
multipliers for
domain
filtering

x1

Relaxation (function factorization)

min x1 + x2
4y = 1

Associated Lagrange
multiplier in solution of
relaxation is 2 = 1.1

2 x1 + x2 2
x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2
x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2
x j x j x j , j = 1,2

CPAIOR tutorial
May 2009 Slide 275

Relaxation (function factorization)

min x1 + x2

Associated Lagrange
multiplier in solution of
relaxation is 2 = 1.1

4y = 1

2 x1 + x2 2
x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2
x2 x1 + x1x2 x1x2 y x2 x1 + x1x2 x1x2
x j x j x j , j = 1,2

This yields a valid inequality for propagation:

2 x1 + x2 2
Value of
relaxation
CPAIOR tutorial
May 2009 Slide 276

1.854 1.25
= 1.451
1.1

Lagrange multiplier

Value of incumbent
solution

Dynamic Programming in CP
Example: Capital Budgeting
Domain Filtering
Recursive Optimization

CPAIOR tutorial
May 2009 Slide 277

Motivation
Dynamic programming (DP) is a highly versatile technique that
can exploit recursive structure in a problem.
Domain filtering is straightforward for problems modeled as a
DP.
DP is also important in designing filters for some global
constraints, such as the stretch constraint (employee scheduling).
Nonserial DP is related to bucket elimination in CP and exploits
the structure of the primal graph.
DP modeling is the art of keeping the state space small while
maintaining a Markovian property.
We will examine only one simple example of serial DP.

CPAIOR tutorial
May 2009 Slide 278

Example: Capital Budgeting

We wish to built power plants with a total cost of at most 12 million
Euros.
There are three types of plants, costing 4, 2 or 3 million Euros
each. We must build one or two of each type.
The problem has a simple knapsack packing model:

Number of
factories of type j

CPAIOR tutorial
May 2009 Slide 279

4 x1 + 2 x2 + 3 x3 12
x j {1,2}

State sk

Example: Capital Budgeting

4 x1 + 2 x2 + 3 x3 12
x j {1,2}

Stage k

In general the recursion for ax b is

fk (sk ) = max {fk +1(sk + ak xk )}

x3=2

xk Dxk

= 1 if there is
a path from
state sk to a
feasible
solution,
0 otherwise

CPAIOR tutorial
May 2009 Slide 280

State is sum
of first k terms
of ax

x3=1

f3(8) = max{f4(8+31), f4(8+32)} = max{1,0} = 1

f4(14)=0
f4(11)=1

Example: Capital Budgeting

4 x1 + 2 x2 + 3 x3 12
x j {1,2}

0
0

In general the recursion for ax b is

fk (sk ) = max {fk +1(sk + ak xk )}

0
1

1
1

1 if sn +1 b
fn +1(sn +1 ) =
0 otherwise

1
1

fk(sk) for each state sk

CPAIOR tutorial
May 2009 Slide 281

0
0

xk Dxk

Boundary condition:

Example: Capital Budgeting

4 x1 + 2 x2 + 3 x3 12
x j {1,2}

0
0

The problem is feasible.

Each path to 1 is a feasible
solution.
Path 1: x = (1,2,1)

0
1

1
1

1
1

Path 3: x = (1,1,1)

CPAIOR tutorial
May 2009 Slide 282

0
0

Path 2: x = (1,1,2)

Possible costs are 9,11,12.

fk(sk) for each state sk

Domain Filtering
4 x1 + 2 x2 + 3 x3 12
x j {1,2}
To filter domains: observe what
values of xk occur on feasible
paths.

Dx3 = {1,2}
Dx2 = {1,2}
Dx1 = {1}

CPAIOR tutorial
May 2009 Slide 283

x1=1

x2=2

x3=1
x3=2

x2=1
x3=1

Recursive Optimization
max 15 x1 + 10 x2 + 12 x3
4 x1 + 2 x2 + 3 x3 12

Maximize
revenue

x j {1,2}

The recursion includes arc values:

fk (sk ) = max {ck xk + fk +1(sk + ak xk )}

xk Dxk

= value on max
value path from
sk to final stage
(value to go)

CPAIOR tutorial
May 2009 Slide 284

122

121

Arc value

f3(8) = max{121+f4(8+31), 122+f4(8+32)}

= max{12,} = 12
f4(14)=
f4(11)=0

Recursive optimization

max 15 x1 + 10 x2 + 12 x3
x j {1,2}

Boundary condition:

12

34
24

0
0

if sn +1 b
0
fn +1(sn +1 ) =
otherwise
fk(sk) for each state sk

CPAIOR tutorial
May 2009 Slide 285

49

The recursion includes arc values:

fk (sk ) = max {ck xk + fk +1(sk + ak xk )}

4 x1 + 2 x2 + 3 x3 12

Recursive optimization

max 15 x1 + 10 x2 + 12 x3
x j {1,2}

12

34
24

0
0

(x1,x2,x3) = (1,1,2)

fk(sk) for each state sk

CPAIOR tutorial
May 2009 Slide 286

49

The maximum revenue is 49.

The optimal path is easy to
retrace.

4 x1 + 2 x2 + 3 x3 12

CP-based Branch and Price

Basic Idea
Example: Airline Crew Scheduling

CPAIOR tutorial
May 2009 Slide 287

Motivation
Branch and price allows solution of integer programming
problems with a huge number of variables.
The problem is solved by a branch-and-relax method. The
difference lies in how the LP relaxation is solved.
Variables are added to the LP relaxation only as needed.
Variables are priced to find which ones should be added.
CP is useful for solving the pricing problem, particularly when
constraints are complex.
CP-based branch and price has been successfully applied
to airline crew scheduling, transit scheduling, and other
transportation-related problems.
CPAIOR tutorial
May 2009 Slide 288

Basic Idea
Suppose the LP relaxation of an integer
programming problem has a huge number of
variables:
We will solve a restricted master problem,
which has a small subset of the variables:
Column j of A

min cx
Ax = b
x0

c x

min

j J

A x
jJ

=b

xj 0

Adding xk to the problem would improve the solution if xk has a

negative reduced cost:

rk = ck Ak < 0

CPAIOR tutorial
May 2009 Slide 289

( )

Basic Idea
Adding xk to the problem would improve the solution if xk has a
negative reduced cost:

rk = ck Ak < 0

Computing the reduced cost of xk is known as pricing xk.

Cost of column y

So we solve the pricing problem:

min c y y
y is a column of A

If the solution y* satisfies cy* y* < 0, then we can add column y to

the restricted master problem.

CPAIOR tutorial
May 2009 Slide 290

Basic Idea
The pricing problem

max y
y is a column of A

need not be solved to optimality, so long as we find a column with

negative reduced cost.
However, when we can no longer find an improving column, we
solved the pricing problem to optimality to make sure we have the
optimal solution of the LP.
If we can state constraints that the columns of A must satisfy,
CP may be a good way to solve the pricing problem.

CPAIOR tutorial
May 2009 Slide 291

Example: Airline Crew Scheduling

We want to assign crew members to flights to minimize
cost while covering the flights and observing complex
work rules.
Flight data

A roster is the sequence of flights assigned to

a single crew member.
The gap between two consecutive flights in a
roster must be from 2 to 3 hours. Total flight
time for a roster must be between 6 and 10
hours.
For example,
flight 1 cannot immediately precede 6
flight 4 cannot immediately precede 5.

Start
time

Finish
time

CPAIOR tutorial
May 2009 Slide 292

The possible rosters are:

(1,3,5), (1,4,6), (2,3,5), (2,4,6)

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

CPAIOR tutorial
May 2009 Slide 293

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

Rosters that cover flight 1.

CPAIOR tutorial
May 2009 Slide 294

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

Rosters that cover flight 2.

CPAIOR tutorial
May 2009 Slide 295

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

Rosters that cover flight 3.

CPAIOR tutorial
May 2009 Slide 296

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

Rosters that cover flight 4.

CPAIOR tutorial
May 2009 Slide 297

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

Rosters that cover flight 5.

CPAIOR tutorial
May 2009 Slide 298

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

Rosters that cover flight 6.

CPAIOR tutorial
May 2009 Slide 299

Airline Crew Scheduling

There are 2 crew members, and the possible rosters are:
1
2
3
4
(1,3,5), (1,4,6), (2,3,5), (2,4,6)
The LP relaxation of the problem is:
Cost c12 of assigning crew member 1 to roster 2
= 1 if we assign crew member 1
to roster 2, = 0 otherwise.
Each crew member is assigned to
exactly 1 roster.
Each flight is assigned at least 1
crew member.

In a real problem, there can be millions of rosters.

CPAIOR tutorial
May 2009 Slide 300

Airline Crew Scheduling

We start by solving the problem with a subset
of the columns:
Optimal
dual
solution

u1
u2
v1
v2
v3
v4
v5
v6

CPAIOR tutorial
May 2009 Slide 301

Airline Crew Scheduling

We start by solving the problem with a subset
of the columns:
Dual
variables
u1
u2
v1
v2
v3
v4
v5
v6

CPAIOR tutorial
May 2009 Slide 302

Airline Crew Scheduling

We start by solving the problem with a subset
of the columns:
Dual
variables
u1
u2
v1
v2
v3
v4
v5
v6

CPAIOR tutorial
May 2009 Slide 303

The reduced cost of an

excluded roster k for
crew member i is

cik ui

vj

j in roster k

We will formulate the

pricing problem as a
shortest path problem.

Pricing problem

Crew
member 1

Crew
member 2

CPAIOR tutorial
May 2009 Slide 304

Pricing problem

Each s-t path corresponds to a roster,

provided the flight time is within bounds.
2

Crew
member 1

Crew
member 2

CPAIOR tutorial
May 2009 Slide 305

Pricing problem

Cost of flight 3 if it immediately follows

flight 1, offset by dual multiplier for flight 1
2

Crew
member 1

Crew
member 2

CPAIOR tutorial
May 2009 Slide 306

Pricing problem

Cost of transferring from home to flight 1,

offset by dual multiplier for crew member 1
2

Crew
member 1
Dual multiplier
omitted to break
symmetry

Crew
member 2

CPAIOR tutorial
May 2009 Slide 307

Pricing problem

Length of a path is reduced cost of the

corresponding roster.
2

Crew
member 1

Crew
member 2

CPAIOR tutorial
May 2009 Slide 308

Pricing problem

Arc lengths using dual solution of LP

relaxation
5

10

2
2

Crew
member 1

0
Crew
member 2

2
3
4

-9
5

CPAIOR tutorial
May 2009 Slide 309

1
6

Pricing problem
Solution of shortest path problems
5

10

2
2

Crew
member 1

Reduced cost = 1
Add x12 to problem.

0
Crew
member 2
Reduced cost = 2
Add x23 to problem.
CPAIOR tutorial
May 2009 Slide 310

2
3
4

-9
5

1
6

After x12 and x23 are added to the problem, no

remaining variable has negative reduced cost.

Pricing problem
The shortest path problem cannot be solved by traditional shortest
path algorithms, due to the bounds on total duration of flights.
It can be solved by CP:
Set of flights
assigned to crew
member i
Path global constraint
Setsum global constraint

Path
length

Graph

Path( X i , zi ,G ), all flights i

Tmin ( f j s j ) Tmax
jX i

X i {flights}, zi < 0, all i

Duration of flight j
CPAIOR tutorial
May 2009 Slide 311

CP-based Benders Decomposition

Benders Decomposition in the Abstract
Classical Benders Decomposition
Example: Machine Scheduling

CPAIOR tutorial
May 2009 Slide 312

Motivation
Benders decomposition allows us to apply CP and OR to
different parts of the problem.
It searches over values of certain variables that, when fixed,
result in a much simpler subproblem.
The search learns from past experience by accumulating
Benders cuts (a form of nogood).
The technique can be generalized far beyond the original OR
conception.
Generalized Benders methods have resulted in the greatest
speedups achieved by combining CP and OR.

CPAIOR tutorial
May 2009 Slide 313

Benders Decomposition in the Abstract

Benders decomposition
can be applied to
problems of the form

min f ( x, y )
S( x, y )
x Dx , y Dy

When x is fixed to some

value, the resulting
subproblem is much
easier:

min f ( x , y )
S( x , y )
y Dy

perhaps
because it
decouples into
smaller problems.

For example, suppose x assigns jobs to machines, and y schedules

the jobs on the machines.
When x is fixed, the problem decouples into a separate scheduling
subproblem for each machine.
CPAIOR tutorial
May 2009 Slide 314

Benders Decomposition
We will search over assignments to x. This is the master problem.
In iteration k we assume x = xk
and solve the subproblem

min f ( x k , y )
S( x k , y )
y Dy

We generate a Benders cut (a type of nogood)

that satisfies Bk+1(xk) = vk.

and get optimal

value vk

v Bk +1( x )

Cost in the original problem

The Benders cut says that if we set x = xk again, the resulting cost v
will be at least vk. To do better than vk, we must try something else.
It also says that any other x will result in a cost of at least Bk+1(x),
perhaps due to some similarity between x and xk.
CPAIOR tutorial
May 2009 Slide 315

Benders Decomposition
We will search over assignments to x. This is the master problem.
In iteration k we assume x = xk
and solve the subproblem

min f ( x k , y )
S( x k , y )
y Dy

We generate a Benders cut (a type of nogood)

that satisfies Bk+1(x) = vk.

and get optimal

value vk

v Bk +1( x )

Cost in the original problem

We add the Benders cut to the master problem, which becomes

min v
v Bi ( x ), i = 1,, k + 1
x Dx
CPAIOR tutorial
May 2009 Slide 316

Benders cuts
generated so far

Benders Decomposition
min v
We now solve the v B ( x ), i = 1,, k + 1
i
master problem
x Dx

to get the next

trial value xk+1.

The master problem is a relaxation of the original problem, and its

optimal value is a lower bound on the optimal value of the original
problem.
The subproblem is a restriction, and its optimal value is an upper
bound.
The process continues until the bounds meet.
The Benders cuts partially define the projection of the feasible set
onto x. We hope not too many cuts are needed to find the optimum.
CPAIOR tutorial
May 2009 Slide 317

Classical Benders Decomposition

The classical method
applies to problems
of the form

min f ( x ) + cy
g ( x ) + Ay b
x Dx , y 0

and the subproblem

is an LP

min f ( x ) + cy
Ay b g ( x k ) ( )
y 0
k

whose dual is

max f ( x k ) + ( b g ( x k ) )

A c
0

Let k solve the dual.

By strong duality, Bk+1(x) = f(x) + k(b g(x)) is the tightest lower
bound on the optimal value v of the original problem when x = xk.
Even for other values of x, k remains feasible in the dual. So by
weak duality, Bk+1(x) remains a lower bound on v.
CPAIOR tutorial
May 2009 Slide 318

Classical Benders
So the master problem

becomes

min v
v Bi ( x ), i = 1,, k + 1
x Dx

min v
v f ( x ) + i (b g ( x )), i = 1,, k + 1
x Dx

In most applications the master problem is

an MILP
a nonlinear programming problem (NLP), or
a mixed integer/nonlinear programming problem (MINLP).

CPAIOR tutorial
May 2009 Slide 319

Example: Machine Scheduling

Assign 5 jobs to 2 machines (A and B), and schedule the
machines assigned to each machine within time windows.
The objective is to minimize makespan.
Time lapse between
start of first job and
end of last job.

Assign the jobs in the master

problem, to be solved by MILP.
Schedule the jobs in the
subproblem, to be solved by CP.

CPAIOR tutorial
May 2009 Slide 320

Machine Scheduling
Job Data

Once jobs are assigned, we can

minimize overall makespan by
minimizing makespan on each
machine individually.
So the subproblem decouples.

Machine A
Machine B

CPAIOR tutorial
May 2009 Slide 321

Machine Scheduling
Job Data

Once jobs are assigned, we can

minimize overall makespan by
minimizing makespan on each
machine individually.
So the subproblem decouples.

Minimum makespan
schedule for jobs 1, 2, 3, 5
on machine A

CPAIOR tutorial
May 2009 Slide 322

Machine Scheduling
The problem is

Start time of job j

min M
M s j + px j j , all j
r j s j d j px j j , all j

Time windows
Jobs cannot overlap

disjunctive ( (s j x j = i ),( pij x j = i ) ) , all i

CPAIOR tutorial
May 2009 Slide 323

Machine Scheduling
The problem is

Start time of job j

min M
M s j + px j j , all j
r j s j d j px j j , all j

Time windows
Jobs cannot overlap

disjunctive ( (s j x j = i ),( pij x j = i ) ) , all i

For a fixed assignment x the subproblem on each machine i is

min M
M s j + px j j , all j with x j = i
r j s j d j px j j , all j with x j = i
CPAIOR tutorial
May 2009 Slide 324

disjunctive ( (s j x j = i ),( pij x j = i ))

Benders cuts
Suppose we assign jobs 1,2,3,5 to machine A in iteration k.
We can prove that 10 is the optimal makespan by proving that the
schedule is infeasible with makespan 9.

Edge finding derives infeasibility by reasoning only with jobs 2,3,5.

So these jobs alone create a minimum makespan of 10.
So we have a Benders cut

10 if x2 = x3 = x4 = A
v Bk +1( x ) =
0 otherwise

CPAIOR tutorial
May 2009 Slide 325

Benders cuts
We want the master problem to be an MILP, which is good for
assignment problems.
So we write the Benders cut

10 if x2 = x3 = x4 = A
v Bk +1( x ) =
0 otherwise

Using 0-1 variables:

v 10 ( x A 2 + x A3 + x A5 2 )
= 1 if job 5 is
v 0
assigned to
machine A

CPAIOR tutorial
May 2009 Slide 326

Master problem
The master problem is an MILP:

min v
5

p
j =1

Aj

x Aj 10, etc.
Constraints derived from release times

p
j =1

Constraints derived from time windows

Bj

xBj 10, etc.

j =1

j =3

v pij xij , v 2 + pij xij , etc., i = A, B

v 10( x A 2 + x A3 + x A5 2)
v 8 xB 4
xij {0,1}
CPAIOR tutorial
May 2009 Slide 327

Benders cut from machine A

Benders cut from machine B

Stronger Benders cuts

If all release times are the same, we can strengthen the Benders cuts.
We are now using the cut

v Mik xij J ik + 1
jJik

Min makespan
on machine i
in iteration k

Set of jobs
assigned to
machine i in
iteration k

A stronger cut provides a useful bound even if only some of the jobs in
Jik are assigned to machine i:

v Mik

(1 x

j Jik

ij

)pij

These results can be generalized to cumulative scheduling.

CPAIOR tutorial
May 2009 Slide 328

CPAIOR tutorial
May 2009 Slide 329