IEDA 3010

Prescriptive Analytics
Discrete Optimization 2

IEDA 3010
Discrete Optimization - Algorithm

Dr. Jin QI
Department of Industrial Engineering and Decision Analytics
Hong Kong University of Science and Technology
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Introduction
• “Natural solutions” for IP problems include
– Solve LP relaxations and round the non-integer values to
integers, since IP problems are LP problems with integer
restrictions
– Exhaustive enumeration, since pure IP problems with a
bounded feasible region have a finite number of feasible
solutions

2
 gramming problems, sometimes it is tempting to use the appro
IEDA 3010
ply applying the simplex method to the LP Prescriptive Analyticsand then
relaxation
values to integers in the resulting solution. ThisOptimization
Discrete approach 2 may

plications, especially if the values of the variables are quite larg

LP relaxation and rounding
relatively little error. However, you should beware of two pitfalls
One pitfall is that an optimal linear programming solution i
after it is rounded. Often it is difficult to see in which way the
• An optimal LP solution may not be feasible for the IP
to retain feasibility. It may even be necessary to change the va
problem after rounding
one or more units after rounding. To illustrate, consider the foll

Maximize Z ! x2,
subject to
1
"x1 # x2 $ %%
2
1
"x1 # x2 $ 3%%
2
and
x1 & 0, x2 & 0
x1, x2 are integers.
As Fig. 11.2 shows, the optimal solution for the LP relaxation
3 is
1_464-536.qxd 11/19/08 08:33 AM Page 490

IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

LP relaxation and rounding

CHAPTER 11 INTEGER PROGRAMMING

The rounded solutions

x2 are not feasible

3
( , 2)
2
2

Optimal solution for

the LP relaxation
1

URE 11.2
ample of an IP problem Feasible region for
the optimal solution the LP relaxation
LP relaxation cannot
nded in any way that
feasibility. 0 1 2 3 4 x1
4
 example of an IP problem Feasible region for
ere the optimal solution the LP relaxation IEDA 3010
the LP relaxation cannot Prescriptive Analytics
Discrete Optimization 2
rounded in any way that
ains feasibility. 0 1 2 3 4
LP relaxation and rounding
• Even if an optimal LP solution
integer solution. In fact, itismay
rounded
even be far from optimal in terms
successfully,jective
it may be far
function. Thisfrom
fact isoptimal
illustrated by the following problem:
Maximize Z ! x1 " 5x2,
subject to
x1 " 10x2 # 20
x1 # 2
and
x1 $ 0, x2 $ 0
x1, x2 are integers.
Because there are only two decision variables, this problem can b
as shown in Fig. 11.3. Either the graph or the simplex method m
the optimal solution for the LP relaxation is x1 ! 2, x2 ! 5%95%, with
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

LP11.6relaxation and
THE BRANCH-AND-BOUND rounding
TECHNIQUE AND ITS APPLICATION

x2
Optimal IP solution
3

Optimal solution for

2 the LP relaxation
Z* ! 10
! x1 "
5x2

1
RE 11.3 Rounded solution
mple where rounding
mal solution for the
ation is far from
for the IP problem. 0 1 2 3 x1

6 types
for various combinatorial problems such as IP problems. Three prominent
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Exhaustive enumeration
• Unlike LP problems, optimal solutions are not
necessarily corner points of the feasible region
– The number of feasible solutions may be huge

7
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Branch-and-Bound Method
• Basic idea: divide and conquer
– Divide
• Partition the entire set of feasible solutions into smaller and
smaller subsets (branching)
– Conquer
• Bound how good the best solution in the subset can be
(bounding)
• Discard the subset if its bound indicates that it cannot possibly
contain an optimal solution for the original problem (fathoming)

8
 and smaller subsets. The conquering ( fathoming) is done partially by bo
the best solution in the subset can be and then discardingIEDA the
3010 subset if it
Prescriptive Analytics
that it cannot possibly contain an optimal solution for the
Discrete original
Optimization 2 prob
We shall now describe in turn these three basic steps—branchin
B&B for Binary IP Problems
fathoming—and illustrate them by applying a branch-and-bound algori
type example (the California Manufacturing Co. problem) presented in
peated here (with the constraints numbered for later reference).
Maximize Z ! 9x1 " 5x2 " 6x3 " 4x4,
subject to
(1) 6x1 " 3x2 " 5x3 " 2x4 # 10
(2) 3 " 3x2 " 5x3 " 2x4 # 1
(3) $x1 " 3x3 # 0
(4) x1 "$x2 " 5x3 " x4 # 0
and
(5) xj is binary, for j ! 1, 2, 3, 4.

Branching
When you are dealing with binary variables, the most straightforward w
set of feasible solutions into subsets is to fix the value of one of9the var
 492 CHAPTER 11 INTEGER PROGRAMMING
IEDA 3010
Prescriptive Analytics
Discrete Optimization 2
some success to a variety of OR problems, but it is especia
cation to IP problems.

B&B for Binary IP Problems The basic concept underlying the branch-and-bound tec
quer. Since the original “large” problem is too difficult to be
into smaller and smaller subproblems until these subproblem
• Bounding viding (branching) is done by partitioning the entire set of f
and smaller subsets. The conquering ( fathoming) is done par
– LP relaxation
(5) 0  xj  1,the best forsolution
j = in1,the
2,subset
3, 4 can be and then discarding the
(x , x , x , xthat
) it cannot
= 5 possibly contain an optimal solution for
, 1, 0, 1 , with Z = 16 1 the o
– LP solution 1 2 3 4 6 now describe in turn these three basic
We shall 2 steps
5 – 1LP, valuewith Z = 16 1
4 ) = 6 , 1, 0,
fathoming—and illustrate them by applying a branch-and-b
2 type example (the California Manufacturing Co. problem) p
und for whole – Bound
problem: Z  16 peated here (with the constraints numbered for later referen
Maximize Z ! 9x1 " 5x2 " 6x3 " 4x4,
subject to
(1) 6x1 " 3x2 " 5x3 " 2x4 # 10
(2) 3 " 3x2 " 5x3 " 2x4 # 1
(3) $x1 " 3x3 # 0
(4) x1 "$x2 " 5x3 " x4 # 0
and
(5) xj is binary, for j ! 1, 2, 3, 4.
10
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Binary IP Problems

• Branching x1
Branching variable
x1 = 0

All

x1 = 1

– Some subproblems can be conquered immediately

– Some subproblems need to be divided further into even
smaller subproblems

11
and Fix x1 ! 0 so the resulting subproblem reduces to
(5) xj is binary, for j ! 1, 2, 3, 4. Maximize Z ! 5x2 " 6x3 " IEDA
4x4, 3010
Prescriptive Analytics
subject to Discrete Optimization 2
Branching
(1) 3x2 " 5x3 " 2x4 # 10
When you are dealing with binary variables, the most straightforward wayx3to"partition
x4 # the
B&B for Binary IP Problems
x1 ! 0 for one subset and at x1 ! 1 for the other subset. (4)
(2)
set of feasible solutions into subsets is to fix the value of(3)
one of the variables
Doing this
x3 (say,#x1)0at
1

$x2for 5xthe3 "

prototype
x4 # ex-0
ample divides the whole problem into the two smaller subproblems
(5) xj is shown
binary, next.for j ! 2, 3, 4.
Subproblem 1: Subproblem 2:
Fix x1 ! 0 so the resulting subproblem reduces to Fix x1 ! 1 so the resulting subproblem reduces to
Maximize Z ! 5x2 " 6x3 " 4x4, Maximize Z ! 9 " 5x2 " 6x3 " 4x4,
subject to subject to
(1) 3x2 " 5x3 " 2x4 # 10 (1) 3x2 " 5x3 " 2x4 # 4
(2) x3 " x4 # 1 (2) x3 " x4 # 1
(3) x3 # 0 (3) x3 #1
(4) $x2 5x3 " x4 # 0 (4) $x2 5x3 " x4 # 0
(5) xj is binary, for j ! 2, 3, 4. (5) xj is binary, for j ! 2, 3, 4.
Subproblem 2:
Fix x1 ! 1 so the resulting subproblem reduces to
Maximize Z ! 9 " 5x2 " 6x3 " 4x4,
subject to
(1) 3x2 " 5x3 " 2x4 # 4
(2) x3 " x4 # 1
(3) x3 #1 12
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Binary IP Problems

• Bounding
– LP relaxation of subproblem 1
(5) x1 = 0, 0  xj  1, for j = 2, 3, 4
optimal solution:
(x1 , x2 , x3 , x4 ) = (0, 1, 0, 1) , with Z = 9
– LP relaxation of subproblem 2
(5) x1 = 1, 0  xj  1, for j = 2, 3, 4
optimal solution:
(x1 , x2 , x3 , x4 ) = 1, 54 , 0, 45 , with Z = 16 15
Bound for subproblem 1: Z9
Bound for subproblem 2: Z  16

13
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Binary IP Problems

Branching
x1
variable

x1 = 0
9
(0, 1, 0, 1)
All
16
(⅚, 1, 0, 1) x1 = 1
16
(1, ⅘, 0, ⅘)

14
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Binary IP Problems

• Fathoming
– A subproblem can be conquered (fathomed), and thereby
dismissed from further consideration in three ways
1. Optimal solution for the LP relaxation is an integer solution
• Optimal for the subproblem itself
• Incumbent (the best feasible solution found so far) for the
whole problem
Z ⇤ = value of Z for current incumbent
• Can be updated later
2. The bound given by the LP relaxation is smaller than the value of
the objective function for current incumbent, i.e.
Bound  Z ⇤
3. The LP relaxation has no feasible solutions
15
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Branching
x1
variable
Sub1
F(1)
x1 = 0
9 = Z*
(0, 1, 0, 1)
All = incumbent
16
(⅚, 1, 0, 1) Sub2
x1 = 1
16
(1, ⅘, 0, ⅘)

16
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Binary IP Problems

• Summary of the algorithm
– Initialization: Set Z* = -∞.
– Steps for each iteration:
1. Branching: Among the unfathomed subproblems, select the
one that was created most recently and divide it into two new
subproblems. (Break ties according to which has the larger
bound.)
2. Bounding: For each new subproblem, solve its LP relaxation
to find the upper bound
3. Fathoming: For each new subproblem, discard those
subproblems that are fathomed by any of the three tests
Optimality test: Stop when there are no remaining subproblems;
the current incumbent is optimal

17
 Fix x1 ! 1, x2 ! 0 so the resulting subproblem reduces to
ping around in the branching tree by selecting the subproblem with the best bound.
Maximize Z ! 9 " 6x3 " 4x4,
IEDA 3010
Completing the Example Prescriptive Analytics
subject to
The pattern for the remaining iterations will be quite similar to that for the first iteration de- Discrete Optimization 2
(1) 5x3 " 2x4 # 4
scribed above except for the ways in which fathoming occurs. Therefore, we shall summa-
(2) x3 " x4 # 1

Iteration 2
rize the branching and bounding steps fairly briefly and then focus on the fathoming step.
(3) x3 #1
Iteration 2. The only remaining subproblem corresponds to the (4)x1 ! 1 node inx4Fig.
# 011.6,
so we shall branch from this node to create the two new subproblems (5) given below.
xj is binary, for j ! 3, 4.
Subproblem 3: Subproblem 4:
Fix x1 ! 1, x2 ! 0 so the resulting subproblem reduces to Fix x1 ! 1, x2 ! 1 so the resulting subproblem reduces to
Maximize Z ! 9 " 6x3 " 4x4, Maximize Z ! 14 " 6x3 " 4x4,
subject to subject to
(1) 5x3 " 2x4 # 4 (1) 5x3 " 2x4 # 1
(2) x3 " x4 # 1 (2) x3 " x4 # 1
(3) x3 # 1 (3) x3 # 1
(4) x4 # 0 (4) x4 # 1
(5) xj is binary, for j ! 3, 4. (5) xj is binary, for j ! 3, 4.
Subproblem 4: The LP relaxations of these subproblems are obtained by addin
LPxrelaxation:
Fix LP relaxation:
1 ! 1, x2 ! 1 so the resulting subproblem reduces to straint shown below into the relaxed version of constraint (5). Their o
(5) 0  xj  1, for j = 3, 4 are
(5)shown
0 below.
xj  1, for j = 3, 4
Maximize Z ! 14 " 6x3 " 4x4,
Optimal solution: Optimal solution:
subject to
(x1 , x2 , x3 , x4 ) = 1, 0, 45 , 0 , with Z = 13 45 (x1LP
, x2relaxation
, x3 , x4 ) of
= subproblem
1, 1, 0, 12 3:, (5) x1 $ 1, x2 # 0, and
with Z = 16
(1) Z5x≤ for j ! 1, 2, 3, 4.
Bound: 3"132x4 # 1 Bound: Z ≤ 16
(2) x3 " x4 # 1
(3) x3 #1
(4) x4 # 1
(5) xj is binary, for j ! 3, 4.
18
The LP relaxations of these subproblems are obtained by adding the additional con-
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Branching
x1 x2
variable
Sub1
F(1)
x1 = 0
9 = Z*
(0, 1, 0, 1)
All Sub3
= incumbent
x2 = 0
16 13
(⅚, 1, 0, 1) Sub2 (1, 0, ⅘, 0)
x1 = 1
16
Sub4
x2 = 1
16
(1, 1, 0, ½)

19
 the existence of an optimal solution). Alas, test 3 fails as well, because both optimal so-
lutions include variables with noninteger values.
Figure 11.7 shows the resulting branching tree at this point. The lack of an F to the IEDA 3010
hil76299_ch11_464-536.qxd 11/19/08 08:33 AM Page 498
right of either new node indicates that both remain unfathomed. Prescriptive Analytics
Discrete Optimization 2
Iteration 3. So far, the algorithm has created four subproblems. Subproblem 1 has been
fathomed, and subproblem 2 has been replaced by (separated into) subproblems 3 and 4,

Iteration 3
but these last two remain under consideration. Because they were created simultaneously,
but subproblem 4 (x1 ! 1, x2 ! 1) has the larger bound (16 % 13), the next branching is
done from the (x1, x2) ! (1, 1) node
498in the branching tree, which CHAPTER
creates the11following
INTEGER PROGRAMMING
new subproblems (where constraint 3 disappears because it does not contain x4).
Subproblem 5: Subproblem 6:
Fix x1 ! 1, x2 ! 1, x3 ! 0 so the resulting subproblem reduces to Fix x1 ! 1, x2 ! 1, x3 ! 1 so the resulting subproblem reduces to
Maximize Z ! 14 & 4x4, Maximize Z ! 20 " 4x4,
subject to subject to
(1) 2x4 $ 1 (1) 2x4 # $4
(2), (4) x4 $ 1 (twice) (2) x4 # $0
(5) x4 is binary. (4) x4 # $1
(5) x4 is binary.

LP relaxation:
Variable: x1 x2 LPThe corresponding LP relaxations have the relaxed version of constraint
relaxation:
timal solution, and the bound (when it exists) shown below.
er (5) 0  x4  1 (5) 0  x4  1
branch- x1 = 0 F(3) LP relaxation of subproblem 5:
or the Optimal solution: Optimal solution: None since no feasible
9 ! Z* (5) x1 % 1, x2 % 1, x3 # 0, and 0 # xj # 1
1, 0, 12
(x1 , x2 , x3 ,(0,x41,)0,=1) !1,incumbent , with Z = 16 solution for j ! 1, 2, 3, 4.
All
Bound: Z ≤ 16 Bound: None
! "
x2 = 0 1
Optimal solution: (x , x , x , x ) ! 1, 1, 0, && ,
1 2 3 4 w
2
16
13 Bound: Z # 16.
x1 = 1
( )
1, 0, 4 , 0
5 LP relaxation of subproblem 6:
16 (5) x1 % 1, x2 % 1, x3 % 1, and 0 # xj # 1
x2 = 1 for j ! 1, 2, 3, 4.
Optimal solution: None since there are no feasib
16
(1, 1, 0, 12 ) Bound: None
20
For both of these subproblems, the relaxed version of constraint (5) has the eff
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Branching
x1 x2 x3
variable
Sub1
F(1)
x1 = 0
9 = Z*
(0, 1, 0, 1)
All Sub3
= incumbent
x2 = 0
16 13
(⅚, 1, 0, 1) Sub2 (1, 0, ⅘, 0) Sub5
x1 = 1 x3 = 0
16 16
Sub4
x2 = 1 (1, 1, 0, ½)
16
Sub6
F(3)
x3 = 1

21
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Binary IP Problems

• Iteration 4
– Subproblem 5 is selected for further branching
x4 = 0 : (x1 , x2 , x3 , x4 ) = (1, 1, 0, 0) is feasible, with Z = 14
x4 = 1 : (x1 , x2 , x3 , x4 ) = (1, 1, 0, 1) is infeasible
– The first solution passes fathoming test 1
• Better than the incumbent, so it becomes the new incumbent
with Z* = 14
• Subproblem 3 now passes fathoming test 2 since its bound =
13 ≤ Z* = 14
– The second solution passes fathoming test 3

22
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Branching
x1 x2 x3 x4
variable
Sub1
F(1)
x1 = 0
9
Sub7 F(1)
All Sub3 F(2) x4 = 0
x2 = 0 14 = Z*
16 13 (1, 1, 0, 0)
(⅚, 1, 0, 1) Sub2 (1, 0, ⅘, 0) Sub5 = incumbent
x1 = 1 x3 = 0 (optimal)
16 16
Sub4
x2 = 1 (1, 1, 0, ½)
Sub8
16 F(3)
Sub6 x4 = 1
F(3)
x3 = 1

23
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

• B&B method is basically the same with 4 changes
1. Choice of branching variables
• Before: the next variable in the natural order, i.e. x1, x2, …
• Now: consider only the integer-restricted variables
2. Values assigned to the branching variable
• Before: the binary variable was fixed at 0 or 1
• Now: the general integer variable is dispatched to two ranges
3. Bounding step
• Before: Z’s value was rounded down to obtain the bound
• Now: rounding down may not be necessary
4. Fathoming test 1
• Before: the optimal solution for the LP relaxation is integer
• Now: the integer-restricted variables be integer in the optimal solution
24
 the new incumbent and test 1 is reapplied to all unfathomed subprobl
the new larger Z*.) IEDA 3010
Prescriptive Analytics
Optimality test: Stop when there are no remaining subproblems;
Discrete the current
Optimization 2 incu
optimal.8 Otherwise, perform another iteration.

B&BAnfor Mixed
MIP Example. IPnowProblems
We will illustrate this algorithm by applying it to th
ing MIP problem:
Maximize Z ! 4x1 " 2x2 # 7x3 " x4,
subject to
x1 # 5x3 $ 10
x1 # x2 " x3 $ 1
6x1 " 5x2 " 2x4 $ 0
"x1 5x2 # 2x3 " 2x4 $ 3
and
xj % 0, for j ! 1, 2, 3, 4
xj is an integer, for j ! 1, 2, 3.
Note that the number of integer-restricted variables is I ! 3, so x4 is the only co
variable.

Initialization. After setting Z* ! "&, we form the LP relaxation of this pr

deleting the set of constraints that xj is an integer for j ! 1, 2, 3. Applying the
method to this LP relaxation yields its optimal solution below.
25
5 3 7
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

• Initialization
– Set Z* = -∞
– Form the LP relaxation of the whole problem by deleting
integer restrictions
LP relaxation: (x1 , x2 , x3 , x4 ) = 1 41 , 1 12 , 1 34 , 0 , with Z = 14 14

26
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

• Iteration 1
– Branching variable: x1 (The first integer-restricted variable
that has a non-integer value is x1 = 1¼)
• Subproblem 1: add constraints to original problem
x1  1
LP relaxation has optimal solution:
(x1 , x2 , x3 , x4 ) = 1, 1 15 , 1 45 , 0 , with Z = 14 15
Bound: Z ≤ 14⅕
• Subproblem 2: add constraints to original problem
x1 2
LP relaxation has no feasible solutions
Fathomed by test 3
Bound: None

27
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

Sub1
x1 ≤ 1
14⅕
(1, 1⅕, 1⅘, 0)
All
14¼
(1¼, 1½, 1¾, 0) Sub2
F(3)
x1 ≥ 2

28
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

• Iteration 2
– Branching variable: x2
• Subproblem 3: add constraints to original problem
x1  1 and x2  1
LP relaxation has optimal solution:
5 5
(x1 , x2 , x3 , x4 ) = 6 , 1, 1 6, 0 , with Z = 14 16
Bound: Z ≤ 14⅙
• Subproblem 4: add constraints to original problem
x1  1 and x2 2
LP relaxation has optimal solution:
5 5
(x1 , x2 , x3 , x4 ) = 6 , 2, 1 6 , 0 , with Z = 12 16
Bound: Z ≤ 12⅙

29
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

Sub3
x2 ≤ 1
14⅙
Sub1 (⅚, 1, 1⅚, 0)
x1 ≤ 1
14⅕
Sub4
All x2 ≥ 2
14¼ 12⅙
Sub2 (⅚, 2, 1⅚, 0)
F(3)
x1 ≥ 2

30
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

• Iteration 3
– subproblem 3 has a larger bound so is selected for
branching
– x1 = ⅚ becomes the branching variable
• Subproblem 5: add constraints to original problem
x1  1, x2  1, and x1  0
LP relaxation has optimal solution:
(x1 , x2 , x3 , x4 ) = 0, 0, 2, 12 , with Z = 13 12
Fathomed by test 1, an incumbent (0, 0, 2, ½) with Z* = 13½
• Subproblem 6: add constraints to original problem
x1  1, x2  1, and x1 1
LP relaxation has no feasible solutions
Fathomed by test 3. Bound: None
• Subproblem 4: fathomed by test 2 since its bound 12⅙ ≤ Z*
31
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

B&B for Mixed IP Problems

Sub5
F(1)
x1 ≤ 0
13½
Sub3 (0, 0, 2, ½) =
x2 ≤ 1 incumbent
(optimal)
14⅙
Sub1 Sub6
F(3)
x1 ≤ 1 x1 ≥ 1
14⅕
Sub4
All F(2)
x2 ≥ 2
14¼ 12⅙
Sub2
F(3)
x1 ≥ 2

32
 IEDA 3010
Prescriptive Analytics
Discrete Optimization 2

Summary
• Branching
– BIP: fix the branching variable to either 0 or 1
– MIP: dispatch the branching variable to two ranges
• Ranges are determined by its value in the LP optimal solution
• Bounding
– BIP: rounding down non-integer values
– MIP: rounding down may not be necessary
• Fathoming:
– Test 1
• BIP: LP optimal solution has integer values for all variables
• MIP: LP optimal solution has integer values for the integer-
restricted variables
– Test 2: bound ≤ Z*
– Test 3: no feasible solutions 33