OPERATIONS RESEARCH

Chapter 7

DYNAMIC PROGRAMMING

Lecturer: Assoc. Prof. HỒ THANH PHONG

Department ISE

1 Dynamic Programming
 GIÔÙI THIEÄU

GS Richard Bellman

2 Dynamic Programming
 INTRODUCTION
⚫ Dynamic Programming (DP) is a special technique for solving the
optimization problem. DP is applied to large, complex problems in
which the decision-maker needs to make a series of decisions.
⚫ The DP divides the problem into decision stages The decision of
each stage will affect the decision of the next stage.
⚫ At each stage, the principle of recursive will be applied: the policy
equation properly applied to each stage.
⚫ This technique is very useful for multi-stage decision-making
problems in business such as:
‐ Determine the number of labors in production over the periods.
‐ Distribution of investment capital
‐ Distribute salespeople to each region.
‐ Assess investment opportunities

3 Dynamic Programming
 4 STEPS IN DP
1. Divide the root problem into small problems or stages.

2. Solve the problem each stage so as to satisfy all conditions or

states.

3. Solve the first stage to final if we apply forward computation,

final stage to the first if we apply backward computation.

4. To identify optimal policies and solutions for each stage, using

the principle of recursion applicable to other stages.

4 Dynamic Programming
 SHORTEST PATH PROBLEM
This is a typical problem using DP.
Example:
Mr. Tim wants to go from city A (node 1) to H (node 8), between these
two cities there are many small towns. The map is shown in Figure 1.
He wants to find the shortest path. Help find the shortest route using DP.
- Nodes on the network describe towns.
- Arcs describe paths.
- The distance between the nodes is given on the bows.
This article can use IP, however, we will apply DP to solve.

5 Dynamic Programming
 SHORTEST PATH PROBLEM

D 10 F
4 6

5 11
4 5

E
3 5 H
2 8 2
A C 8
1 3

8
8
5 G
15 7
B
2

Figure 1. Network diagram

6 Dynamic Programming
 We can solve it with two approaches: forward or backward.
The first step is to divide the network into stages, each of which has
Inputs
Equations that show activity
Output
The equation shows the activity:
Shortest distance from start node to node j: 𝐹(𝑗)
Distance from node i to node j: 𝑑𝑖𝑗
We can solve it with two approaches: forward or backward.

𝐹(𝑗) = min 𝐹(𝑖) + 𝑑𝑖𝑗

𝑖 𝑡𝑜 𝑗

7 Dynamic Programming
 SHORTEST PATH PROBLEM
Step 1: Divide the problem into stages
Stage 1 Stage 2 Stage 3 Stage 4

D 10 F
4 6
5
5 11
4 E
5

3 H
A 2 C 8
2
1 3 8
8
8
5

B
15 G
2
7

8 Dynamic Programming
 SHORTEST PATH PROBLEM
Forward computation:
Stage 1 Stage 2 Stage 3 Stage 4

4 4 4 10 10
D D D 10 F F
4 4 4 6 6
5 5
5 5 11
4 E E
5 5
20
2 2 H
A 2 C C 3 8
2
1 3 3
8
8

8
5
5 5 12 12
B B 15 G G
2 2 7 7

Shortest distance to node H(8) is 20.

Shortest route is: A (1) →C(3) → E(5) →F(6) →G(7) → H (8)
9 Dynamic Programming
 SHORTEST PATH PROBLEM
⚫ F(A-1) = 0 = F0 = 0
⚫ F(B-2) = Min(0+5) = 5; F(C-3) = Min(0+2) = 2; F(D-4) = Min(0+4) = 4
F1 = Min(5, 2, 4) = 2
⚫ F(E-5) = Min 4,3 (4+5, 2+3) = 5 (go through node 3-C);
F2 = 5
⚫ F(F-6) = Min 4,5,2 (4+10, 5+5, 5+8) = 10 (go through node 5 –E)
⚫ F(G-7) = Min 2,5,6 (5+15, 5+8, 10+2) = 12 (go through node 6 –F);
F3 = 10
⚫ F(H-8) = Min 6, 7 (10+11, 12+8) = 20 (go through node 7 –G);
F4 = 20

Shortest distance to node H(8) is 20,

Shortest route is: A (1) →C(3) → E(5) →F(6) →G(7) → H (8)

10 Dynamic Programming
 SHORTEST PATH PROBLEM
Backward computation:
Stage 1 Stage 2 Stage 3 Stage 4

D 10 F
4 6
5
5 11
4 E
5

3 H
A 2 C 8
2
1 3 8
8
8
5

B
15 G
2
7

11 Dynamic Programming
 SHORTEST PATH PROBLEM
• F(H-8) = Min 6,7 {F(F-6)+11, F(G-7)+8} (1)
• F(G-7) = Min 2,5,6 {F(B-2)+15, F(E-5)+8+F(F-6)+2} (2)
• F(F-6) = Min 4,5,2 {F(D-4)+10, F(E-5)+5, F(B-2)+8} (3)
• F(E-5) = Min 4,3 {F(D-4)+5, F(C-3)+3} (4)
• F(D-4) = Min(F(A-1) +4) = 4; F(C-3) = Min(F(A-1) +2) = 2
• F(B-2) = Min(F(A-1) +5) = 5
Submit F(D-4) and F(C-3) into (4) we obtain:
• F(E-5) = Min 4,3 {4+5, 2+3}= 5 (through node C-3). Submit to (3),
• F(F-6) = Min 4,5,2 {4+10, 5+5, 5+8}= 10 (through node E-5). Submit to (2)
F(G-7) = Min 2,5,6 {5+15, 5+8+10+2}= 12 (through node F-6).
• Submit to (1) F(H-8) = Min 6,7 {10+11, 12+8}=20 (through node G-7)
Shortest distance to node H(8) is 20,
Shortest route is: A (1) →C(3) → E(5) →F(6) →G(7) → H (8)
12 Dynamic Programming
 Important Terminology
Stage: a period or a sub-problem.
State variables: conditions or states of a stage. The state variable is also
known as the input variable.
Decision variables: choices or decisions at a stage
Return of the Stage: the return function reflects the objective of
decision.
Forward

13 Dynamic Programming
 Forward Decision Decision Decision
d1 di dn

S2 Si Si+1 Sn
Stage 1 ... Stage i ... Stage n
State 2 State i State i+1 State n

Return Return
Return
r1 ri 𝑆𝑖+1 = 𝑆𝑖 − 𝑤𝑖 𝑑𝑖 rn

Backward
Decision Decision Decision
dn di d1

Sn S i+1 Si S2
Stage n ... Stage i ... Stage 1
State n State i+1 State i State 2

Return Return
Return
rn ri r1
𝑆𝑖 = 𝑆𝑖+1 − 𝑤𝑖 𝑑𝑖
14 Dynamic Programming
 Knapsack/Fly-Away kit/
Cargo-Loading Model Problem
⚫ The knapsack model classically deals with determining the most
valuable items a combat soldier carries in a backpack. The
objective is to maximize the total effectiveness of soldier.
⚫ The problem represents a general resource allocation model in
which limited resources are used by a number of economic
activities. The objective is to maximize the total return.
⚫ Other examples are choosing items to place in the cargo
compartment of an airplane or selecting which payloads to put
on the next NASA Space Shuttle are examples. The objective is
to maximize the total return.
⚫ The restriction can be volume, weight, or both.

15 Dynamic Programming
 Knapsack Problem
⚫ The recursive equation is developed for the general problem of al-
locating n items to a knapsack with weight capacity W. Let di be the
number of units of item i in the knapsack, and define ri and wi as the
unit revenue and weight of item i. The general problem can be
represented as:
𝑀𝑎𝑥. 𝑍 = 𝑟1 𝑑1 + 𝑟2 𝑑2 + . . . + 𝑟𝑛 𝑑𝑛
𝑆𝑢𝑏. 𝑡𝑜:
𝑤1 𝑑1 + 𝑤2 𝑑2 + ⋯ + 𝑤𝑛 𝑑𝑛 ≤ 𝑊
The three elements of the model are
⚫ Stage i is represented by item i, i = 1, 2, … , n.
⚫ State of stage i is the total remaining weights si .
⚫ The decision at stage i are the number of units of item i, di.
⚫ The return for di is ri di.

16 Dynamic Programming
 Knapsack Problem - Recursive equation
Define function 𝑓𝑖 is max of return from beginning to stage i.
Forward recursion
𝑓𝑖 (𝑆𝑖+1 ) = 𝑀𝑎𝑥. 𝑜𝑓 𝑟𝑒𝑡𝑢𝑟𝑛 𝑡𝑜 𝑠𝑡𝑎𝑔𝑒 𝑖 = 𝑀𝑎𝑥𝑓𝑒𝑎𝑠𝑖𝑏𝑙𝑒 𝑑𝑖 𝑓 𝑆𝑖 + 𝑟𝑖 𝑑𝑖
Note: 𝑆𝑖+1 = 𝑆𝑖 − 𝑤𝑖 𝑑𝑖
where ( 𝑤𝑖 𝑑𝑖 ) is the weights of total item (i)
𝑊
𝑑𝑖 = 0, 1, … , [ ](or max number available) is the number of item
𝑤𝑖
(i) will be taken. Decision
di

Si Si+ 1
Stage i
State i State i+1

Return
ri

17 Dynamic Programming
 Backward recursion
𝑓𝑖 (𝑆𝑖 ) = 𝑀𝑎𝑥. 𝑜𝑓 𝑟𝑒𝑡𝑢𝑟𝑛 𝑡𝑜 𝑠𝑡𝑎𝑔𝑒 𝑖 = 𝑀𝑎𝑥𝑓𝑒𝑎𝑠𝑖𝑏𝑙𝑒 𝑑𝑖 𝑓 𝑆𝑖+1 + 𝑟𝑖 𝑑𝑖
Note: 𝑆𝑖 = 𝑆𝑖+1 − 𝑤𝑖 𝑑𝑖
where ( 𝑤𝑖 𝑑𝑖 ) is the weights of total item (i)
𝑊
𝑑𝑖 = 0, 1, … , [ ] (or max number available) is the number of item
𝑤𝑖
(i) will be taken. Decision
di

S i+1 Si
Stage i
State i+1 State i

Return
ri

18 Dynamic Programming
 Knapsack Problem - Example
⚫ There is ships cargo by plane from USA to Canada. The capacity for one of
the flights is 4 tons and there are 3 items that can ship. Data is given in Table
7.1.
Table 7.1
Item Weight wi Profit/Unit
1 2 $31
2 3 47
3 1 14

⚫ Problem Setup: This problem will be solved using dynamic programming.

Use Backward computation: Stage 3 is item 3, stage 2 is item 2, stage 1 is item
1.

19 Dynamic Programming
 Stage 3.The exact weight to be allocated to stage 3 (item 3) is not known
in advance but can assume one of the values d3= 0, 1, . . , 4 (because W
= 4 tons and w3 = 1 ton).
The revenue for item 3 is 14d3. Thus, the recursive equation for stage 3 is
𝑓 𝑆3 = 𝑀𝑎𝑥(14𝑑3 )

𝑓 𝑆3 = 𝑀𝑎𝑥(14𝑑3 ) Optimum value

𝑆3 𝑑3 = 0 𝑑3 = 1 𝑑3 = 2 𝑑3 = 3 𝑑3 = 4 𝑓 𝑆3 𝑑3 *
0 0 - - - - 0 0
1 0 14 - - - 14 1
2 0 14 28 - - 28 2
3 0 14 28 42 - 42 3
4 0 14 28 42 56 56 4

20 Dynamic Programming
 Item Weight wi Profit/Unit
1 2 $31
2 3 47
3 1 14

Stage 2: Because max(d2) = [4/3] = 1, so d2 will take value of 0 and 1

𝑓 𝑆3 = 𝑀𝑎𝑥(47𝑑2 + 𝑓(𝑆2 − 3𝑑2 )) Optimum value

𝑆2 𝑑2 = 0 𝑑2 = 1 𝑓 𝑆2 𝑑2 *
0 0+0 = 0 - 0 0
1 0+14 = 14 - 14 0
2 0+28 = 28 - 28 0
3 0+42 = 42 47+0=47 47 1
4 0+56 = 56 47+14 = 61 61 1

21 Dynamic Programming
 Stage 1: Because max(d1) = [4/2] = 2, so d1 will take value of 0, 1, 2 and
𝑓 𝑆1 = 𝑀𝑎𝑥(31𝑑1 + 𝑓(𝑆1 − 2𝑑1 ))

𝑓 𝑆1 = 𝑀𝑎𝑥(31𝑑1 + 𝑓(𝑆1 − 2𝑑1 )) Optimum value

𝑆1 𝑑1 = 0 𝑑1 = 1 𝑑1 = 2 𝑓 𝑆1 𝑑1 *
0 0+0 = 0 - - 0 0
1 0+14 = 14 - - 14 0
2 0+28 = 28 31+0-31 - 31 1
3 0+47 = 47 31+14=45 - 47 0
4 0+61 = 61 31+28 = 59 62+0=62 62 2

The optimum solution is determined in the following manner: given W

=4 ton, in stage 1, d1 =2, occupied 2x2 = 4 tons. It leaves 0 ton for
stage 2 and stage 3, therefor d2 =0 and d3 =0.
Optimum value of return is 62.

22 Dynamic Programming
 Exercises
1. A 4 item cargo loading is given, weighting capacity is 10 ton.
Item Weight wi Profit/Unit Max
number
1 1 $3 6
2 4 9 1
3 3 8 2
4 2 5 2

Solve the problem using backward and forward computation

2. Resolve the example using forward computation.

23 Dynamic Programming