Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Discrete Dynamic Programming Problem

Ramkumar Sinha (25)

School of Studies in Mthematics

Pt. Ravishankar Shukla University, Raipur
April 17, 2025
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Contents

1 Introduction

2 Formulation of a dynamic programming problem

3 Question

4 Question (Cargo Load Problem)

5 References
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Introduction

Discrete Dynamic Programming (DP) is a powerful method used in solving

complex optimization problems by breaking them down into simpler subproblems. It is
especially useful when the problem involves a sequence of decisions, and the variables
involved take on discrete (finite or countable) values, such as integers.

In many real-world and theoretical problems, we aim to find the optimal

solution—such as the maximum profit, minimum cost, or shortest path—under certain
constraints. Discrete DP helps by storing the results of solved subproblems and reusing
them, thus avoiding redundant computations. This makes it much more efficient than
brute-force methods.
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Formulation of a dynamic programming problem

Formulation of a dynamic programming problem requires the creation of a

recursive relationship. It needs us to define an optimum payoff function for every stage
and state. The recursive relationship establishes the relationship between the optimum
payoff functions at successive stages.
A recursive relationship identifies the optimal policy for stage n, given the optimal
policy for stage (n + 1) available.
Using dynamic programming notation, we define the following:
N = number of stages,
n = index for the current stage (n = 1, 2, ..., N),
(n + 1) = previous stage (it may be noted that successive stages are numbered in
descending order),
sn = state of the system in the current stage for which the recursive relation holds,
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

sn+1 = state of the previous stage,

xn = decision variable at stage n,
xn∗ = optimum value of xn (given sn ),
fn (sn , xn ) = total payoff for each alternative, starting from state sn in stage n, to
the end of the process,
fn (sn , xn ) = contribution of stages n, n + 1, n + 2, ..., N to the objective function,
if the system starts in state sn at stage n, immediate decision is xn , and an
optimal decision is made thereafter; the function is known as the stage
transformation function.
fn∗ (sn ) = optimal total payoff from state sn in state n,
∗ (s
fn+1 n+1 ) = optimal total payoff obtained in the previous stage.
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

The basic structure of dynamic programming can be described diagrammatically as

shown under:
Stage n Stage (n + 1)
Decision variable xn
State: sn sn+1
Contribution of xn
Value: fn (sn , xn ) ∗ (s
Value: fn+1 n+1 )

Making use of above notations, the recursive relationship (for minimization) for state
sn at Stage n may be stated as under:

fn∗ (sn ) = min pn (xn ) + fn+1

∗
 
(sn+1 ) ,
xn
where, pn (xn ) is the immediate payoff in stage n when decision xn is made for a
specific value of sn .
For dynamic programming problems in which the payoff is sought to be maximized, the
recursive relationship would be:
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Question

An electronic device consists of four components, each of which must be functional for
the system to function. The system reliability can be improved by installing parallel
units in one or more components. The reliability of components, R, with one, two, or
three parallel units, and the corresponding cost, C , are given below. The maximum
amount available for this device is 100. The problem is to determine the number of
parallel units in each component.

Number of Components
Parallel Units 1 2 3 4
R C R C R C R C
1 0.70 10 0.50 20 0.70 10 0.60 20
2 0.80 20 0.70 40 0.90 30 0.70 30
3 0.90 30 0.80 50 0.95 40 0.90 40
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Solution: The reliability of the given electronic device is the product of the reliabilities
of each of its four components. If Rj and ui represent the reliability of the component
j and units in parallel in component i, then the reliability of the whole system,
consisting of n components in series, will be:
R1 u1 × R2 u2 × · · · × Rn un
Since the objective is to maximize the reliability of the system, the problem can be
stated as:
Maximize Z = R1 u1 × R2 u2 × · · · × Rn un
subject to the constraint
c1 u1 + c2 u2 + · · · + cn un ≤ C
where
cj uj = cost of component j
C = total capital available
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Since the electronic device consists of n = 4 components, for solving this problem
consider each component as a stage. The state at any stage will be the capital to be
allocated. Let us adopt the following notations:
xj = capital allocated to stage j, through first stage inclusive
fj (xj ) = return when available capital cj is allocated optimally over n stages
(components)
Rj uj = reliability of component j (j = 1, 2, 3, 4)
Now the recursive equation can be expressed as:

fj (xj ) = max {[R1 u1 × R2 u2 × · · · × Rn un ] · fj−1 (xj − cj uj )} , j = 1, 2, 3, 4.

0≤cj uj ≤xj

The electronic device in this problem will consist of at least one unit in each
component. Thus, the range of investment, xj (j = 1, 2, 3, 4) in each case will be as
follows:
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

The constraints:

c11 ≤ x1 ≤ C − c21 − c31 − c41 or 10 ≤ x1 ≤ 50

c11 + c12 ≤ x2 ≤ C − c31 − c41 or 30 ≤ x2 ≤ 70
c11 + c12 + c13 ≤ x3 ≤ C − c41 or 40 ≤ x3 ≤ 80
c11 + c12 + c13 + c14 ≤ x4 ≤ C or 60 ≤ x4 ≤ 100
The computation of return at each stage, using the forward induction approach, is
shown in Tables 22.8 to 22.11.
Table 1 Stage 1(j = 1)
f1 (x1 ) = R1 u1
States x1 Optimal Return f∗
1 (x1 ) Optimal Decision u∗
1
u1 = 1 u1 = 2 u1 = 3
R = 0.70; C = 10 R = 0.80; C = 20 R = 0.90; C = 30
10 0.70 − − 0.70 1
20 0.70 0.80 − 0.80 2
30 0.70 0.80 0.90 0.90 3
40 0.70 0.80 0.90 0.90 3
50 0.70 0.80 0.90 0.90 3
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Table 2 Stage 2(j = 2)

f2 (x2 ) = R2 u2 × f1∗ (x2 − c2 x2 )
States x2 Optimal Return f∗
2 (x2 ) Optimal Decision u∗
2
u2 = 1 u2 = 2 u2 = 3
R = 0.50; C = 20 R = 0.70; C = 40 R = 0.80; C = 50
30 0.5 × 0.7 = 0.35 − − 0.35 1
40 0.5 × 0.8 = 0.40 − − 0.40 1
50 0.5 × 0.9 = 0.45 0.7 × 0.7 = 0.49 − 0.49 2
60 0.5 × 0.9 = 0.45 0.7 × 0.8 = 0.56 0.8 × 0.7 = 0.56 0.56 2, 3
70 0.5 × 0.9 = 0.45 0.7 × 0.9 = 0.63 0.8 × 0.8 = 0.64 0.64 3

Table 3 Stage 3(j = 3)

f3 (x3 ) = R3 u3 × f2∗ (x3 − c3 x3 )
States x3 Optimal Return f∗
3 (x3 ) Optimal Decision u∗
3
u3 = 1 u3 = 2 u3 = 3
R = 0.70; C = 10 R = 0.90; C = 30 R = 0.95; C = 40
40 0.7 × 0.35 = 0.245 − − 0.245 1
50 0.7 × 0.40 = 0.280 − − 0.280 1
60 0.7 × 0.49 = 0.343 0.9 × 0.35 = 0.315 − 0.343 1
70 0.7 × 0.56 = 0.392 0.9 × 0.40 = 0.360 0.95 × 0.35 = 03325 0.392 1
80 0.7 × 0.64 = 0.448 0.9 × 0.49 = 0.441 0.95 × 0.40 = 0.380 0.448 1
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Table 4 Stage 4(j = 4)

f4 (x4 ) = R4 u4 × f3∗ (x4 − c4 u4 )
States x4 Optimal Return f∗
4 (x4 ) Optimal Decision u∗
4
u4 = 1 u4 = 2 u4 = 3
R = 0.60; C = 20 R = 0.70; C = 30 R = 0.90; C = 40
60 0.6 × 0.245 = 0.147 − − 0.147 1
70 0.6 × 0.280 = 0.168 0.7 × 0.245 = 0.171 − 0.171 2
80 0.6 × 0.343 = 0.2058 0.7 × 0.280 = 0.196 0.9 × 0.245 = 0.2205 0.2205 3
90 0.6 × 0.392 = 0.235 0.7 × 0.343 = 0.2401 0.9 × 0.280 = 0.252 0.252 3
100 0.6 × 0.448 = 0.2688 0.7 × 0.392 = 0.274 0.9 × 0.343 = 0.3087 0.308 3

In Table 4, at stage 4, the value of return function f4 (x4 ) is the maximum, i.e. 0.308
at x4 = 100, and u4 = 3. This makes x3 = 100 − 40 = 60. In Table 3, x3 = 60
corresponds to u3 = 1, and we are left with x2 = . . . 60 − 10 = 50. In Table 2, x2 = 50
corresponds to u2 = 2, and we are left with x1 = 50 − 40 = 10. In Table 1, x1 = 10
corresponds to u1 = 1.
Hence, for maximum reliability, the device must have 1, 2, 1, and 3 units in
components 1, 2, 3, and 4, respectively, in order to attain a maximum reliability of
0.308 or 30.8 per cent.
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Question 2:(Cargo Load Problem)

A vessel is to be loaded with stocks of 3 items. Each unit of item ’n’ has a weight wn
and value vn . The maximum cargo weight the vessel can take is 5, and the details of
the three items are as follows:
Item (n) Weight (wn ) in tons Value (vn )
1 1 30
2 3 80
3 2 65

Develop the recursive equation for the above case and find the most valuable cargo
load without exceeding the maximum cargo weight by using dynamic programming.
[B.Tech. (Nov.), Hyderabad Univ. 2010]
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Solution
We have to determine how many units of three items are to be loaded. So, it is a
three-stage problem. Define:
n: the stage (item) under consideration,
xn : the number of units of item of type n (n = 1, 2, 3),
W : maximum capacity available,
sn : state at stage n, indicating the weight available.
(
Let fn (sn ) denote the value of optimal allocation for the three types of items. If fn sn )
(
be the value associated with optimum solution fn sn ), n = 1, 2, 3; then

fn∗ (sn ) = max{fn (sn , xn )}

xn
∗
and fn (sn ) = max{pn (xn ) + fn−1 (sn − wn xn )}, n = 1, 2, 3;
xn

where, pn (xn ) denote the expected value obtained from allocation of xn units of weight
to item ’n’.
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

For one-stage problem (i.e., for one item cargo loading), we write

f1 (s1 ) = max{30x1 },
x1
h i
W
5
where the largest value of x1 is w1 = 1 = 5. We have the following tabular
computations:

s3 f1 (s1 , x1 ) = {30x1 } Optimum solution

x1 = 0 x1 = 1 x1 = 2 x1 = 3 x1 = 4 x1 = 5 f3∗ (s3 ) x3∗
0 0 0 0
1 0 30 30 1
2 0 30 60 60 2
3 0 30 60 90 90 3
4 0 30 60 90 120 120 4
5 0 30 60 90 120 150 150 5
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

5
For 2-stage problem, the largest value of x2 is 3 = 1, and

f2 (s2 ) = max{80x2 + f1∗ (s2 − 3x2 )}

x2

s3 f2 (s2 , x2 ) = {80x2 + f1∗ (s2 − 3x2 )} Optimum solution

x2 = 0 x2 = 1 f2∗ (s2 ) x2∗
0 0+0=0 0 0
1 0 + 30 = 30 30 0
2 0 + 60 = 60 60 0
3 0 + 90 = 90 80 + 0 = 80 90 0
4 0 + 120 = 120 80 + 30 = 110 120 0
5 0 + 150 = 150 80 + 60 = 140 150 0
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

5
For 3-stage problem, the largest value of x3 is 2 = 2, and

f3 (s3 ) = max{65x3 + f2∗ (s3 − 2x3 )}

x3

s3 f3 (s3 , x3 ) = (65x3 + f2∗ (s3 − 2x3 )) Optimum solution

x3 = 0 x3 = 1 x3 = 2 f3∗ (s3 ) x3
0 0+0 0 0
1 0 + 30 30 0
2 0 + 60 65 + 0 = 65 65 1
3 0 + 90 65 + 30 = 95 95 1
4 0 + 120 65 + 60 = 125 130 + 0 = 130 130 2
5 0 + 150 65 + 90 = 155 130 + 30 = 160 160 2

Given W = 5, the optimum solution, therefore, is x3∗ = 2. This gives the allocation of
2 units of item 3 (weighing 2 tons). The remaining 5 − 2 × 2, i.e., one ton is allocated
in an optimal manner according to stage 2.
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

For this quantity, we have x2∗ = 0, so that no unit of item 2 is included. Proceeding to
stage 3, the optimal allocation is x1∗ = 1 (measuring 1 ton).
Hence, the optimum solution is to load 2 units of item 3, 1 unit of item 1 and no unit
of item 2. The maximum value is 2 × 65 + 1 × 30, i.e., 160 units.
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

References

J.K. Sharma, Operations Research: Theory and Applications, Trinity Press, an

imprint of Laxmi Publications Pvt. Ltd., New Delhi, 6th Edition (2016).
Kanti Swarup, P.K. Gupta & Man Mohan, Operations Research: Introduction to
Management Science, Sultan Chand & Sons, 20th Revised Edition (2022).
Introduction Formulation of a dynamic programming problem Question Question (Cargo Load Problem) References

Thank You!