Introduction Formulation of a dynamic programming problem Question References

Discrete Dynamic Programming Problem

Ramkumar Sinha (25)

School of Studies in Mthematics

Pt. Ravishankar Shukla University, Raipur
April 17, 2025
Introduction Formulation of a dynamic programming problem Question References

Contents

1 Introduction

2 Formulation of a dynamic programming problem

3 Question

4 References
Introduction Formulation of a dynamic programming problem Question References

Introduction

Discrete Dynamic Programming (DP) is a powerful method used in solving

complex optimization problems by breaking them down into simpler subproblems. It is
especially useful when the problem involves a sequence of decisions, and the variables
involved take on discrete (finite or countable) values, such as integers.

In many real-world and theoretical problems, we aim to find the optimal

solution—such as the maximum profit, minimum cost, or shortest path—under certain
constraints. Discrete DP helps by storing the results of solved subproblems and reusing
them, thus avoiding redundant computations. This makes it much more efficient than
brute-force methods.
Introduction Formulation of a dynamic programming problem Question References

Formulation of a dynamic programming problem

Formulation of a dynamic programming problem requires the creation of a

recursive relationship. It needs us to define an optimum payoff function for every stage
and state. The recursive relationship establishes the relationship between the optimum
payoff functions at successive stages.
A recursive relationship identifies the optimal policy for stage n, given the optimal
policy for stage (n + 1) available.
Using dynamic programming notation, we define the following:
N = number of stages,
n = index for the current stage (n = 1, 2, ..., N),
(n + 1) = previous stage (it may be noted that successive stages are numbered in
descending order),
sn = state of the system in the current stage for which the recursive relation holds,
Introduction Formulation of a dynamic programming problem Question References

sn+1 = state of the previous stage,

xn = decision variable at stage n,
xn∗ = optimum value of xn (given sn ),
fn (sn , xn ) = total payoff for each alternative, starting from state sn in stage n, to
the end of the process,
fn (sn , xn ) = contribution of stages n, n + 1, n + 2, ..., N to the objective function,
if the system starts in state sn at stage n, immediate decision is xn , and an
optimal decision is made thereafter; the function is known as the stage
transformation function.
fn∗ (sn ) = optimal total payoff from state sn in state n,
∗ (s
fn+1 n+1 ) = optimal total payoff obtained in the previous stage.
Introduction Formulation of a dynamic programming problem Question References

The basic structure of dynamic programming can be described diagrammatically as

shown under:
Stage n Stage (n + 1)
Decision variable xn
State: sn sn+1
Contribution of xn
Value: fn (sn , xn ) ∗ (s
Value: fn+1 n+1 )

Making use of above notations, the recursive relationship (for minimization) for state
sn at Stage n may be stated as under:

fn∗ (sn ) = min pn (xn ) + fn+1

∗
 
(sn+1 ) ,
xn
where, pn (xn ) is the immediate payoff in stage n when decision xn is made for a
specific value of sn .
For dynamic programming problems in which the payoff is sought to be maximized, the
recursive relationship would be:
Introduction Formulation of a dynamic programming problem Question References

Question

An electronic device consists of four components, each of which must be functional for
the system to function. The system reliability can be improved by installing parallel
units in one or more components. The reliability of components, R, with one, two, or
three parallel units, and the corresponding cost, C , are given below. The maximum
amount available for this device is 100. The problem is to determine the number of
parallel units in each component.

Number of Components
Parallel Units 1 2 3 4
R C R C R C R C
1 0.70 10 0.50 20 0.70 10 0.60 20
2 0.80 20 0.70 40 0.90 30 0.70 30
3 0.90 30 0.80 50 0.95 40 0.90 40
Introduction Formulation of a dynamic programming problem Question References

Solution: The reliability of the given electronic device is the product of the reliabilities
of each of its four components. If Rj and ui represent the reliability of the component
j and units in parallel in component i, then the reliability of the whole system,
consisting of n components in series, will be:
R1 u1 × R2 u2 × · · · × Rn un
Since the objective is to maximize the reliability of the system, the problem can be
stated as:
Maximize Z = R1 u1 × R2 u2 × · · · × Rn un
subject to the constraint
c1 u1 + c2 u2 + · · · + cn un ≤ C
where
cj uj = cost of component j
C = total capital available
Introduction Formulation of a dynamic programming problem Question References

Since the electronic device consists of n = 4 components, for solving this problem
consider each component as a stage. The state at any stage will be the capital to be
allocated. Let us adopt the following notations:
xj = capital allocated to stage j, through first stage inclusive
fj (xj ) = return when available capital cj is allocated optimally over n stages
(components)
Rj uj = reliability of component j (j = 1, 2, 3, 4)
Now the recursive equation can be expressed as:

fj (xj ) = max {[R1 u1 × R2 u2 × · · · × Rn un ] · fj−1 (xj − cj uj )} , j = 1, 2, 3, 4.

0≤cj uj ≤xj

The electronic device in this problem will consist of at least one unit in each
component. Thus, the range of investment, xj (j = 1, 2, 3, 4) in each case will be as
follows:
Introduction Formulation of a dynamic programming problem Question References

The constraints:

c11 ≤ x1 ≤ C − c21 − c31 − c41 or 10 ≤ x1 ≤ 50

c11 + c12 ≤ x2 ≤ C − c31 − c41 or 30 ≤ x2 ≤ 70
c11 + c12 + c13 ≤ x3 ≤ C − c41 or 40 ≤ x3 ≤ 80
c11 + c12 + c13 + c14 ≤ x4 ≤ C or 60 ≤ x4 ≤ 100
The computation of return at each stage, using the forward induction approach, is
shown in Tables 22.8 to 22.11.
Table 1 Stage 1(j = 1)
f1 (x1 ) = R1 u1
States x1 Optimal Return f∗
1 (x1 ) Optimal Decision u∗
1
u1 = 1 u1 = 2 u1 = 3
R = 0.70; C = 10 R = 0.80; C = 20 R = 0.90; C = 30
10 0.70 − − 0.70 1
20 0.70 0.80 − 0.80 2
30 0.70 0.80 0.90 0.90 3
40 0.70 0.80 0.90 0.90 3
50 0.70 0.80 0.90 0.90 3
Introduction Formulation of a dynamic programming problem Question References

Table 2 Stage 2(j = 2)

f2 (x2 ) = R2 u2 × f1∗ (x2 − c2 x2 )
States x2 Optimal Return f∗
2 (x2 ) Optimal Decision u∗
2
u2 = 1 u2 = 2 u2 = 3
R = 0.50; C = 20 R = 0.70; C = 40 R = 0.80; C = 50
30 0.5 × 0.7 = 0.35 − − 0.35 1
40 0.5 × 0.8 = 0.40 − − 0.40 1
50 0.5 × 0.9 = 0.45 0.7 × 0.7 = 0.49 − 0.49 2
60 0.5 × 0.9 = 0.45 0.7 × 0.8 = 0.56 0.8 × 0.7 = 0.56 0.56 2, 3
70 0.5 × 0.9 = 0.45 0.7 × 0.9 = 0.63 0.8 × 0.8 = 0.64 0.64 3

Table 3 Stage 3(j = 3)

f3 (x3 ) = R3 u3 × f2∗ (x3 − c3 x3 )
States x3 Optimal Return f∗
3 (x3 ) Optimal Decision u∗
3
u3 = 1 u3 = 2 u3 = 3
R = 0.70; C = 10 R = 0.90; C = 30 R = 0.95; C = 40
40 0.7 × 0.35 = 0.245 − − 0.245 1
50 0.7 × 0.40 = 0.280 − − 0.280 1
60 0.7 × 0.49 = 0.343 0.9 × 0.35 = 0.315 − 0.343 1
70 0.7 × 0.56 = 0.392 0.9 × 0.40 = 0.360 0.95 × 0.35 = 03325 0.392 1
80 0.7 × 0.64 = 0.448 0.9 × 0.49 = 0.441 0.95 × 0.40 = 0.380 0.448 1
Introduction Formulation of a dynamic programming problem Question References

Table 4 Stage 4(j = 4)

f4 (x4 ) = R4 u4 × f3∗ (x4 − c4 u4 )
States x4 Optimal Return f∗
4 (x4 ) Optimal Decision u∗
4
u4 = 1 u4 = 2 u4 = 3
R = 0.60; C = 20 R = 0.70; C = 30 R = 0.90; C = 40
60 0.6 × 0.245 = 0.147 − − 0.147 1
70 0.6 × 0.280 = 0.168 0.7 × 0.245 = 0.171 − 0.171 2
80 0.6 × 0.343 = 0.2058 0.7 × 0.280 = 0.196 0.9 × 0.245 = 0.2205 0.2205 3
90 0.6 × 0.392 = 0.235 0.7 × 0.343 = 0.2401 0.9 × 0.280 = 0.252 0.252 3
100 0.6 × 0.448 = 0.2688 0.7 × 0.392 = 0.274 0.9 × 0.343 = 0.3087 0.308 3

In Table 4, at stage 4, the value of return function f4 (x4 ) is the maximum, i.e. 0.308
at x4 = 100, and u4 = 3. This makes x3 = 100 − 40 = 60. In Table 3, x3 = 60
corresponds to u3 = 1, and we are left with x2 = . . . 60 − 10 = 50. In Table 2, x2 = 50
corresponds to u2 = 2, and we are left with x1 = 50 − 40 = 10. In Table 1, x1 = 10
corresponds to u1 = 1.
Hence, for maximum reliability, the device must have 1, 2, 1, and 3 units in
components 1, 2, 3, and 4, respectively, in order to attain a maximum reliability of
0.308 or 30.8 per cent.
Introduction Formulation of a dynamic programming problem Question References

References

J.K. Sharma, Operations Research: Theory and Applications, Trinity Press, an

imprint of Laxmi Publications Pvt. Ltd., New Delhi, 6th Edition (2016).
Kanti Swarup, P.K. Gupta & Man Mohan, Operations Research: Introduction to
Management Science, Sultan Chand & Sons, 20th Revised Edition (2022).
Introduction Formulation of a dynamic programming problem Question References

Thank You!