Dynamic Programming and

Applications
Reservoir Operation

Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Objectives

 To develop the steady state operational policy of a single

reservoir

 To develop backward recursive equations for this

operational policy

 To demonstrate the method using a numerical example

2 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Reservoir Operation – Steady State
Policy
Consider a single reservoir receiving inflow it and making releases rt for each time
period t

Maximum capacity of the reservoir is K

Reservoir receives benefits from hydropower generation, irrigation, recreation etc

Optimization problem: Find the sequence of releases to be made from the reservoir
that maximizes the total net benefits

St : initial storage for time period t

St+1 : final storage for time period t

Net benefits can be expressed as a function of rt, St and St+1 as

NBt St , St 1 , rt 

3 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Reservoir Operation – Steady State Policy…
Let there be T periods in a year, then the objective function is to maximize the
total net benefits from all periods

Maximize
 NB S , S
t 1
t t t 1 , rt 

The constraints are

Continuity Equation after neglecting all the minor losses and assuming no
overflow
St 1 St  it  rt for t 1,2,...T

Capacity constraint
S t K for t 1,2,...T
4 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc
 Steady State Policy : Backward
Recursion
Take stages as the time periods and the states as the storage volumes

Assume that there are T periods in a year

To find the steady state policy, select a period in a particular year, after
which it is assumed that the reservoir is no longer useful (maximum
lifetime of the reservoir)

Usually in almost all problems, the last period T is taken as the terminal
period

At this stage, the optimal release will be independent of the inflow and
also the net benefit.

5 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Backward Recursion…
Consider the terminal period as T of a particular year after which reservoir is no
longer useful

Solving this problem in a backward recursion method, let t represents the period
in a year from T to 1 and n represents the periods remaining from t till end

t will take values starting from T, decreasing to 1 (which will complete one
year) and then again taking a value of T and repeating the values

Value of n starts from 1 (while considering the Tth period of last year) and while
moving backwards its value keeps on increasing i.e. at the beginning of the last
year, the value of n = T and at the beginning of second last year its value will
be equal to 2T and so on.

6 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Backward Recursion…

Starting from the last period T of a particular year, there is only one period
remaining

The maximum net benefit for this last period is

f T1 S T   max
rT 0
NBT S T , S T  iT  rT , rT 
rT ST iT
rT ST iT  K

This is solved for all ST values from 0 to K

7 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Backward Recursion…
Considering the last two stages together, the objective function is

f T2 1 S T  1   max
rT  1 0
NB
T  1 S T  1 , S T  1  iT  1  
rT  1 , rT  1   f T1 S T  1  iT  1  rT  1 
rT  1 ST  1 iT  1
rT  1 ST  1 iT  1  K

ST-1 also ranges from 0 to K

In general, for a period t with n periods remaining, the function can be written as

f t n St   max NBt St , St  it  rt , rt  f t n1 1 St  it  rt 
rt 0

rt S t it
rt S t it  K

where
the index t decreases from T to 1 and then takes the value T again and the cycle repeats
for another year
the index n starts from 1 and increases at each successive stage

8 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Backward Recursion…

The cycle is repeated till the optimum values of rt for an initial

storage St will be the same as the corresponding rt and St of previous

year

Such a solution is called stationary solutiont S t  f t n S t 

n T
f

Maximum net benefit is the difference of and for

any St and t.

9 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical
Example
Consider a reservoir for which the desirable constant storage is 20 units and
the constant release is 25 units.

Capacity of the reservoir is 30 units.

Inflows for three seasons are 10, 50 and 20 units.

Optimization problem: To find the optimum rt and St that minimizes the total
squared deviation from the release and storage targets given.

Thus, the objective function is

Minimize 20  S   25  r  

t
2
t
2

10 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

St take the discrete values of 0, 10, 20, 30 and rt take the values of 10, 20,
30, 40

Solution:

Backward recursion is used to solve the problem

Consider a year after which the reservoir is no longer useful.

Expressing the problem as a sequential process as shown in the figure below

11 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

 Considering the last period for which t = 3 and n = 1, the optimization

function is
 Minimize 
f 31 S 3   20  S 3   25  r3 
2 2

 Inflow for 3rd season, I3 = 20 units and capacity of the reservoir, K = 30

units
 The release constraints can be expressed as

r3  S 3  I 3 r3  S 3  I 3 K
and
 S 3  20  S 3  20  30

12 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…
Computation for the first subproblem (n = 1)

13 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

 Considering the last two periods (n =2), the optimization function is

 
f 22 S 2   20  S 2   25  r2   f 31 S 2  I 2  r2 
2 2

 Inflow for 2nd season, I2 = 50 units

 Release constraints can be expressed as

r2 S 2  50 andr2 S 2  50  30

 While computing the optimal releases for S2=30,

r2  S 2  50  30 i.e. r2 50

 Since r2 can take values only of 10, 20, 30 and 40 only, the release cannot be made

for S2=30.

14 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

Computation for the second subproblem (n = 2)

15 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

The same procedure is repeated for all stages till n = 7. The summarized
solution for this problem is given in the table below

Solution for the last year

16 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

Solution for the second last year

17 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

Solution for the last period of third last year

18 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy : Numerical Example…

At this stage, the value of r3* at n = 7 and n = 4 are exactly the same. Also the
f 37 S 3   f 34 S 3  275
difference is same for all St

This value is the minimum total squared deviations from the target release
and storage

Stationary optimal policy

obtained is shown

19 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Steady State Policy: Limitations
Assumption in dynamic programming: Decisions made at one stage is

dependent only on the state variable and is independent of the

decisions taken in other stages

Therefore dynamic programming will not be an appropriate technique

where decisions are dependent on the earlier decisions.

20 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Bibliography / Further Reading
1. Bellman, R., Dynamic Programming, Princeton University Press, Princeton, N.J, 1957.

2. Hillier F.S. and G.J. Lieberman, Operations Research, CBS Publishers & Distributors, New
Delhi, 1987.
3. Loucks, D.P., J.R. Stedinger, and D.A. Haith, Water Resources Systems Planning and Analysis,
Prentice-Hall, N.J., 1981.
4. Rao S.S., Engineering Optimization – Theory and Practice, Fourth Edition, John Wiley and

Sons, 2009.
5. Taha H.A., Operations Research – An Introduction, 8th edition, Pearson Education India, 2008.

6. Vedula S., and P.P. Mujumdar, Water Resources Systems: Modelling Techniques and Analysis,
Tata McGraw Hill, New Delhi, 2005.

21 Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc

 Thank You

Water Resources Planning and Management: M4L5 D Nagesh Kumar, IISc