Dynamic Programming

Dr. Chandan Bhar Dept. of Management Studies, Indian School of Mines, Dhanbad Dynamic Programming is a special kind of optimization technique, which subdivides an original problem into number of sub problems as the variables. Then each sub problem is solved individually. The solution of the original problem is obtained by integrating the solutions of each sub problems. Applications of Dynamic Programming The dynamic programming can be applied in many real life situations. Few applications areas of dynamic programming are listed below. a) Capital Budgeting problem b) Stage Coach problem c) Reliability Improvement problem d) Linear programming problem e) Cargo Loading problem Terminologies used in Dynamic Programming Stage i: i th. sub problem of the original problem is known as Stage i. Alternative mi: In stage i, there may be more than one choice of carrying out a task. Each choice is known as an alternative mi. State Variable xi: A possible value of resource within its permitted range at a given stage i is known as state variable xi. Recursive Function [fi(xi)]: A function which links the measure of performance of current stage with the cumulative measure of performance of preceding/succeeding stages as a function of the state variable of the current stage is known as Recursive function of current stage. Let, f1(x1) = max [R(m1)] fi (xi) = max [{R(mi)} + fi-1{xi-c(mi)}] for i = 2,3,.n where, n the total number of stages, R (mi) measure of performance due to alternatives mi of stage i. C (mi) Cost/resources required for alternative mi of stage i

fi (xi)

Value of measure of performance up to the current stage i from stage 1.

Best Recursive Function Value In a given stage i, the lowest value (for minimization problem) of the recursive function for a given value of xi is known as the best recursive function value. Best Alternative in a Given Stage In a given stage i, the alternative corresponding to the best recursive function value for a given value of xi is known as the best alternative for that value of x i. Backward Recursive Function Here computation begins from the last stage and this stage will be numbered as stage 1, while the first sub problem will be numbered as the last stage. Since the recursion proceeds in a backward direction, this type of recursion function is known as backward recursive function. Forward Recursive Function In this case, the computation begins with the sub problem 1 as stage 1 and last sub problem as last stage. The recursion proceeds in forward direction, thus this recursive function is known as forward recursive function.

Capital Budgeting Problem An organization is planning to diversify its business with a maximum outlay of Rs.5 crore. It has identified three different locations to install plants. The organization can invest in one or more of these plants subject to availability of funds. The different possible alternatives and their investments (in crore rupees) and present worth of returns during the useful life (in crores of rupees) of each of these plants are given below. Plant 1 Plant 2 Plant 3 Alternatives Cost Return Cost Return Cost Return 1 0 0 0 0 0 0 2 1 15 2 14 1 3 3 2 18 3 18 2 7 4 4 28 4 21 Find the optimal allocation of capital to different plants, which will maximize the corresponding sum of the present worth of returns. Solution to the Capital Budgeting Problem Maximum capital available 5 crore Each plant is treated as a separate stage Total number of stages in this problem 3 Plant 1 Stage 1 Plant 2 Stage 2 Plant 3 Stage 3 Forward Recursion function is used State variable xi Alternatives mi For Stage 1, state variable is xi and alternatives are m1. Corresponding returns against each state variables is presented below.

Stage 1: Recursive Function: F1(x1) = R (m1) Alternatives m1 State Variable 1 2 3 4 Cost Return Cost Return Cost Return Cost Return f1*(x1) m1* xi 0 0 1 15 2 18 4 28 0 0 0 1 1 0 15 15 2 2 0 15 18 18 3 3 0 15 18 18 3 4 0 15 18 28 28 4 5 0 15 18 28 28 4 Stage 2: Recursive Function: f2 (x2) = R(m2) + f1 (x2 - C(m2)) Alternatives m2 State Variable 1 2 3 4 Cost Return Cost Return Cost Return Cost Return f2*(x2) m2* xi 0 0 2 14 3 18 4 21 0 0 0 1 1 0+15=15 15 1 2 0+18=18 14+0=14 18 1 3 0+18=18 14+15=29 18+0=18 29 2 4 0+28=28 14+18=32 18+15=33 21+0=21 33 3 5 0+28=28 14+18=32 18+18=36 21+15=36 36 3&4

Stage 3: Recursive function: f3 (x3) = R(m3) + f2 (x3 - C(m3)) As all the 5 crore rupees is to be invested, so the state variable for Stage 3 is only 5. Alternatives m3 State Variable 1 2 3 Cost Return Cost Return Cost Return f3*(x3) m3* xi 0 0 1 3 2 7 5 0+36=36 3+33=36 7+29=36 36 1,2&3

Final Result Stage 3 Stage 2 Stage 1 Optimal Alternative C* m3* C* m2* C* m1* 1 2 3 5 1 5-0=5 3 5-3=2 3 3 3 1 5 1 5-0=5 4 5-4=1 2 2 4 1 5 2 5-1=4 3 4-3=1 2 2 3 2 5 3 5-2=3 2 3-2=1 2 2 2 3

Reliability Improvement Problem In electronic equipment, the components are connected either in series or in parallel. If the components are connected in series, then a failure of a component will make the equipment inoperative. The reliability of the equipment can be improved by providing optimal number of standby units to each of the components connected in series such that the total reliability of the equipment is maximized subject to a cost constraint. Example An electronic item has three components connected in series. The reliability of the system is equal to the product of the reliabilities of the three components, i.e., R=r1*r2*r3. The reliability of the system can be improved by providing standby units at extra cost. The details of costs and reliabilities for different number of standby units for each of the components of the system are summarized below. No. of Component 1 Component 2 Component 3 Standby Cost Reliability Cost Reliability Cost Reliability units (Rs.) (Rs.) (Rs.) 1 1 0.75 3 0.84 2 0.8 2 2 0.88 4 0.94 3 0.91 3 4 0.94 6 0.97 5 0.96 The total capital budgeted for this purpose is Rs.9. Determine the optimal number of standby units for each of the components of the system such that the total reliability of the system is maximized. Solution to the Reliability Improvement Problem Maximum capital budgeted = Rs.9 Each Component is considered as a separate stage Backward recursion is used Thus, Component 3 = Stage 1 Component 2 = Stage 2 Component 1 = Stage 3 Range for state variable x1 at stage 1 The money required for one standby unit is Rs.2. Thus the lower limit of state variable is RS.2. But a sum of Rs.4 (Rs.3+Rs.1) is required to have at least one standby units in stage 2 and stage 3. Thus the upper limit of the state variable in stage 1 is 9-4=Rs.5.

Thus the effective range of the state variable x1 is 2 x1 5 Range for state variable x2 at stage 2 Minimum amount of money required to have at least one standby unit in each of the stages up to the current stage is Rs.5 (Sum of cost of one standby unit in stage 1 and stage 2). Thus the lower limit of state variable x2 is Rs.5. Similarly the amount required to have at least one standby unit in stage 3 is Re.1. Thus the upper limit of x2 is 9-1=8. The effective range of x2 is, 5 x2 8 Range for state variable x3 at stage 3 Minimum money required to have at least one stand by unit at each stage is Rs.6. Thus the lower limit of x3 is 6 and as the stage 3 is the last stage, the upper limit of x3 is 9. 6 x3 9 Stage 1 F1(x1) = r (m1) State Alternative m1 Variable 1 2 3 * * X1 Cost Reliability Cost Reliability Cost Reliability f1 (x1) m1 2 0.8 3 0.91 5 0.96 2 0.8 0.8 1 3 0.8 0.91 0.91 2 4 0.8 0.91 0.91 2 5 0.8 0.91 0.96 0.96 3 Stage 2 f2 (x2) = r (m2) + f1(x2-C(m2)) State Alternative m2 Variable 1 2 3 X2 Cost Reliability Cost Reliability Cost Reliability 3 0.84 4 0.94 6 0.97 5 0.84*0.8=0.672 6 0.84*0.91=0.764 0.94* 0.8 = 0.752 7 0.84*0.91=0.764 0.94*0.91=0.855 8 0.84*0.96= 0.806 0.94*0.91=0.855 0.97*0.8=0.776

f2*(x2) m2* 0.672 0.764 0.855 0.855 1 1 2 2

Stage 3 f3 (x3) = r (m3) + f2(x3-C(m3)) State Alternative m3 Variable 1 2 3 X3 Cost Reliability Cost Reliability Cost Reliability 1 0.75 2 0.88 4 0.94 6 0.75*0.672=0.504 7 0.75*0.764=0.573 0.88*0.672=0.591 8 0.75*0.855=0.641 0.88*0.764=0.672 9 0.75*0.855=0.641 0.88*0.855=0.752 0.94*0.672=0.632 Final Result 3 C 9
*

f3*(x3) m3* 0.504 0.591 0.672 0.752 1 2 2 2

Stage 2 m3 2
*

1 m2 2
*

C 9-2=7

C 7-4=3

m1* 2

Alternatives Stage Stage Stage 1 2 3 2 2 2

Thus the optimum number of standby components for each stage is 2.

Stage-Coach (Shortest path) Problem In this problem the objective is to find the shortest distance and the corresponding path from a given source node to a given destination node in a given network. A distance network consists of eleven nodes, which are distributed as shown in the following network. Find the shortest path from Node 1 to Node 11 and also the corresponding distances.

Solution Each pair of adjacent vertical column of nodes is treated as a stage. In this problem there is four stages. Backward recursion is used. In Stage 1 the possible alternative is only one, i.e., Node 11 and the state variables are Node 9 and Node 10. The recursive function f1(x1) for a given combination of the state variable x1 and alternative m1 is as follows. f1(x1) = d (x1,m1) Where, d (x1,m1) is the distance between Node x1 and Node m1. The recursive function for a given combination of the state variable x i and alternative mi in the stage i for i greater than 1 is as below.

fi (xi) = d (xi, mi) + f i-1 (xi-1-mi) Where, d (xi, mi) is the distance between the Nodes xi and mi. fi (xi) is the minimum distance from Node xi in the current stage i to the destination Node (Node 11) in Stage 1. Stage 1 f1(x1) = d (x1,m1) State Variable x1 Alternative m1 11 9 5 10 8 Stage 2 f2 (x2) = d (x2, m2) + f 1 (x1=m2) State Variable x2 6 7 8 Alternative m2 9 10 3+5=8 5+8=13 5+5=10 1+8=9 5+8=13 f2* (x2) 8 9 13 m2* 9 10 10 f1* (x1) 5 8 m1* 11 11

Stage 3 f3 (x3) = d (x3, m3) + f 2 (x2=m3) State Variable x3 2 3 4 5 Alternative m3 7 4+9=13 6+9=15 8+9=17 f3* (x3) 8 1+13=14 12 13 15 14 m3* 6 7 7 8

6 4+8=12 8+8=16 -

Stage 4 f4 (x4) = d (x4, m4) + f 3 (x3=m4) State Variable x4 1 Alternative m4 3 4 7+13=20 4+15=19 f4* (x4) 5 2+14=16 16 m4* 5

2 8+12=20

Thus optimum route is 1 5 8 10 11 The optimum distance is 16 units. Cargo Loading Problem This model deals with the problem of loading items on a vessel with limited capacity. Each item produces a level of revenue. The objective is to load the vessel with most valuable cargo. The backward recursive equation is developed for the general problem of n-item, W ton vessel. Let mi be the number of units of i th. item loaded in the cargo, ri is the revenue per unit from i th. item. The above can be formulated as an integer linear programming problem. Maximize, Z= r1m1+ r2m2+ ..+rnmn Subject to constraints, w1m1+w2m2+w3m3+ +wnmn W mi 0 for i = 1,2,.n Where, wi is the unit weight of i th. item. W is the capacity of the vessel. The elements of the model are as follows. a) Stage i is represented by item i, where i = 1,2,.n b) The alternatives at stage i are represented mi , the number of units loaded in the cargo. The associated return is rimi, Where mi = 0,1,2, largest integer less than (W/wi) The development of recursive equation is a two-step process. Step 1 fi (xi) = max [ rimi + f i+1 (xi+1)] for i = 1,2,.,n mi = 0,1,2,. (W/wi) xi = 0,1,2,.W Step 2 Express xi+1 as a function of xi to ensure that fi (xi) is a function of xi . xi - xi+1 = wimi xi+1 = xi - wimi fi (xi) = max [ rimi + f i+1 (xi - wimi )] for i = 1,2,.,n mi = 0,1,2,. (W/wi) xi = 0,1,2,.W

Example A 4-ton vessel is loaded with one or more of three items. The following table gives the unit weight, wi in tons and the unit revenue ri in thousand dollars for item i. How should the vessel be loaded to maximize the total revenue. Item i wi ri 1 2 31 2 3 47 3 1 14 In this problem wi and W are integers thus state variables xi can assume integer values only. Backward recursion is used, i.e., item 3 is taken as Stage 1 and item 1 as Stage 3. Stage 1 In this stage the state variable x1 can take the values of 0,1,2,3,4 as W=4. f1 (x1) = max (14m1) State 14m1 fi* (xi) Variable x1 m1=0 m1=1 m1=2 m1=3 m1=4 0 0 0 1 0 14 14 2 0 14 28 28 3 0 14 28 42 42 4 0 14 28 42 56 56 Stage 2 f2 (x2) = max [47m2 + f1 (x2-3m2)] max (m2) = 4/3 = 1 State 47m2 + f1 (x2-3m2) f2* (x2) Variable x2 m2=0 m2=1 0 0+0=0 0 1 0+14=14 14 2 0+28=28 28 3 0+42=42 47+0=47 47 4 0+56=56 47+14=61 61 Stage 3 f3 (x3) = max [31m3 + f2 (x3-2m3)] max (m3) = 4/2 = 2

m1* 0 1 2 3 4

m2* 0 0 0 1 1

State Variable x3 0 1 2 3 4

31m3 + f2 (x3-2m3) f3* (x3) m3=0 m3=1 m3=2 0+0=0 0 0+14=14 14 0+28=28 31+0=31 31 0+47=47 31+14=45 47 0+61=61 31+28=59 62+0=62 62

m3* 0 0 1 0 2

So the optimum solution is as follows, 2 units of item 1, 0 unit of item 2, 0 unit of item 3 to loaded in the vessel of 4-ton capacity. Calculus Method of Solution of Dynamic Programming A company has accepted a contract to supply 80 chairs at the end of first month and 120 chairs at the end of second month. The cost of manufacturing chair is represented as 50x + 0.2x2 rupees, where x is the number of units manufactured in that month. If the company manufactures more number of chairs in the first month, there is an inventory holding cost of Rs.8 for each unit carried to the next month. Find the number of chairs to be manufactured in each month to minimize the cost. The company has enough facilities to manufacture 200 chairs per month and that there is no initial inventory. Formulation of the Problem Minimize, f (x1,x2) = (50x1+0.2x12) + (50x2+0.2x22) + 8 (x1-80) Subject to the constraints, x1 80 x1+x2 = 200 x1 0, x20 x1 Number of chairs manufactured in first month. x2 Number of chairs manufactured in second month. Total cost incurred in manufacturing chairs during first month is, R1 (x1) = 50x1+0.2x12 Let optimal solution for x1 gives an inventory of I at the beginning of second month. Thus,

x1* = 80 + I By substituting x1 = x1* in the expression for R1 (x1) we get, f1* = R1 (x1*) = 50(80+I) + 0.2 (80+I)2 = 5280 + 82I + 0.2I2 The cost incurred in manufacturing chairs during second month is, R2 (x1, I) = 50x2+0.2x22 + 8I It includes manufacturing cost plus inventory holding cost for an inventory of quantity I. Thus the total cost of manufacturing 200 chairs can be represented as follows. f2 (I, x2, x1*) = R1 (x1*) + R2 (x2, I) = 5280 + 82I + 0.2I2 +50x2+0.2x22 + 8I = 5280 + 90I + 0.2I2 +50x2+0.2x22 Again during second month, x2 + I = 120 or, I = 120 x2 By putting the value of I in total cost function we get, f2 (I, x2, x1*) = 5280 + 90(120 x2)+ 0.2 (120 x2)2 +50x2+0.2x22 = 18960 88 x2 + 0.4 x22 To optimize the quantity, f2/x2 = 0.8 x2 88 =0 x2* = 110 2f2/x22 = 0.8 > 0 Hence, x2* = 110 corresponds to a minimum value of f2. Thus the optimum production schedule is, First month = 200 110 = 90 chairs Second month = 110 chairs. and inventory at the end of first moth is 10 units. f2* (x2) = Rs.14120/Work Force Size Model In some construction projects hiring and firing are exercised to maintain a labour force that meets the needs of the project. Given that the activities of hiring and firing both incur additional costs, how should the labour force be maintained throughout the life of the project? Let us assume that the project will be executed over the span of n weeks and that the minimum labour force required in week i is bi labourers. Under ideal conditions, we would like the size of the labour force in the week i to be exactly bi. However, depending on the cost parameters, it may be more economical to allow the size of the labour force to vary above and below the minimum requirements.

Given xi is the actual number of labourers employed in week i, two costs can be incurred in week i: 1. C1(xi - bi), the cost for maintaining the excess labour force xi - bi, and 2. C2( xi - xi-1), the cost of hiring an additional xi - xi-1 labourers. The elements of the dynamic programming model are defined as follows. 1. Stage i is represented by week i, where i = 1,2,..,n 2. The alternatives at stage i is xi, the number of labourers in week i. 3. The state at stage i is represented by the number of labourers at stage (week) i-1, i.e., xi-1. The dynamic programming recursive equation is given as, fi (xi-1) = min [C1(xi - bi) + C2( xi - xi-1) + f i+1 (xi)] for i = 1,2,.,n xi bi Where, fn+1(xn) = 0. The computation starts at stage n with xn = bn and terminate at stage 1. Problem for Work Force Determination A construction contractor estimates that the size of the work force needed over the next five weeks to be 5, 7, 8, 4, and 6 workers respectively. Excess labour kept on the force will cost $300 per worker per week and new hiring in any week will incur a fixed cost of $400 plus $200 per worker per week. Expressing C1 and C2 in hundreds of dollars we get, b1 = 5, b2 = 7, b3 = 8, b4 = 4, and b5 = 6 C1(xi - bi) = 3 (xi - bi), xi > bi, for i = 1,2,.5 C2( xi - xi-1) = 4 + 2 ( xi - xi-1), xi > xi-1, for i = 1,2,.5 Stage 5 (b5 = 6) State Variable x4 4 5 6 Stage 4 (b4 = 4) State Variable x3 8 C1 (x4-4) + C2 (x4-x3) + f5 (x4) x4 = 4 x4 = 5 x4 = 6 3*0+0+8=8 3*1+0+6= 9 3*2+0+0=6 Optimum solution f4 (x3) x4* 6 6 C1 (x5-6) + C2 (x5-x4) x5 =6 3*0 + 4 + 2*2 = 8 3*0 + 4 + 2*1 = 6 3*0 + 0 = 0 Optimum solution f5 (x4) x5* 8 6 6 6 0 6

Stage 3 (b3 = 8) State Variable x2 7 8 C1 (x3-8) + C2 (x3-x2) + f4 (x3) x3 =8 3*0 + 4 + 2*1 +6 = 12 3*0 +0 + 0 +6 = 6 Optimum solution f3 (x2) x3* 12 8 6 8

Stage 2 (b2 = 7) State Variable x1 5 6 7 8 C1 (x2-7) + C2 (x2-x1) + f3 (x2) x2 = 7 x2 = 8 3*0+4+2*2+12=20 3*1+4+2*3+6=19 3*0+4+2*1+12=18 3*1+4+2*2+6=17 3*0+0+0+12=12 3*1+4+2*1+6=15 3*0+0+0+12=12 3*1+0+0+6=9 Optimum solution f2 (x1) x2* 19 8 17 8 12 7 9 8

Stage 1 (b1 = 5) State Variable x0 0 C1 (x1-5) + C2 (x1-x0) + f2 (x1) x1 = 5 3*0+4+2*5+ 19= 33 x1 = 6 3*1+4+2*6 +17= 36 x1 = 7 3*2+4+2*7 +12=36 Optimum solution x1 = 8 f1 (x0) x1* 3*3+4+2*8 33 5 +9=38

Thus the optimum Work Force size would be x0 = 0, x1* = 5, x2* = 8, x3* = 8, x4* = 6, and x5* = 6\ Optimal Subdividing Problem Consider a cable of length k units. The objective is to subdivide the cable in n parts, each having a length of pi, where i varies from 1 to n such that product of the lengths of the parts is maximized. A mathematical model for the above situation can be formulated as follows, Maximize, Z = p1*p2*p3*..* pn

Subject to the constraint, p1 + p2 + p3 ++pn = k pi > 0 for i = 1,2,..n The problem can be solved by using dynamic programming technique. Length of the cable = k units Number of parts to be cut out of k units of length = n Part i of the cable corresponds to Stage i of the dynamic programming problem. Length of the part of the cable in Stage i is pi for i = 1,2,..,n Sum of the lengths of the parts of the cable up to Stage i (i.e., p 1 + p2 + p3 ++pi ) is yi. yi is a non-negative variable. Recursive Function f1(y1) = y1 and fi (yi) = maximize [pifi-1(yi-pi)], for i = 2,3,..,n Here, yi - pi is the total length of the cable, which accounts for the stages from 1 to i-1. By substituting the value of k we get, f1(k) = k and f2 (k) = maximize [p2f1(k-p2)] = maximize [p2(k-p2)] As, f1(k-p2) = k-p2 Thus, f2 (k) = maximize [p2k - p22] In order to maximize the value of f2 (k), we find the following, df2 (k)/dp2 = k - 2 p2 = 0, 2 p2 = k , p2 = k/2 f2 (k) = (k/2) [ k (k/2)] = (k/2)2 Similarly we can get, f3 (k) = (k/3)3 .. fn (k) = (k/n)n Thus the optimal length of each part of the cable is p1 = p2 = p3 = .= pn = (k/n) Corresponding value of the objective function is (k/n)n

Problem on Optimal Subdividing Problem Solve the following model using dynamic programming technique for optimal subdivision of a cable of length 10 units into 3 parts, such that the product of their lengths is maximized. The model for the problem is as follows, Maximize, Z = p1*p2*p3 Subject to the constraint, p1 + p2 + p3 = 10 pi > 0 for i = 1,2,3 Solution of the Problem - Length of the cable = 10 units - Number of parts to be cut out of 10 units of cable length = 3 - Part i corresponds to Stage i of the dynamic programming problem, where i = 1,2,3 - yi is the sum of the lengths of the part 1 and up to part i of the cable. Recursive Function f1(y1) = y1 and fi (yi) = maximize [pifi-1(yi-pi)], for i = 2,3 yi - pi is the total length of the cable which accounts for the stages from 1 to i-1. Substitute y1 and yi , for i varying from 2 to 3 with 10 in the first and second functions respectively. f1(10) = 10, and f2 (10) = maximize [p2 f1(10 - p2)] = maximize [p2 (10 - p2)] To maximize f2 (10) we get, df2(10)/dp2 = 10 - 2 p2 = 0 or, p2 = (10/2) f2 (10) = (10/2)[10 (10/2)] = (10/2)2 Similarly, f3 (10) = (10/3)3 Thus the optimal length of each part of the cable is p1= p2 = p3 = 10/3 Optimum objective function value = (10/3)3