Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

A Petri Net Representation for Dynamic Programming Problems in Management

Applications

Holger Mauch
University of Hawaii at Manoa
Dept. of Information and Computer Science
1680 East-West Road
Honolulu, HI 96822
[email protected]

Abstract representation introduced in this paper it becomes possi-

ble to automatically solve many practical problem instances
Dynamic programming (DP) is a very general optimiza- with DP. This would constitute a considerable improvement
tion technique, which can be applied to numerous man- of today’s use of DP, where usually for each problem at
agement decision problems. In order to develop a soft- hand specialized software development takes place — re-
ware system that automates many of the tasks a user en- peatedly developing the same subtasks over and over again.
counters when attempting to solve an instance of an opti- In addition, with the proposed system, the user no longer
mization problem with discrete DP an intermediate prob- needs skills in software development to solve his problem.
lem representation in the form of a Petri net (PN) turns out Having insight into the nature of the problem to be solved,
to be useful. The specialized PN model presented in this he needs to be able to give a precise DP specification (in
paper captures the essential components of a DP problem particular the DP functional equation) of the problem. That
instance. It uses the standard semantics of place/transition is, it is assumed that the DP model is given by the user. The
nets, a low-level PN class, whereas previous work [3, 4, 5] DP model is then translated into the the intermediate PN
relied on high-level PNs. This approach is illustrated by a representation presented in this paper. Using the PN model
simple financing example, but the methodology works for a as input, transformation software can then produce an out-
wide range of management problems and can be applied to put that solves the problem. In other words, the PN model
more complex instances. Among the benefits of this repre- is used as an internal computer representation of a DP func-
sentation are the possibility to perform consistency checks tional equation. Naive users would not need to be aware
on the PN level and the existence of a simple procedure to of the internal PN model. Because of this transparency,
translate a model instance into executable code that could they need only perform the DP modeling to construct the
be integrated into existing solvers. Also, a software sys- DP functional equation required by our sofware system.
tem currently under development will automate the task of The suggested PN model is only applicable to “discrete”
transforming a DP functional equation into the PN model optimization problems for which a DP functional equation
suggested in this paper. Users need not construct the PN with a finite number of states and decisions can be ob-
model directly. tained. Continuous DP problems are not supported by our
PN model. Other than this, there are no known theoretical
restrictions.
1. Overview and Motivation The computational requirements of the approach pre-
sented here are asymptotically the same as the requirements
Operations Research (OR) deals with the efficient or op- for other DP codes implementing the DP functional equa-
timal allocation of resources and is thus of considerable im- tion directly, since the PN model as given may be regarded
portance in practice. For the optimization of multistage de- as just another representation of a DP functional equa-
cision processes dynamic programming (DP) is a powerful tion. So for example, since a DP solution of the traveling-
technique to determine an optimal policy. salesman problem is inefficient (exponential-time), so is a
After modeling a discrete DP problem and transforming PN modeled solution.
it — with software support — into the Petri net (PN) model The value of a system as suggested here becomes appar-

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 1

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

ent when we survey currently existing software supporting ations of the simplex algorithm), the nonlinear solver (us-
the solution of DP problems in section 3. There seems to be ing succesive linear programming (SLP) or generalized re-
no system that can apply DP to a wide variety of problems. duced gradient (GRG) algorithms), or the branch and bound
Current software systems can only solve a narrow class of solver.
DP problems with a rather simple DP functional equation. DP software accompanying textbooks such as [2] does
The suggested model is illustrated by a simple financing not generalize well. It is usually restricted to instances of
problem to display real-world relevance. As a second exam- one problem.
ple a typical textbook type of DP problem has been chosen: Microsoft Excel and mathematical packages such as
chained matrix multiplication. Matlab and Mathematica have only limited capabilities to
solve optimization problems.
2. Previous Work
4. Introduction to Petri Nets
While there are several papers addressing the use of DP
to solve PN problems, there is little prior work addressing Petri Nets are named for Prof. Dr. Carl Adam Petri, Uni-
the use of PN to solve DP problems. versity of Hamburg, Germany, who developed schemata
Richard [8] showed that an integer linear program (ILP) that allowed the modeling of parallel and independent
can be associated with sequential timed Petri nets (STPN). events in an illustrative manner. The following formal defi-
The decision variables of the ILP correspond to the PN’s nitions follow the notation used in [7].
characteristic vector, i.e. the vector which tells how often The first definition desribes the basic syntactic properties
each transition is fired. Linearity is necessary for this model that a net, as used in the following, should have.
and it is only determined that a solution in the form of a fir-
ing sequence within the PN exists, but not how to construct Definition 4.1 (Net)
such a firing sequence. A triple N = (S, T, F ) is called a net iff
Mikolajczak and Rumbut [5] formally specify dis-
tributed DP problems with colored PNs. The examples used 1. S and T are disjoint sets and
are the optimal polygon triagulation problem and the op-
timal binary search tree problem. One of the conclusions 2. F ⊆ (S × T ) ∪ (T × S) is a binary relation called the
is that the sugested PN model allows reuse of concurrency flow relation of N.
patterns and is thus useful for problems to be computed on
distributed systems. Mathematically speaking our notion of a net is a directed
bipartite graph. For convenience we use the following nota-
tion.
3. Software Systems for Dynamic Program-
ming Problems Definition 4.2
Let N = (S, T, F ) be a net. For x ∈ S ∪ T
We briefly survey existing software system that enable •x = {y|(y, x) ∈ F } is called the preset of x,
the user to solve instances of DP problems. x• = {y|(x, y) ∈ F } is called the postset of x.
Attempts to develop general-purpose DP computer codes
have apparently been made by Hastings [1] and others.
In this paper we use a low-level PN class called
While Hasting’s “DYNACODE” is mentioned in many ref-
place/transition net.
erence lists, there is no readily available information about
it. Sniedovich characterizes these codes in [9, p.193] as
Definition 4.3 (Place/Transition Net)
problem specific and concludes that “no general-purpose
Let N denote the set of nonnegative integers, and
dynamic programming computer codes seem to be available
let ω denote an infinite capacity. A 6-tuple N =
commercially.”
(S, T, F, K, M, W ) is called a place/transition net iff
The optimization package system LINGO / LINDO
solves optimization problems across a wide problem do- 1. (S, F, T ) is a finite net,
main with a variety of solution methods. While its strength
lies in the areas of linear programming, and to some ex- 2. K : S → N ∪ {ω} gives a (possibly unlimited) capac-
tent in integer programming and nonlinear programming, it ity for each place,
is only of very limited use to tackle DP problems. After a
general specification of the optimization problem, LINGO 3. W : F → N − {0} attaches a weight to each arc of
determines whether to employ the linear solver (using vari- the net, and

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 2

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

4. M : S → N ∪ {ω} is the initial marking where all Here s denotes a state, f is the minimization functional,
the capacities are respected, i.e. M (s) ≤ K(s) for all a is a decision that can be made in state s, d is the decision
s ∈ S. cost function, and s(k) are the K next states, and s0 are base
case (initial condition) states. The PN model corresponding
Elements of S are called places, elements of T are called to this DP problem has
transitions.
1. a state place ps for each state s, that has, except for the
Graphically, places will be represented by circles; tran- base case state places, a unique minimization transition
sitions will be represented by squares. If p ∈ S is a place, in its preset,
then the fact that p contains k tokens is represented by writ-
ing the number k inside p’s circle. The weighted arcs of the 2. a min-in place pm for each decision a, that has a min-
flow relation are represented by weighted directed edges in imization transition in its postset, and whose initial
the bipartite graph. marking equals d(a),
The dynamic behavior of a place/transition net is de-
scribed in the following definitions. 3. a minimization transition associated with each state
place ps , except for the base case state places, that has
Definition 4.4 in its preset the min-in places for each decision a,
Let N = (S, T, F, K, M, W ) be a place/transition net. A
transition t ∈ T is activated if 4. a copy transition that transfers tokens from the state
place associated with s(k) to those min-in places whose
1. for every place p in the preset •t the number of tokens decision a involves the summand f (s(k) ).
at p is greater or equal to the weight W (p, t) of the
arrow from p to t and 5.2. The Role of Transitions in the PN Model
2. for every place p in the postset t• the number of tokens
at p plus the weight W (t, p) of the arrow from t to p Transitions serve two different purposes in the model.
does not exceed the capacity K(p) of p.
• Processing Transitions: Transitions can be considered
Definition 4.5 as processing elements. E.g. there are minimization
Let N = (S, T, F, K, M, W ) be a place/transition net. An transitions, addition transitions, multiplication transi-
activated transition t ∈ T may fire by tions, etc. In other words there are transitions repre-
senting arbitrary functions f .
1. decreasing the number of tokens for all p ∈ •t by In the examples designed so far, only minimization
W (p, t) and transitions are used. Minimization transitions have
2. increasing the number of tokens for all p ∈ t• by the advantage that they conform to the standard place-
W (t, p). transition net semantics described in section 4.

For a more detailed introduction to PNs, see [6] and [7]. • Copy Transitions: Another purpose of transitions is to
make the results of solved subproblems available to the
superproblems requesting these results. Each of these
5. Petri Net Model for Dynamic Programming special “copy transitions” distributes a result from one
“state place” to multiple “min-in places”.
The details of our suggested PN model for DP problems
are discussed in this section. The last subsection discusses 5.3. The Role of Places in the PN Model
Bellman Nets - an alternative to the suggested PN model.
There are two types of places in the PN model.
5.1. Construction of the PN Model
• State Places: Such a place represents a state encoun-
Suppose a discrete optimization problem can be solved tered during the solution process of a DP problem. Ex-
by an integer dynamic programming equation of the form cept for the base case state places, state places are the
K output places for minimization transitions.


f (s) = min{d(a) + f (s(k) )},
a • Min-in Places: These are the input places for mini-
k=1
mization transitions. They are also the output places
with nonnegative integral base cases values f (s0 ) given. for copy transitions.

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 3

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

5.4. The Role of Markings in the PN Model One example would be to check the PN for circularity.
Circularity is undesirable, if the PN represents a DP prob-
• Marking of State Places: Let ps be a state place. Im- lem instance, because it can cause the DP solver to loop
mediately after all minimization transition in its preset infinitely.
•ps have fired, but just before firing any of the copy
transitions in its postset ps •, the marking of a state 7. Examples
place ps representing state s contains f (s) tokens, rep-
resenting the optimal value of the subproblem associ-
ated with state s. Base case state places are initially The examples in this section were chosen to illustrate the
marked with an appropriate number of tokens as spec- basic ideas, and are not meant to reflect the class of prob-
ified in the base cases of the DP functional equation. lems that can be handled.
That is a base case state place associated with a base Two example DP problems are discussed. The first prob-
case state s0 is initially marked with f (s0 ) tokens. All lem (optimal financing) is easier than the second one in
other state places are initially marked with no tokens. the sense that an optimal solution is represented as a path
in the PN, since after each decision exactly one successor
• Marking of Min-in Places: Let pm be a min-in place. state needs to be evaluated. For these kind of problems
After all transitions in its preset •pm have fired, but a weighted directed graph representation, where states are
just before firing any transitions in its postset pm •, the nodes and edge weights represent the cost of a decision,
marking of these places corresponds to the values to be would be sufficient.
minimized by a minimization transition. Min-in places The other problem (matrix chain multiplication) has a
are initially marked with d(a) tokens. solution which is represented as a (binary) tree in the PN,
since after each decision exactly two successor states need
5.5. Bellman Nets: An Alternative PN Model to be evaluated. For these kind of problems a weighted
directed graph representation as mentioned before is obvi-
Lew describes in [3] another PN model, which is suit- ously no longer sufficient.
able as an intermediate DP problem representation: Bell- Problems with a more complex state space, such as the
man Nets. In contrast to the PN model described earlier, traveling-salesman problem, can be handled as well with
they are high-level PNs with numerically-colored tokens. the suggested PN model, but have been left out here due to
Two special types of transitions are introduced - one for paper size restrictions.
minimization, another for expression evaluation. Input to-
kens are not removed when firing a transition. To allow for 7.1. Optimal Financing Problem
standard PN semantics [4] modifies the Bellman Net model,
such that input tokens are removed unless they are replaced
For illustrative purposes we use the following simple
using self-loops.
problem instance. Suppose we need to borrow $ 6 million
The relationship between Lew’s Bellman Nets and the
in t = 0. Each of our creditors gives us several financing
specialized PNs described in this paper is still under inves-
alternatives, as shown in table 1. It is not possible to ne-
tigation. Probably, their expressional power is equivalent
gotiate other options, e.g. to request intermediate amounts
(which needs to be proven). While it is easier to trans-
not mentioned in the table. It is not possible to choose two
form Bellman nets to executable computer code, the low
alternatives from the same creditor. Alternative 0 is to not
level PN model introduced here is easier to examine with
borrow from that creditor. Creditor 1 considers larger credit
respect to consistency, and other net theoretic issues. If the
amounts as more risky and charges more interest. Credi-
equivalence of the two models can be shown, and a prac-
tor 2 has excessive amounts of cash to invest and gives us
tical transformation algorithm between the two models can
incentives to borrow larger amounts. Creditor 3 only deals
be designed, then we could take advantage of the benefits
with larger amounts. The goal is to minimize the net present
of both models.
value (NPV), denoted of our total future payments.
We present the DP approach to this problem for illus-
6. Consistency Checks on the PN Level trative purposes. It is a 3-stage decision process, since we
have to choose exactly one alternative ai where i = 1, 2, 3
As soon as a PN representation of the problem instance from each of the three creditors. Define the state x at stage
has been derived, existing PN theory can be applied to check i as the amount of money secured so far from creditors 1
static and dynamic properties. This allows for semantic through i − 1. Let y(ai ) denote the NPV of the future pay-
checks of the specification which would not easily be pos- ments to creditor i under alternative ai . Let c(ai ) denote the
sible without the PN model. principal amount of the loan from creditor i under alterna-

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 4

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

or if there are places with infinitely many tokens (which is

Table 1. A financing problem the case here) until only the copy transitions in the postset
cash flow NPV of future of these places are activated, and no other transitions are
in t = 0 payments activated.
c(ai ) y(ai ) The PN, in its final marking, is depicted in figure 2. We
creditor 1 can see that at stage 1 the goal state place associated with
alternative 0 0 0 x = 0 contains f1 (0) = 31 tokens now, the correct value
alternative 1 1 4 indeed. The actual solution, i.e. the decision to make at each
alternative 2 2 12 stage can easily be constructed from the final marking. Just
alternative 3 3 21 trace the path from the goal state place back to the base case
creditor 2 state place following empty min-in places.
alternative 0 0 0
alternative 1 1 6 7.2. The Matrix Chain Multiplication Problem
alternative 2 2 11
alternative 3 3 16 Given a product A1 A2 · · · An of matrices of various
alternative 4 4 20 (however compatible) dimensions, the goal is to find a
parenthesization, which minimizes the number of compo-
creditor 3
nentwise multiplications. The dimensions of Ai are de-
alternative 0 0 0
noted by di−1 and di . The DP functional equation can be
alternative 1 3 16
expressed as
alternative 2 4 22 
 mink∈{i,...,j−1} {f (i, k)
f (i, j) = +f (k + 1, j) + di−1 dk dj } if i < j

0 if i = j.
tive ai . For i = 1, 2, 3 the DP functional equation is
The total number of componentwise multiplications is com-
fi (x) = min{y(ai ) + fi+1 (x + c(ai ))}. puted as f (1, n).
ai
For instance, let n = 4 and (d0 , d1 , d2 , d3 , d4 ) =
The base case is (3, 4, 5, 2, 2). Apply the DP functional equation in a bot-

∞ if x < 6 tom up fashion to compute f (1, 4) = 76. If we do some
f4 (x) = bookkeeping to keep track of which arguments minimize
0 if x ≥ 6.
the min-expressions in each case, we find that the optimal
which assigns a penalty of ∞ if our decisions lead to a total parenthesization is (A1 (A2 A3 ))A4 .
principal amount that is less than the required $ 6 million. The PN model of this instance, in its initial marking, is
The optimal total NPV associated with our problem in- depicted in figure 3 - however this figure is simplified in
stance can be computed as f1 (0) = 31, which is associated that base case state places such as (1, 1), which never con-
with the decisions to choose alternative 1 from creditor 1, tain any tokens, are left out, as well as the copy transitions
alternative 2 from creditor 2, and alternative 1 from credi- which belong to these places. Furthermore the trivial mini-
tor 3. malization transitions in the presets of places (1, 2), (2, 3),
The PN model of this instance, in its initial marking, is and (3, 4) are left out because they minimize over a single
depicted in figure 1. In order to avoid too many details in input. Instead the states (1, 2), (2, 3), and (3, 4) are initial-
this figure we simplified it in the following way. Base case ized with an appropriate number of tokens.
state places at stage 4, which contain either 0 or infinitely To obtain the final marking of the PN we fire transitions
many tokens, are left out, as well as the copy transitions until the net is dead. The PN, in its final marking, is depicted
which belong to these places. Furthermore the minimiza- in figure 4 - note that this figure is simplified in the same
tion transitions in the presets of the stage 3 places are left way as before. We can see that the final state place (1, 4)
out. Instead the stage 3 states are initialized with an appro- contains f (1, 4) = 76 tokens now, the correct value indeed.
priate number of tokens. That is, Note that the order in which transitions fire is not rele-
 vant for the value of the final result f (1, 4). However values

 ∞ if 0 ≤ x ≤ 1
 of intermediate states, e.g. f (1, 3), are only meaningful, if
22 if x = 2
f3 (x) = a “controlled” firing order is used.

 16 if 3 ≤ x ≤ 5
 Again, the solution can easily be constructed from the
0 if 6 ≤ x ≤ 7.
final marking. Just trace the tree from the goal state place
To obtain a final marking of the PN we fire transitions (1, 4) back to the base case state places following empty
until the net is dead (i.e. no transition can fire any more), min-in places.

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 5

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

f3(0)
infty
6
f2(0)
min 11
f3(1)
16
infty
20

f3(2)
22
6
f2(1)
4 min 11
f3(3)
16
16
f1(0) 20
min
f3(4)
16
f2(2) 6
12 min
11
f3(5)
16
16
20

f3(6)

6
f2(3)
21 min 11
f3(7)
16

20

Figure 1. Initial marking for a financing problem

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 6

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

f3(0)
30
infty
38
f2(0)
min 1
1
f3(1)
infty
4

37 f3(2)

f2(1) 1
min
f3(3)
5

f1(0) 9
31 min
f3(4)
2

f2(2) 2
1 min
7
f3(5)
12

f3(6)

6
f2(3)
6 min 11
f3(7)

Figure 2. Final marking for a financing problem

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 7

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

copy (1,2)
60

24 24
copy (1,3)
min
30
(1,4) 40

min 30 copy (2,3)

40
min
copy (2,4) 16

12 20
copy (3,4)

Figure 3. Initial marking for a matrix chain multiplication problem

copy (1,2)

4
copy (1,3)
min
26
(1,4)
76 min 34 copy (2,3)
4
min
copy (2,4)

copy (3,4)

Figure 4. Final marking for a matrix chain multiplication problem

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 8

 Proceedings of the 37th Hawaii International Conference on System Sciences - 2004

8. Conclusion [5] B. Mikolajczak and J. T. Rumbut, Jr. Visualization of recur-

rent distributed artificial intelligence problems using colored
One advantage using a PN as an intermediate (internal Petri nets. In Proceedings of the Workshop on Intelligent In-
formation Systems, Zakopane, Poland, June 9–13, pages 16–
computer) representation is that from it ”object programs”
25, 1997.
in a variety of languages, including ”conventional” com- [6] T. Murata. Petri nets: Properties, analysis and applications.
puter programming languages or spreadsheets can be auto- Proceedings of the IEEE, 77(4):541–580, Apr. 1989.
matically generated. [7] W. Reisig. Petri Nets: an Introduction. Springer-Verlag,
OPL and many other optimization software systems in- Berlin, Germany, 1985.
corporate search mechanisms. This can in the best case be [8] P. Richard. Modelling integer linear programs with Petri nets.
very efficient, but in the worst case not lead to the optimal RAIRO - Recherche Opérationnelle - Operations Research,
solution at all. 34(3):305–312, Jul-Sep 2000.
The PN approach guarantees that optimal solution to the [9] M. Sniedovich. Dynamic Programming. Marcel Dekker, Inc.,
New York, New York, 1992.
DP instance will be found, provided of course that the DP
functional equation is correct and its solution does exist.
The PN approach has the added advantage that there is an
abundance of theoretical results in the PN literature, some
of which may be useful in analyzing DP models. Tests for
circularity (see section 6) is an obvious example. This paper
might prompt other researchers to find other such examples.
Nice features of the suggested PN model are:
• The standard place/transition net semantics are used.
• There are no conflicts in the net. The order in which
transitions fire is not relevant for the final marking of
the goal place. (However values of intermediate states
are only meaningful if a “controlled” firing order is
used).
• The solution can easily be constructed from the final
marking. Just trace the tree from the goal state place
back to the base case state places following empty min-
in places.
A software system which generates executable code
from the PN model suggested here is under development.
The software will support the solution of many DP prob-
lems arising in management science and operations re-
search.

References

[1] N. Hastings. DYNACODE Dynamic Programming Systems

Handbook. Management Center, University of Bradford,
Bradford, England, 1974.
[2] F. S. Hillier and G. J. Lieberman. Introduction to Operations
Research. McGraw-Hill Publishing Company, New York, 5th
edition, 1990.
[3] A. Lew. A Petri net model for discrete dynamic programming.
In Proceedings of the 9th Bellman Continuum: International
Workshop on Uncertain Systems and Soft Computing, Beijing,
China, July 24–27, pages 16–21, 2002.
[4] A. Lew and H. Mauch. Solving integer dynamic programming
using Petri nets. In Proceedings of the Multiconference on
Computational Engineering in Systems Applications (CESA),
Lille, France, July 9–11, 2003.

0-7695-2056-1/04 $17.00 (C) 2004 IEEE 9