Dynamic Probabilistic Systems Volume 2 - Ronald A. Howard
Uploaded by
georgetacaprarescuDynamic Probabilistic Systems Volume 2 - Ronald A. Howard
Uploaded by
georgetacaprarescu----
DYNAMIC PROBABILISTIC SYSTEMS
Volume II: SEMI-MARKOV AND DECISION PROCESSES
RONALD A. {!fO}VARD
Stanford University
Stanford, California
JOHN WILEY & SONS, INC.,
New York · London · Sydney · Toron
PREFACE
Uncertainty, complexity, and dynamism have been continuing challenges to man's
understanding and control of his physical environment. In the development of
logical structures to describe these phenomena, the model originated by A. A. Markov
stands as a major accomplishment. Where previous contributors had modeled
uncertainty as a sequence of independent trials, Markov saw the advantage of
introducing dependence of each trial on the result of its predecessor. While it is
tempting to consider even more complex dependency of the present trial on the
results of past trials, such temptation usually leads to results that are both analyti-
cally and computationally intractable.
Consequently, Markov models represent the first outpost in the domain of
dependent models that is powerful both in capturing the essence of many dependent
systems observed in practice and in producing the analytical and computational
results necessary to gain insight into their behavior. The purpose of this book is to
equip the reader to formulate, analyze, and evaluate simple and advanced Markov
models of systems that range from genetics to space engineering to marketing.
This book should be viewed not as a collection of techniques (although many will
value it as such), but rather as a consistent application of the fundamental principles
of probability and linear system theory.
It is often said that good ideas are simple; the Markov process is no exception.
In fact, there is no problem considered in this book that cannot be made clear to a
child. The device we use to make such expositions simple is a pond covered with
lily pads among which a frog may jump. Although his jumps may be random, the
frog never falls into the water. Furthermore, he may occupy the same pad after
successive jumps. We can use the lily pond analogy to clarify any discussion in the
text.
The book is an integrated work published in two volumes. The first volume,
Chapters 1-9, treats the basic Markov process and its variants; the second, Chapters
10-15, semi-Markov and decision processes. Although the reader who is already
familiar with Markov process terminology will understand the design of the book
from the Contents, it should be helpful to all readers to discuss each chapter
using the lily pond analogy.
ix
x PREFACE
Chapter 1 introduces the basic Markov model. The number of pads in the lily
pond is assumed to be finite. The Markov assumption states that the probability
of the frog's making his next jump to any pad depends only on the pad he currently
occupies and not on how he reached his present pad. The discussion then considers
the question, given the starting pad and the fact that /1 jumps have been made
without our observing the pond, what probability must we assign to the frog's
occupying each pad in the pond. Graphical and transform methods are used to
find these probabilities and to illustrate the typical behavior of a variety of
processes.
Chapter 2 is a diversion to present the theory of linear systems and graphical
methods for their analysis. Chapter 3 shows how to apply these powerful techniques
to answering the problems of Chapter 1 with respect to the lily pond.
Chapter 4 considers the case where certain pads, once occupied, cannot be left
and investigates the statistics for the number of jumps required before such trapping
occurs. Chapter 5 treats two types of statistics: First, how many jumps will be made
onto each pad in a given number of total jumps and, second, how many jumps
will be required to reach one pad from another pad for the first time.
Chapter 6 finds the statistics of jumps from one set of pads to another set of
pads. It allows identifying certain jumps as being of special interest and then finding
the statistics of these jumps. Furthermore, it considers the implications of the
information provided by these jumps for probability assignments on the frog's
location.
Chapter 7 focuses on the case where we observe only one pad and where the
frog's returns to that pad can be governed by an arbitrary probability distribution.
We explore the statistics of these returns from several pcints of view. This chapter
also discusses for the first time the possibility that the pond may have an infinite
number of pads and illustrates the analytic modifications necessary to treat this
situation.
Chapter 8 investigates the possibility of having several frogs jumping in the
pond at the same time. It expands the discussion to the case where at certain times
frogs may be added to each pad in the pond in varying numbers. Finally, it treats
the case where the frogs may breed to produce new generations of jumpers.
Chapter 9 considers the case where the probability of a jump from one pad to
another may be different on successive jumps. For example, the frog may be
getting tired.
Chapter 10 introduces another dimension to the discussion. We explicitly con-
sider the time between jumps as a multiple of some time unit and let this be a
random variable that depends on the jump made by the frog. We can now consider
the probability of the frog's occupying each pad given both the number of jumps
he has made and the total time units he has been jumping. All previous questions
about the frog's behavior are re-examined in this generalized form.
PREFACE xi
Chapter 11 allows the time between jumps to be a continuous random variable and
extends all previous discussions to this case. Chapter 12 considers the continuous-
time Markov process, the case in which the time that the frog has occupied the
same pad has no bearing on the amount of time he will continue to occupy it. It
also discusses the infinite-pad lily pond once again.
Chapter 13 concerns the payment of rewards on the basis of the frog's jumping.
He may earn a reward for making a particular jump or from occupying a pad.
We develop and illustrate expressions for the expected total reward the frog will
earn in a finite and infinite time with and without discounting.
Chapter 14 presents another diversion, a discussion of the optimization proce-
dure of dynamic programming, the natural way to formulate optimization problems
for Markov processes.
Chapter 15 combines the material of the last several chapters to develop proce-
dures for optimizing the expected reward generated by the frog. Control is effected
by altering the probabilistic structure and reward structure of his jumping.
Since the level of interest in frog-jumping is relatively low, the examples through-
out are drawn from a variety of fields: consumer purchasing, taxicab operation,
inventory control, rabbit reproduction, coin-tossing, gambling, family-name
extinction, search, car rental, machine repair, depreciation, production, action-
timing, reliability, reservation policy, machine maintenance and replacement,
network traversal, project scheduling, space exploration, and success in business.
The applicability of the models to still other areas is also made evident.
The background suggested for the reader if he is to achieve greatest benefit
from the book is a foundation in calculus, probability theory, and matrix theory.
The emphasis is on the fundamental rather than the esoteric in these areas, so
preparation should not be a major stumbling block. The book may be read with
ease by anyone with the basic background whether his field be engineering, manage-
ment, psychology, or whatever. Specialized material from any field used in examples
is pr:esented as required. Consequently, the professional reading independently to
expand his knowledge of Markov models should be able to proceed rapidly and
effectively.
This book developed from courses taught at M.I.T. and Stanford University
over a decade. Although the advanced undergraduate can master the subject with
no difficulty, the course has always been offered at the graduate level. The material
is developed naturally for course presentation. To cover the whole book typically
requires the entire academic year. The two volumes serve well as texts for successive
one-semester courses. If a two-quarter version is desired, then Chapters 8 and 9
can be placed in the optional category.
One feature of the development that is becoming increasingly advantageous to
exploit is the suitability of examples and problems for solution by computer,
particularly the time-shared variety. Virtually every chapter contains material
