Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.

org/terms-privacy

with Applications
Stochastic Processes
9
 Books in the Classics in Applied Mathematics series are monographs and textbooks declared out
of print by their original publishers, though they are of continued importance and interest to the
mathematical community. SIAM publishes this series to ensure that the information presented in these
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

texts is not lost to today's students and researchers.

Editor-in-Chief
Robert E. O'Malley, Jr., University of Washington

Editorial Board
John Boyd, University of Michigan
Leah Edelstein-Keshet, University of British Columbia
William G. Faris, University of Arizona
Nicholas J. Higham, University of Manchester
Peter Hoff, University of Washington
Mark Kot, University of Washington
Peter Olver, University of Minnesota
Philip Protter, Cornell University
Gerhard Wanner, L'Universite de Geneve

Classics in Applied Mathematics

C. C. Lin and L. A. Segel, Mathematics Applied to Deterministic Problems in the Natural Sciences
Johan G. F. Belinfante and Bernard Kolman, A Survey of Lie Groups and Lie Algebras with Applications and
Computational Methods
James M. Ortega, Numerical Analysis: A Second Course
Anthony V. Fiacco and Garth P. McCormick, Nonlinear Programming: Sequential Unconstrained
Minimization Techniques
F. H. Clarke, Optimization and Nonsmooth Analysis
George F. Carrier and Carl E. Pearson, Ordinary Differential Equations
Leo Breiman, Probability
R. Bellman and G. M. Wing, An Introduction to Invariant Imbedding
Abraham Berman and Robert J. Plemmons, Nonnegative Matrices in the Mathematical Sciences
Olvi L. Mangasarian, Nonlinear Programming
*Carl Friedrich Gauss, Theory of the Combination of Observations Least Subject to Errors: Part One,
Part Two, Supplement. Translated by G. W. Stewart
Richard Bellman, Introduction to Matrix Analysis
U. M. Ascher, R. M. M. Mattheij, and R. D. Russell, Numerical Solution of Boundary Value Problems for
Ordinary Differential Equations
K. E. Brenan, S. L. Campbell, and L. R. Petzold, Numerical Solution of Initial-Value Problems
in Differential- Algebraic Equations
Charles L. Lawson and Richard J. Hanson, Solving Least Squares Problems
J. E. Dennis, Jr. and Robert B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear
Equations
Richard E. Barlow and Frank Proschan, Mathematical Theory of Reliability
Cornelius Lanczos, Linear Differential Operators
Richard Bellman, Introduction to Matrix Analysis, Second Edition
Beresford N. Parlett, The Symmetric Eigenvalue Problem
Richard Haberman, Mathematical Models: Mechanical Vibrations, Population Dynamics, and Traffic Flow
Peter W. M. John, Statistical Design and Analysis of Experiments
Tamer Ba§ar and Geert Jan Olsder, Dynamic Noncooperative Game Theory, Second Edition
Emanuel Parzen, Stochastic Processes
*First time in print.
 Classics in Applied Mathematics (continued)

Petar Kokotovic, Hassan K. Khalil, and John O'Reilly, Singular Perturbation Methods in Control: Analysis
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

and Design
Jean Dickinson Gibbons, Ingram Olkin, and Milton Sobel, Selecting and Ordering Populations: A New
Statistical Methodology
James A. Murdock, Perturbations: Theory and Methods
Ivar Ekeland and Roger Temam, Convex Analysis and Variational Problems
Ivar Stakgold, Boundary Value Problems of Mathematical Physics, Volumes I and II
J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables
David Kinderlehrer and Guido Stampacchia, An Introduction to Variational Inequalities and Their Applications
F. Natterer, The Mathematics of Computerized Tomography
Avinash C. Kale and Malcolm Slaney, Principles of Computerized Tomographic Imaging
R. Wong, Asymptotic Approximations of Integrals
O. Axelsson and V. A. Barker, Finite Element Solution of Boundary Value Problems: Theory and Computation
David R. Brillinger, Time Series: Data Analysis and Theory
Joel N. Franklin, Methods of Mathematical Economics: Linear and Nonlinear Programming, Fixed-Point
Theorems
Philip Hartman, Ordinary Differential Equations, Second Edition
Michael D. Intriligator, Mathematical Optimization and Economic Theory
Philippe G. Ciarlet, The Finite Element Method for Elliptic Problems
Jane K. Cullum and Ralph A. Willoughby, Lanczos Algorithms for Large Symmetric Eigenvalue
Computations, Vol. I: Theory
M. Vidyasagar, Nonlinear Systems Analysis, Second Edition
Robert Mattheij and Jaap Molenaar, Ordinary Differential Equations in Theory and Practice
Shanti S. Gupta and S. Panchapakesan, Multiple Decision Procedures: Theory and Methodology
of Selecting and Ranking Populations
Eugene L. Allgower and Kurt Georg, Introduction to Numerical Continuation Methods
Leah Edelstein-Keshet, Mathematical Models in Biology
Heinz-Otto Kreiss and Jens Lorenz, Initial-Boundary Value Problems and the Navier-Stokes Equations
J. L. Hodges, Jr. and E. L. Lehmann, Basic Concepts of Probability and Statistics, Second Edition
George F. Carrier, Max Krook, and Carl E. Pearson, Functions of a Complex Variable: Theory and
Technique
Friedrich Pukelsheim, Optimal Design of Experiments
Israel Gohberg, Peter Lancaster, and Leiba Rodman, Invariant Subspaces of Matrices with Applications
Lee A. Segel with G. H. Handelman, Mathematics Applied to Continuum Mechanics
Rajendra Bhatia, Perturbation Bounds for Matrix Eigenvalues
Barry C. Arnold, N. Balakrishnan, and H. N. Nagaraja, A First Course in Order Statistics
Charles A. Desoer and M. Vidyasagar, Feedback Systems: Input-Output Properties
Stephen L. Campbell and Carl D. Meyer, Generalized Inverses of Linear Transformations
Alexander Morgan, Solving Polynomial Systems Using Continuation for Engineering and Scientific Problems
I. Gohberg, P. Lancaster, and L. Rodman, Matrix Polynomials
Galen R. Shorack and Jon A. Wellner, Empirical Processes with Applications to Statistics
Richard W. Cottle, Jong-Shi Pang, and Richard E. Stone, The Linear Complementarity Problem
Rabi N. Bhattacharya and Edward C. Waymire, Stochastic Processes with Applications
Robert J. Adler, The Geometry of Random Fields
Mordecai Avriel, Walter E. Diewert, Siegfried Schaible, and Israel Zang, Generalized Concavity
Rabi N. Bhattacharya and R. Ranga Rao, Normal Approximation and Asymptotic Expansions
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

F ^
Stochastic Processes
with Applications
b ci

Rabi N. Bhattacharya
University of Arizona
Tucson, Arizona

Edward C. Waymire
Oregon State University
Corvallis, Oregon

pia m o
Society for Industrial and Applied Mathematics
Philadelphia
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Copyright © 2009 by the Society for Industrial and Applied Mathematics

This SIAM edition is an unabridged republication of the work first published by John
Wiley & Sons (SEA) Pte. Ltd., 1992.

10987654321

All rights reserved. Printed in the United States of America. No part of this book may
be reproduced, stored, or transmitted in any manner without the written permission of
the publisher. For information, write to the Society for Industrial and Applied
Mathematics, 3600 Market Street, 6th Floor, Philadelphia, PA 19104-2688 USA.

Library of Congress Cataloging-in-Publication Data

Bhattacharya, R. N. (Rabindra Nath), 1937-

Stochastic processes with applications / Rabi N. Bhattacharya, Edward C. Waymire.
p. cm. -- (Classics in applied mathematics ; 61)
Originally published: New York : Wiley, 1990.
Includes index.
ISBN 978-0-898716-89-4
1. Stochastic processes. I. Waymire, Edward C. II. Title.
QA274.B49 2009
519.2'3--dc22
2009022943

S1L2JTL. is a registered trademark.

Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

with love
To Gouri and Linda,
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Contents

Preface to the Classics Edition xiii

Preface xv

Sample Course Outline xvii

I Random Walk and Brownian Motion 1

1. What is a Stochastic Process?, 1
2. The Simple Random Walk, 3
3. Transience and Recurrence Properties of the Simple Random Walk, 5
4. First Passage Times for the Simple Random Walk, 8
5. Multidimensional Random Walks, 11
6. Canonical Construction of Stochastic Processes, 15
7. Brownian Motion, 17
8. The Functional Central Limit Theorem (FCLT), 20
9. Recurrence Probabilities for Brownian Motion, 24
10. First Passage Time Distributions for Brownian Motion, 27
11. The Arcsine Law, 32
12. The Brownian Bridge, 35
13. Stopping Times and Martingales, 39
14. Chapter Application: Fluctuations of Random Walks with Slow Trends
and the Hurst Phenomenon, 53
Exercises, 62
Theoretical Complements, 90

II Discrete-Parameter Markov Chains 109

1. Markov Dependence, 109
2. Transition Probabilities and the Probability Space, 110

ix
 X CONTENTS
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

3. Some Examples, 113

4. Stopping Times and the Strong Markov Property, 117
5. A Classification of States of a Markov Chain, 120
6. Convergence to Steady State for Irreducible and Aperiodic Markov
Processes on Finite Spaces, 126
7. Steady-State Distributions for General Finite-State Markov
Processes, 132
8. Markov Chains: Transience and Recurrence Properties, 135
9. The Law of Large Numbers and Invariant Distributions for Markov
Chains, 138
10. The Central Limit Theorem for Markov Chains, 148
11. Absorption Probabilities, 151
12. One-Dimensional Nearest-Neighbor Gibbs States, 162
13. A Markovian Approach to Linear Time Series Models, 166
14. Markov Processes Generated by Iterations of I.I.D. Maps, 174
15. Chapter Application: Data Compression and Entropy, 184
Exercises, 189
Theoretical Complements, 214

III Birth—Death Markov Chains 233

1. Introduction to Birth—Death Chains, 233
2. Transience and Recurrence Properties, 234
3. Invariant Distributions for Birth—Death Chains, 238
4. Calculations of Transition Probabilities by Spectral Methods, 241
5. Chapter Application: The Ehrenfest Model of Heat Exchange, 246
Exercises, 252
Theoretical Complements, 256

IV Continuous-Parameter Markov Chains 261

1. Introduction to Continuous-Time Markov Chains, 261
2. Kolmogorov's Backward and Forward Equations, 263
3. Solutions to Kolmogorov's Equations in Exponential Form, 267
4. Solutions to Kolmogorov's Equations by Successive Approximation, 271
5. Sample Path Analysis and the Strong Markov Property, 275
6. The Minimal Process and Explosion, 288
7. Some Examples, 292
8. Asymptotic Behavior of Continuous-Time Markov Chains, 303
9. Calculation of Transition Probabilities by Spectral Methods, 314
10. Absorption Probabilities, 318
 CONTENTS Xi
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

11. Chapter Application: An Interacting System: The Simple Symmetric

Voter Model, 324
Exercises, 333
Theoretical Complements, 349

V Brownian Motion and Diffusions 367

1. Introduction and Definition, 367
2. Kolmogorov's Backward and Forward Equations, Martingales, 371
3. Transformation of the Generator under Relabeling of the State Space, 381
4. Diffusions as Limits of Birth—Death Chains, 386
5. Transition Probabilities from the Kolmogorov Equations: Examples, 389
6. Diffusions with Reflecting Boundaries, 393
7. Diffusions with Absorbing Boundaries, 402
8. Calculation of Transition Probabilities by Spectral Methods, 408
9. Transience and Recurrence of Diffusions, 414
10. Null and Positive Recurrence of Diffusions, 420
11. Stopping Times and the Strong Markov Property, 423
12. Invariant Distributions and the Strong Law of Large Numbers, 432
13. The Central Limit Theorem for Diffusions, 438
14. Introduction to Multidimensional Brownian Motion and Diffusions, 441
15. Multidimensional Diffusions under Absorbing Boundary Conditions and
Criteria for Transience and Recurrence, 448
16. Reflecting Boundary Conditions for Multidimensional Diffusions, 460
17. Chapter Application: G. I. Taylor's Theory of Solute Transport in a
Capillary, 468
Exercises, 475
Theoretical Complements, 497

VI Dynamic Programming and Stochastic Optimization 519

1. Finite-Horizon Optimization, 519
2. The Infinite-Horizon Problem, 525
3. Optimal Control of Diffusions, 533
4. Optimal Stopping and the Secretary Problem, 542
5. Chapter Application: Optimality of (S, s) Policies in Inventory
Problems, 549
Exercises, 557
Theoretical Complements, 559
 xii CONTENTS
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

VII An Introduction to Stochastic Differential Equations 563

1. The Stochastic Integral, 563
2. Construction of Diffusions as Solutions of Stochastic Differential
Equations, 571
3. It6's Lemma, 582
4. Chapter Application: Asymptotics of Singular Diffusions, 591
Exercises, 598
Theoretical Complements, 607

0 A Probability and Measure Theory Overview 625

1. Probability Spaces, 625
2. Random Variables and Integration, 627
3. Limits and Integration, 631
4. Product Measures and Independence, Radon—Nikodym Theorem and
Conditional Probability, 636
5. Convergence in Distribution in Finite Dimensions, 643
6. Classical Laws of Large Numbers, 646
7. Classical Central Limit Theorems, 649
8. Fourier Series and the Fourier Transform, 653

Author Index 665

Subject Index 667
Errata 673
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Preface to the Classics Edition

The publication of Stochastic Processes with Applications (SPWA) in the SIAM

Classic in Applied Mathematics series is a matter of great pleasure for us, and we
are deeply appreciative of the efforts and good will that went into it. The book has
been out of print for nearly ten years. During this period we received a number of
requests from instructors for permission to make copies of the book to be used as
a text on stochastic processes for graduate students. We also received many kind
laudatory words, along with inquiries about the possibility of bringing out a second
edition, from mathematicians, statisticians, physicists, chemists, geoscientists, and
others from the U.S. and abroad. We hope that the inclusion of a detailed errata is a
helpful addition to the original.
SPWA was a work of love for its authors. As stated in the original preface,
the book was intended for use (1) as a graduate-level text for students in diverse
disciplines with a reasonable background in probability and analysis, and (2) as a
reference on stochastic processes for applied mathematicians, scientists, engineers,
economists, and others whose work involves the application of probability. It was our
desire to communicate our sense of excitement for the subject of stochastic processes
to a broad community of students and researchers. Although we have often empha-
sized substance over form, the presentation is systematic and rigorous. A few proofs
are relegated to Theoretical Complements, and appropriate references for proofs are
provided for some additional advanced technical material. The book covers a sub-
stantial part of what we considered to be the core of the subject, especially from the
point of view of applications. Nearly two decades have passed since the publication
of SPWA, but the importance of the subject has only grown. We are very happy to
see that the book's rather unique style of exposition has a place in the broader applied
mathematics literature.
We would like to take this opportunity to express our gratitude to all those col-
leagues who over the years have provided us with encouragement and generous
words on this book. Special thanks are due to SIAM editors Bill Faris and Sara
Murphy for shepherding SPWA back to print.
RABI N. BHATTACHARYA
EDWARD C. WAYMIRE

XIII
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Preface

This is a text on stochastic processes for graduate students in science and

engineering, including mathematics and statistics. It has become somewhat
commonplace to find growing numbers of students from outside of mathematics
enrolled along with mathematics students in our graduate courses on stochastic
processes. In this book we seek to address such a mixed audience. For this
purpose, in the main body of the text the theory is developed at a relatively
simple technical level with some emphasis on computation and examples.
Sometimes to make a mathematical argument complete, certain of the more
technical explanations are relegated to the end of the chapter under the label
theoretical complements. This approach also allows some flexibility in
instruction. A few sample course outlines have been provided to illustrate the
possibilities for designing various types of courses based on this book. The
theoretical complements also contain some supplementary results and references
to the literature.
Measure theory is used sparingly and with explanation. The instructor may
exercise control over its emphasis and use depending on the background of the
majority of the students in the class. Chapter 0 at the end of the book may be
used as a short course in measure theoretical probability for self study. In any
case we suggest that students unfamiliar with measure theory read over the first
few sections of the chapter early on in the course and look up standard results
there from time to time, as they are referred in the text.
Chapter applications, appearing at the end of the chapters, are largely drawn
from physics, computer science, economics, and engineering. There are many
additional examples and applications illustrating the theory; they appear in the
text and among the exercises.
Some of the more advanced or difficult exercises are marked by asterisks.
Many appear with hints. Some exercises are provided to complete an argument
or statement in the text. Occasionally certain well-known results are only a few
steps away from the theory developed in the text. Such results are often cited
in the exercises, along with an outline of steps, which can be used to complete
their derivation.
Rules of cross-reference in the book are as follows. Theorem m.n, Proposition

xv
 xvi PREFACE
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

m.n, or Corollary m.n, refers to the nth such assertion in section m of the same
chapter. Exercise n, or Example n, refers to the nth Exercise, or nth Example,
of the same section. Exercise m.n (Example m.n) refers to Exercise n (Example
n) of a different section m within the same chapter. When referring to a result
or an example in a different chapter, the chapter number is always mentioned
along with the label m.n to locate it within that chapter.
This book took a long time to write. We gratefully acknowledge research
support from the National Science Foundation and the Army Research Office
during this period. Special thanks are due to Wiley editors Beatrice Shube and
Kate Roach for their encouragement and assistance in seeing this effort through.

RABI N. BHATTACHARYA
EDWARD C. WAYMIRE

Bloomington, Indiana
Corvallis, Oregon
February 1990
Downloaded 01/24/24 to 50.7.253.52 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Sample Course Outlines

COURSE I
Beginning with the Simple Random Walk, this course leads through Brownian
Motion and Diffusion. It also contains an introduction to discrete/continuous-
parameter Markov Chains and Martingales. More emphasis is placed on concepts,
principles, computations, and examples than on complete proofs and technical
details.
Chapter 1 Chapter II Chapter III
§1-7 (+ Informal Review of Chapter 0, §4) §1-4 §1--3
§13 (Up to Proposition 13.5) §5 (By examples) §5
§11 (Example 2)
§13
Chapter IV Chapter V Chapter VI
§1-7 (Quick survey §1 §4
by examples) §2 (Give transience/recurrence
from Proposition 2.5)
§3 (Informal justification of
equation (3.4) only)
§5-7
§10
§11 (Omit proof of Theorem 11.1)
§12-14
COURSE 2
The principal topics are the Functional Central Limit Theorem, Martingales,
Diffusions, and Stochastic Differential Equations. To complete proofs and for
supplementary material, the theoretical complements are an essential part of this
course.
Chapter I Chapter V Chapter VI Chapter VII
§1-4 (Quick survey) §1-3 §4 §1--4
§6-10 §6-7
§13 §11
§13 -17
COURSE 3
This is a course on Markov Chains that also contains an introduction to
Martingales. Theoretical complements may he used only sparingly.

Chapter I Chapter II Chapter III Chapter IV Chapter VI

§1-6 §1-9 §1 §1-11 §1-2
§13 §11 §5 §4-5
§12 or 15
§13-14
xvii