An Introduction

to Probability Theory
and Its Applications

WILLIAM FELLER (1906-1970)

Eugene Higgins Professor of Mathematics

Princeton University

VOLUME II

SECOND EDITION

JOHN WILEY & SONS

 Contents

CHAPTER

I T H E EXPONENTIAL A N D THE UNIFORM DENSITIES . . . . 1

1. Introduction 1
2. Densities. Convolutions 3
3. The Exponential Density 8
4. Waiting Time Paradoxes. The Poisson Process . . . 11
5. The Persistence of Bad Luck 15
6. Waiting Times and Order Statistics 17
7. The Uniform Distribution 21
8. R a n d o m Splittings 25
9. Convolutions and Covering Theorems 26
10. R a n d o m Directions 29
11. The Use of Lebesgue Measure 33
12. Empirical Distributions 36
13. Problems for Solution 39

CHAPTER

II SPECIAL DENSITIES. RANDOMIZATION 45

1. Notations and Conventions 45

2. Gamma Distributions 47
*3. Related Distributions of Statistics 48
4. Some Common Densities 49
5. Randomization and Mixtures 53
6. Discrete Distributions 55

* Starred sections are not required for the understanding of the sequel and should be
omitted at first reading.
xvii
xviii CONTENTS

7. Bessel Functions and Random Walks 58

8. Distributions on a Circle 61
9. Problems for Solution 64

CHAPTER

III DENSITIES IN HIGHER DIMENSIONS. NORMAL DENSITIES AND

PROCESSES 66

1. Densities 66
2. Conditional Distributions 71
3. Return to the Exponential and the Uniform Distributions 74
•4. A Characterization of the Normal Distribution . . . 77
5. Matrix Notation. The Covariance Matrix 80
6. Normal Densities and Distributions 83
•7. Stationary Normal Processes 87
8. Markovian Normal Densities 94
9. Problems for Solution 99

CHAPTER

IV PROBABILITY MEASURES AND SPACES 103

1. Baire Functions 104

r
2. Interval Functions and Integrals in 3i 106
3. a-Algebras. Measurability 112
4. Probability Spaces. Random Variables 115
5. The Extension Theorem 118
6. Product Spaces. Sequences of Independent Variables. . 121
7. Null Sets. Completion 125

CHAPTER

V PROBABILITY DISTRIBUTIONS IN 3T 127

1. Distributions and Expectations 128

2. Preliminaries 136
3. Densities 138
4. Convolutions 143
 CONTENTS XIX

5. Symmetrization 148
6. Integration by Parts. Existence of Moments . . . . 150
7. Chebyshev's Inequality 151
8. Further Inequalities. Convex Functions 152
9. Simple Conditional Distributions. Mixtures . . . . 156
*10. Conditional Distributions 160
*11. Conditional Expectations 162
12. Problems for Solution 165

CHAPTER

VI A SURVEY OF SOME IMPORTANT DISTRIBUTIONS AND PROCESSES 169

1
1. Stable Distributions in ft 169
2. Examples 173
1
3. Infinitely Divisible Distributions in & 176
4. Processes with Independent Increments 179
*5. Ruin Problems in Compound Poisson Processes . . . 182
6. Renewal Processes 184
7. Examples and Problems 187
8. Random Walks 190
9. The Queuing Process 194
10. Persistent and Transient Random Walks 200
11. General Markov Chains 205
•12. Martingales 209
13. Problems for Solution 215

CHAPTER

VII L A W S OF L A R G E NUMBERS. APPLICATIONS IN ANALYSIS . . 219

1. Main Lemma and Notations 219

2. Bernstein Polynomials. Absolutely Monotone Functions 222
3. Moment Problems 224
*4. Application to Exchangeable Variables 228
*5. Generalized Taylor Formula and Semi-Groups . . . 230
6. Inversion Formulas for Laplace Transforms . . . . 232
XX CONTENTS

*7. Laws of Large Numbers for Identically Distributed

Variables 234
*8. Strong Laws 237
*9. Generalization to Martingales 241
10. Problems for Solution 244

CHAPTER

VIII THE BASIC LIMIT THEOREMS 247

1. Convergence of Measures 247
2. Special Properties 252
3. Distributions as Operators 254
4. The Central Limit Theorem 258
*5. Infinite Convolutions 265
6. Selection Theorems 267
*7. Ergodic Theorems for Markov Chains 270
8. Regular Variation 275
•9. Asymptotic Properties of Regularly Varying Functions . 279
10. Problems for Solution 284

CHAPTER

IX INFINITELY DIVISIBLE DISTRIBUTIONS AND SEMI-GROUPS . . 290

1. Orientation 290
2. Convolution Semi-Groups 293
3. Preparatory Lemmas 296
4. Finite Variances 298
5. The Main Theorems 300
6. Example: Stable Semi-Groups 305
7. Triangular Arrays with Identical D i s t r i b u t i o n s . . . . 308
8. Domains of Attraction 312
9. Variable Distributions. The Three-Series Theorem . . 316
10. Problems for Solution 318
 CONTENTS XXI

CHAPTER

X MARKOV PROCESSES AND SEMI-GROUPS 321

1. The Pseudo-Poisson Type 322

2. A Variant: Linear Increments 324
3. Jump Processes 326
1
4. Diffusion Processes in 31 332
5. The Forward Equation. Boundary Conditions . . . 337
6. Diffusion in Higher Dimensions 344
7. Subordinated Processes 345
8. Markov Processes and Semi-Groups 349
9. The "Exponential Formula" of Semi-Group Theory . . 353
10. Generators. The Backward Equation 356

CHAPTER

XI RENEWAL THEORY 358

1. The Renewal Theorem 358

2. Proof of the Renewal Theorem 364
*3. Refinements 366
4. Persistent Renewal Processes 368
5. The Number Nt of Renewal Epochs 372
6. Terminating (Transient) Processes 374
7. Diverse Applications 377
8. Existence of Limits in Stochastic Processes 379
*9. Renewal Theory on the Whole Line 380
10. Problems for Solution 385

CHAPTER

XII RANDOM WALKS IN ft1 389

1. Basic Concepts and Notations 390

2. Duality. Types of Random Walks 394
3. Distribution of Ladder Heights. Wiener-Hopf Factor-
ization 398
3a. The Wiener-Hopf Integral Equation 402
XX11 CONTENTS

4. Examples 404
5. Applications 408
6. A Combinatorial Lemma 412
7. Distribution of Ladder Epochs 413
8. The Arc Sine Laws 417
9. Miscellaneous Complements 423
10. Problems for Solution 425

CHAPTER

XIII LAPLACE TRANSFORMS. TAUBERIAN THEOREMS. RESOLVENTS . 429

1. Definitions. The Continuity Theorem 429
2. Elementary Properties 434
3. Examples 436
4. Completely Monotone Functions. Inversion Formulas . 439
5. Tauberian Theorems 442
•6. Stable Distributions 448
*7. Infinitely Divisible Distributions 449
*8. Higher Dimensions 452
9. Laplace Transforms for Semi-Groups 454
10. The Hille-Yosida Theorem 458
11. Problems for Solution 463

CHAPTER

XIV APPLICATIONS OF LAPLACE TRANSFORMS 466

1. The Renewal Equation: Theory 466

2. Renewal-Type Equations: Examples 468
3. Limit Theorems Involving Arc Sine Distributions . . . 470
4. Busy Periods and Related Branching Processes . . . 473
5. Diffusion Processes 475
6. Birth-and-Death Processes and Random Walks . . . 479
7. The Kolmogorov Differential Equations 483
8. Example: The Pure Birth Process 488
9. Calculation of Ergodic Limits and of First-Passage Times 491
10. Problems for Solution 495
 CONTENTS XXIU

CHAPTER

XV CHARACTERISTIC FUNCTIONS 498

1. Definition. Basic Properties 498

2. Special Distributions. Mixtures 502
2a. Some Unexpected Phenomena 505
3. Uniqueness. Inversion Formulas 507
4. Regularity Properties 511
5. The Central Limit Theorem for Equal Components . . 515
6. The Lindeberg Conditions 518
7. Characteristic Functions in Higher Dimensions . . .521
*8. Two Characterizations of the Normal Distribution . . 525
9. Problems for Solution 526

CHAPTER

XVI* EXPANSIONS RELATED TO THE CENTRAL LIMIT THEOREM. . 531

1. Notations 532
2. Expansions for Densities 533
3. Smoothing 536
4. Expansions for Distributions 538
5. The Berry-Esseen Theorems 542
6. Expansions in the Case of Varying Components . . . 546
7. Large Deviations 548

CHAPTER

XVII INFINITELY DIVISIBLE DISTRIBUTIONS 554

1. Infinitely Divisible Distributions 554
2. Canonical Forms. The Main Limit Theorem . . . . 558
2a. Derivatives of Characteristic Functions 565
3. Examples and Special Properties 566
4. Special Properties 570
5. Stable Distributions and Their Domains of Attraction . 574
*6. Stable Densities 581
7. Triangular Arrays 583
XXIV CONTENTS

•8. The Class L 588

•9. Partial Attraction. "Universal Laws" 590
•10. Infinite Convolutions 592
11. Higher Dimensions 593
12. Problems for Solution 595

CHAPTER

XVIII APPLICATIONS OF FOURIER METHODS TO RANDOM WALKS . 598

1. The Basic Identity 598
*2. Finite Intervals. Wald's Approximation 601
3. The Wiener-Hopf Factorization 604
4. Implications and Applications 609
5. Two Deeper Theorems 612
6. Criteria for Persistency 614
7. Problems for Solution 616

CHAPTER

XIX HARMONIC ANALYSIS 619

1. The Parseval Relation 619

2. Positive Definite Functions 620
3. Stationary Processes 623
4. Fourier Series 626
*5. The Poisson Summation Formula 629
6. Positive Definite Sequences 633
7. L 2 Theory 635
8. Stochastic Processes and Integrals 641
9. Problems for Solution 647

ANSWERS TO PROBLEMS 651

SOME BOOKS ON COGNATE SUBJECTS 655

INDEX 657