WILEY SERIES IN PROBABILITY AND MATHEMATICAL STATISTICS ESTABLISHED By WALTER A. SHEWHART AND SAMUEL S. WILKS Editors Ralph A. Bradley David G. Kendatt J. Stuart Hunter Geoffrey S. Watson Probability and Mathematical Statistics ANDERSON - An Introduction to Multivariate Statistical Analysis BLACKWELL and GIRSHICK - Theory of Games and Statistical Deci- sions CRAMER The Elements of Probability Theory and Some of Its Appli- cations DOOB - Stochastic Processes FELLER « An Introduction to Probability Theory and Its Applications, Volume I, Third Edition FELLER «An Introduction to Probability Theory and Its Applice- tions, Volume IT ies—An Introduction FRASER + The Structure of Inference GRENANDER and ROSENBLATT - Statistical Analysis of Stationary Time Series HANSEN, HURWITZ, and MADOW - Sample Survey Methods and Theory, Volumes I and IT HOEL - Introduction to Mathematical Statistics, Third Edition KEMPTHORNE - The Design and Analysis of Experiments LARSON - Introduction to Probability Theory and Statistical Inference LEHMANN : Testing Statistical Hypotheses PARZEN « Modern Probability Theory and Its Applications RAO = Linear Statistical Inference and Its Applications RIORDAN - An Introduction to Combinatorial Analysis SCHEFFE - The Analysis of Variance WALD - Sequential Analysis, WILKS - Collected Papers: Contritutions to Mathematical Statistics WILKS - Mathematical Statistics Applied Probability and Statistics BAILEY - The Elements of Stochastic Processes with Applications to the Natural Sciences BARTHOLOMEW - Stochastic Models for Social Processes BENNETT and FRANKLIN - Statistical Analysis in Chen Chemical Industry BOX and DRAPER « Evolutionary Operation: A Statistical Method for Process Improvement BROWNLEE - Statistical Theory and Methodology in Science and Engineering, Second Edition try and theApplied Probability and Statisties (Continued) CHAKRAVARTI, LAHA and ROY + Handbook of Methods of Applied Statistics, Vol. I CHAKRAVARTT, TAHA and ROY » Handhook of Methods of Applied Statistics, Vol. IT CHERNOFF and MOSES - Elementary Decision Theory CHIANG - Introduction to Stochastic Processes in Biostatisti CLELLAND, deCANI, BROWN, BURSK, and MURRAY « Basic Statistics with Business Applications COCHRAN + Sampling Techniques, Second Edition COCHRAN and COX + Experimental Designs, Second Edition COX + Planning of Experiments COX and MILTER « The Theory of Stochastic Processes DAVID « Order Statistics DEMING « Sample Design in Business Rescarch DODGE and ROMIG « Sampling Inspection Tables, Second Edition DRAPER und SMITH + Appiied Regression Anaiysis GOLDBERGER - Econometric Theory GUTTMAN and WILKS - Introductory Engineering Statistics HALD - Statistical Tables and Formulas HALD - Statistical Theory with Engineering Applications HOEFL + Elementary Statistics, Second Edition HUANG « Regression and Econometric Methods JOHNSON and LEONE - Statistics and Experimental Design: In Engi- neering and the Physical Sciences, Volumes I and IT LANCASTER * The Chi Squared Distribution MILTON - Rank Order Probabilities: Two-Sample Normal Shift Alternatives PRABHU « Queues and Inventories: A Study of Their Basic Stochastic Processes SARHAN and GREENBERG - Contributions to Order Statisties SEAL « Stochastic Theory of a Risk Business WILLIAMS - Regression Analysis WOLD and JUREEN + Demand Analysis WONNACOTT and WONNACOTT + Introduction to Econometric Methods YOUDEN - Statistical Methods for Chemists Tracts on Probability and Statistics BILLINGSLEY - Ergodic Theory and Information BILLINGSLEY « Convergence of Probability Measures CRAMER and LEADBETTER - Stationary and Related Stochastic Processes . _ RIORDAN - Combinatorial Identities TAKACS - Combinatorial Methods in the Theory af Stochastic Processes rake pe eh SF wiAn Introduction to Probability Theory and Its Applications“An Introduction to Probability Theory and Its Applications WILLIAM FELLER (1906 - 1970) Eugene Higgins Professor of Mathematics Princeton University VOLUME I THIRD EDITION Revised rining 098697 John Wiley & Sons, Inc. New York » London - Sydney109876543 Copyright, 1950 by William Feller Copyright © 1957, 1968 by John Wiley & Sons, Inc, Al Rights Reserved. This book or any part thereof Imust not be reproduced in any form without the ‘written permission of the publisher. Luprary ot Congress Catalog Cara Number: 68-11708 Printed in the United States of AmericaTo O. E. Neugebauer: o et praesidium et dulce decus meumPreface to the Third Edition WHEN THIS BOOK WAS FIRST CONCEIVED (MORE THAN 25 YEARS AGO) few mathematicians outside the Soviet Union recognized probability as a legitimate branch of mathematics. Applications were limited in scope, and the Weatment of individual problems often led to incredible com- plications. Under these circumstances the book could not be written for an existing audience, or to satisfy conscious needs. The hope was rather to atiract attention to iitie-known aspects of probability, to forge tinks between various parts, to develop unified methods, and to point to potential applications. Because of a growing interest in probability, the book found unexpectedly many users outside mathematical disciplines. Its widespread use was understandable as long as its point of view was new and its material was not otherwise available. But the popularity seems to persist even now, when the contents of most chapters are avaii- able in specialized works streamlined for particular needs. For this reason the character of the book remains unchanged in the new edition. I hope that it will continue to serve a variety of needs and, in particular, that it will continue to find readers who read it merely for enjoyment and enlightenment. ‘Throughout the years I was the grateful recipient of many communica- tions from users, and these led to various improvements. Many sections were rewritten to facilitate study. Reading is also improved by a better typeface and the superior editing job by Mrs. H. McDougal: although a professional editor she has preserved a feeling for the requirements of readers and reason. ‘The greatest change is in chapter II. This chapter was introduced only in the second edition, which was in fact motivated principally by the unexpected discovery that its enticing material could be treated by elementary methods, But this treatment still depended on combinatorial artifices which have now been replaced by simpler and more natural probabilistic arguments. In essence this chapter is new. Most conspicuous among other additions are the new sections on b 8 PF chai 1m the De Maivre-T aplace theorem. Chapter XIII has been rearranged, and throughout the book bsviii PREFACE TO THE THIRD EDITION there appear minor changes as well as new examples and problems. I regret the misleading nature of the author index, but I felt obliged to state explicitly whenever an idea or example could be traced to a particular source. Unfortunately this means that quotations usually refer to an incidental remark, and are rarely indicative of the nature of the paper quoted. Furthermore, many examples and problems were inspired by reading non-mathematical papers in which related situations are dealt with by different methods. (That newer texts now quote these non-mathe- matical papers as containing my examples shows how fast probability has developed, but also indicates the limited usefulness of quotations.) Lack of space as well as of competence precluded more adequate historical indications of how probability has changed from the semi- inysterious discussions of the "twenties Lo ils present flourishing state, For a number of years I have been privileged to work with students and younger colleagues to whose help and inspiration I owe much. . Army Research Office for work in probability at Princeton University. My particular thanks are due to Jay Goldman for a thoughtful memorandum about his teaching experiences, and to Loren Pitt for devoted help with the proofs. Muvit wedit fur this is duc ty ihe support by tie WILLIAM FELLER July, 1967Preface to the First Edition IT WAS THE AUTIIOR’S ORIGINAL INTENTION TO WRITE A BOOK ON analytical methods in probability theory in which the latter was to be treated as a topic in pure mathematics. Such a treatment would have been more uniform and hence more satisfactory from an aesthetic point of view; it would also have been more appealing to pure mathematicians, However, the generous support by the Office of Naval Research of work i bability theory at Corucl! University ied ie auiiwr iv a more ambitious and less thankful undertaking of satisfying heterogeneous needs. It is the purpose of this book to treat probability theory as a self- contained mathematical subject rigorously, avoiding non-mathematic concepts. At the same time, the book tries to describe the empirical background and to develop a feeling for the great variety of practical applications. This purpose is served by many special problems, numerical estimates, and examples which interrupt the main flow of the text. They are clearly set apart in print and are treated in a more picturesque language and with less formality. A number of special topics have been included in order to exhibit the power of general methods and to increase the usefulness of the book to specialists in various fields. To facilitate reading, detours from the main path are indicated by stars. The knowledge of starred sections is not assumed in the remainder. A serious attempt has been made to unify methods. The specialist will find many simplifications of existing proofs and also new results. In particular, the theory of recurrent events has been developed for the purpose of this book. It leads to a new treatnient of Markov chains which permits simplification even in the finite case. The examples are accompanied by ahout 340 problems mostly with complete solutions. Some of them are simple exercises, but most of them serve as additional illustrative material to the text or contain various complements. One purpose of the examples and problems is to develop the reader's intuition and art of probabilistic formulation. Several previously treated examples show that apparently difficult problems may become almost trite ance they are formulated in a natural way and put into the proper context. ixx PREFACE TO THE FIRST EDITION There is a tendency in teaching to reduce probability problems to pure analysis as soon as possible and to forget the specific characteristics of probability theory itself. Such treatments are based on a poorly defined notion of random variables usually introduced at the outset. This book goes to the other extreme and dwells on the notion of sample space, without which random variables remain an artifice, In order to present the true background unhampered by measurability questions and other purely analytic difficulties this valume is restricted to discrete sample spaces. This restriction is severe, but should be welcome to non-mathematical users. It permits the inclusion of special topics which are not easily accessible in the literature. At the same time, this arrangement makes it possible to begin in an elementary way and yet to include a fairly exhaustive treatment of such advanced topics as random walks and Markov chains. The general theory of random variables and their distributions, limit theorems, diffusion theory, etc., is deferred to a succeeding volume. This book would not have been written without the support of the Office of Naval Research. One consequence of this support was a fairly regular personal contact with J. L. Doob, whose constant criticism and encouragement were invaluable. To him go my foremost thanks. The next thanks for help are due to John Riordan, who followed the manu- script through two versions. Numerous corrections and improvements were suggested by my wife who read both the manuscript and proof. The author is also indebted to K. L. Chung, M. Donsker, and S. Goldberg, who read the manuscript and corrected various mistakes; the solutions to the majority of the problems were prepared by S. Goldberg. Finally, thanks are due to Kathryn Hollenbach for patient and expert typing help; to E. Elyash, W. Hoffman, and J. R. Kinney for help in proofreading. WILLiaM FELLER Cornell University January 1950Note on the Use of the Book ‘THE EXPOSITION CONTAINS MANY SIDE EXCURSIONS AND DOES NOT ALWAYS progress from the easy to the difficult; comparatively technical sections appear at the beginning and easy sections in chapters XV and XVII. Inexperienced readers should not attempt to follow many side lines, lest they lose sight of the forest for too many trees. Introductory remarks to the chapters and stars at the beginnings of sections should facilitate jentation and # ce of 0: "The unctarred sections fo: self-contained whole in which the starred sections are not used. A first introduction to the basic notions of probability is contained in chapters I, V, VI, IX; beginners should cover these with as few digressions as possible. Chapter IT is designed to develop the student’s technique and probabilistic intuition; some experience in its contents is desirable, but it is not necessary to cover the chapter systematically: it may prove more profitable to return to the elementary illustrations as occasion arises at later stages. For the purposes of a first introduction, the elementary theory of continuous distributions requires little supplementary explana- tion. (The elementary chapters of volume 2 now provide a suitable text.) From chapter TX an introductory course may proceed directly to chapter XI, considering generating functions as an example of more general transforms. Chapter XI should be followed by some applications in chapters XIIT (rccurtent events) or XI (chain reactions, infinitely divisible distributions). Without generating functions it is possible to turn in one of the following directions: limit theorems and fluctuation theory (chapters VIM, X, TI); stochastic processes (chapter XVII), random walks (chapter III and the main part of XIV). These chapters are almost independent of each other. The Markov chains of chapter XV depend conceptually on recurrent events, but they may be studied independently if the reader is willing to accept without proof the basic ergodic theorem. Chapter TIT stands by itself. Ils contents are appealing in their own right, but the chapter is also highly illustrative for new insights and new methods in probability theory. The results concerning fluctuations in xixii NOTE ON THE USE OF THE BOOK coin tossing show that widely held beliefs about the law of large numbers are fallacious. They are so amazing and so at variance with common intuition that even sophisticated colleagues doubted that coins actually misbehave as theory predicts. The record of a simulated experiment is therefore included in section 6. The chapter treats only the simple coin- tossing game, but the results are representative of a fairly general situation. The sign > is used to indicate the end of a proof or of a collection of examples. It is hoped that the extensive index will facilitate coordination between the several parts.Contents CHAPTER Pace INTRODUCTION: THE NATURE OF PROBABILITY THEORY. 1 1, The Background 2... ee -. 1 2, Procedure... 2... eee eee 3 3. “Statistical” Probability... .. 0.0... 4 4. Summary... .. ee ee 5 5. Historical Note... . : 6 T Tew Sample SPA. 6. 1 1, The Empirical Background... 2. 2... 1 2. Examples 2.2... eee ee ee 9 3. The Sample Space. Events... ......-.. 13 4, Relations among Events... 2... 0.0.0... 14 5, Discrete Sample Spaces... ...... -.: 6. Probabilities in Discrete Sample Spaces: Preparations 19 7. The Basic Definitions and Rules ...... 2. 22 8. Problems for Solution... ..........-. 24 Tl ELEMENTS OF COMBINATORIAL ANALYSIS... . . 26 1, Preliminaries. 2... 2. ee ee ee 26 2. Ordered Samples... . 2... oe + 2B 3. Fxamples : 31 4, Subpopulations and Partitions... . 2... .. 34 *5, Application to Occupancy Problems... . .. . 38 *Sa_ Bose-Finstein and Fermi-Dirac Statistics 40 *5b. Application to Runs... 2.2... 0... . 42 6. ‘The Hypergeometric Distribution... 2... 4B 7. Fxamples for Waiting Times 47 8. Binomial Coefficients ..........- : 50 9. Stirling's Formula... 22. ee 2 Problems for Sohition: 54 10. Exercises and Examples .. 2... 2.0... 54 * Starred sections are not required for the understanding of the sequel and should be omitted at first reading, xlxiv CHAPTER +O *1V VI CONTENTS 11. Problems and Complements of a Theoretical Character... . . : 12, Problems and Identities Involving Binomial Co- efficients... 2... FLUCTUATIONS IN CoIN TossiNc AND RANDOM WALKS . 1, General Orientation. The Reflection Principle . . 2. Random Walks: Basic Notions and Notations . . . 3. The MainLemma....... Fee 4. Last Visits and Long Leads... 2... tei *5. Changes ofSign 2... : 5.52. 11a. 6. An Experimental Illustration... 2... 2 0. 7. Maxima and First Passages... 2... sees 8. Duality. Position of Maxima... ss... 10. : ComminaTION OF EVENTS : 4. UnionofEvents ... =... 2. Application to the Classical Occupancy Problem 3. The Realization of m among N events : 4. Application to Matching and Guessing... . . . 5. Miscellany... 22... .---00. sae 6. Problems for Solution : CONDITIONAL PROBABILITY. STOCHASTIC INDEPENDENCE . 1. Conditional Probability . 2, Probabilities Defined by Conditional Probabilities. Um Models... 2-2. tee 3. Stochastic Independence 4, Product Spaces. Independent Trials... . . . *5, Applications to Genetics... 2. 6, Sex-Linked Characters #7, Selectof. ee et ss 8. Problems tor Solution... 2... 2... THe BINOMIAL AND THE POISSON DiSTRIBUTIONS . . . 1, Bernoulli Irials.. 2... ee ee ee eee 2. The Binomial Distribution... 3. The Central Term and the Tails... . 2.2... 4. ‘The Law of Large Numbers... 22... Pace 58 63 67 68 B 16 8 84 86 88 91 95 9g 98 101 106 107 109 Ml 114 14 118 125 128 132 136 139 140 146 146 147 150CHAPTER 5, The Poisson Approximation... 2.2... 6. The Poisson Distribution. . . . fee 1. Observations Fitting the Poisson Distribution . 8. Waiting Times. The Negative Binomial Distribution 9. The Muitinomial Distribution. ....... 10. Problems for Solution VIL THE NORMAL AvPRUXIMALION 10 1HE BINOMIAL *VIIL Ix x CONTENTS Dimon. 1. The Normal Distribution. : Orientation: Symmetric Distributions ‘The DeMoivre-Laplace Limit Theorem. . . . . Examples 2... 0... Relation to the Poisson Approximation cae Large Deviations... 2... . bee ee Problems for SOWGO6 0 c66 ks ek es . ag vay UNLIMITED SEQUENCES OF BERNOULLI TRIALS . . 1. Infinite Sequences of Trials... 2... 2. 2. Systems of Gambling... 0... oe 3. The Borel-Cantelli Lemmas... 2... 2... 4, The Strong Law of Large Numbers . 5. The Law of the Iterated Logarithm... 2... 6. Interpretation in Number Theory Language = . 7. Problems for Solution : RANDOM VARIABLES; EXPECTATION. ©... . 1. Random Variables .. 2.2.2... 2, Expectations»... . bee 3. Examples and Applications...) ss. 4, The Variance... ‘ : 5, Covariance; Variance ofa Sum oe ee 6. Chebyshev’s Inequality... 2... *7. Kolmogorov’s Inequality +8. The Correlation Coefficient... . an 9. Problems for Solution Laws or Larcc Nummers. . 2... 002 ee 1, Identically Distributed Variables... . . : *2, Proof of the Law of Large Numbers... . 2. xv Pace 153 156 159 164 167 169 174 174 179 182 187 190 192 193 196 196 198 200 202 204 208 210 212 212 220 223 227 29 233 234 236 237 243 243 246