Lecture 1

1 Course Introduction

Probability theory has its roots in games of chance, such as coin tosses or throwing dice. By playing these games, one develops some probabilistic intuition. Such intuition guided the early development of probability theory, which is mostly concerned with experiments (such as tossing a coin or throwing a die) with nitely many possible outcomes. The extension to experiments with innitely (even uncountably) many possible outcomes, such as sampling a real number uniformly from the interval [0, 1], requires for more sophisticated mathematical tools. This is accomplished by Kolmogorovs axiomatic formulation of probability theory using measure theory, which lays the foundation for modern probability theory. Therefore we will rst recall some basic measure theory and Kolmogorovs formulation of probability space and random variables. In this course, we will focus on the study of a sequence of independent real-valued random variables. In particular, we will study the empirical average of a sequence of independent and identically distributed (i.i.d.) real-valued random variables and prove the Law of Large Numbers (LLN), as well as the Central Limit Theorem (CLT) which governs the uctuation of the empirical average. Along the way we will study Fourier transforms of probability measures and dierent notions of convergence of probability measures, in particular the weak convergence. Other topics we aim to cover, which arises from the study of sums of independent random variables, include: innitely divisible distributions, stable distributions, large deviations, extreme order statistics. See the bibliography for references, with [1, 2] being our primary references. If time permits, we will also show how to use measure theory to construct conditional probabilities/expectations when we condition on events of probability 0, which is needed when we study experiments (random variables) with uncountably many possible outcomes. Topics on dependent random variables, such as Markov chains, martingales, and stationary processes, will be covered in a second course. Topics on continuous time processes, in particular stochastic calculus and stochastic dierential equations, is usually covered in a third course. Other topics, such as Lvy processes, large deviations, interacting particle systems, e percolation, stochastic partial dierential equations, etc, are the subjects of special topics courses.

Probability Space

Let us rst motivate the measure-theoretic formulation of probability theory. Let be the so-called probability space, which is the space of possible outcomes for an experiment. If the experiment is the throw of a die, then we should take := {1, 2, , 6}, and specify a probability function f : [0, 1] with i f (i) = 1 such that f (i) is the probability of seeing the outcome i. If the experiment has an uncountable number of outcomes, such as drawing a random number uniformly from [0, 1], then we should take := [0, 1]. However, there is no sensible way of dening a probability function f : [0, 1] with f (x) = 0 for all x [0, 1] and x f (x) = 1 (the sum is undened). 1

An alternative is to dene probabilities for sets of outcomes in , also called events. Thus we also introduce F, a collection of events (subsets of ), and a set function P : F [0, 1] such that P (A) is the probability of the event A F. Since F is the collection of events for which we can determine the probabilities using P (), the larger is F, the more information we have. We expect F and P to satisfy some natural conditions: F and F (we expect P () = 1 and P () = 0), If A F, then Ac F as well (we expect P (Ac ) = 1 P (A) 0), If A, B F, then A B, A B F (we expect P (A B) = P (A) + P (B) if A B = ). A collection of sets F satisfying the above properties is called an algebra (or eld). A set function P () satisfying the above properties is called a nitely-additive probability measure. An important technical condition we need to further impose on F is that, F is a -algebra (or -eld), i.e., nN An F (or equivalently nN An F) if An F for each n N. Similarly, we need to further assume that P is a countably-additive probability measure on the -algebra F, i.e., if (An )nN is a sequence of pairwise-disjoint sets in F, then P(nN An ) = nN P (An ). It is easy to see that Exercise 2.1 A nitely-additive probability measure P on a -algebra F satises countable additivity, i.e., P(nN An ) = nN P (An ) for any sequence of pairwise-disjoint sets An F, if and only if P (Bn ) 0 for any sequence of Bn F decreasing to the empty set . We can now give Kolmogorovs formulation of a probability space: Denition 2.2 [Probability space] A probability space is a triple (, F, P ), where is a set, F is a -algebra on , and P is a probability measure (always assumed to be countablyadditive) on the measurable space (, F). If is a nite or countable set, then a natural choice of F is the collection of all subsets of , and specifying P becomes equivalent to specifying a probability function f : [0, 1] with x f (x) = 1. When is uncountable, a natural question is: How to construct -elds F on and countably-additive probability measures P on F? The rst part of this question is addressed in Exercise 2.3 If B is an algebra on , then there is a unique -algebra F such that it is the smallest -algebra containing B. We call F the -algebra generated by B. The second part of the above question is addressed by Theorem 2.4 [Caratheodory Extension Theorem] If P is a countably-additive probability measure on an algebra B, then P extends uniquely to a countably-additive probability measure on the -algebra F generated by B. The proof of Theorem 2.4 can be found in any of the references in the bibliography; see [2, Sec. 1.2] for a proof sketch. Theorem 2.4 reduces the construction of countably additive probability measures on -algebras to the construction of countably additive probability measures on algebras. We now focus on the case = R. A natural -algebra on R (in fact for any topological space) is the Borel -algebra B, which is the smallest -algebra containing all the open and closed sets. It turns out that there is a one-to-one correspondence between probability measures on (R, B) and their associated distribution functions. 2

Denition 2.5 [Distribution Function] Let P be a countably additive probability measure on (R, B). Then F : R [0, 1] dened by F (x) := P ((, x]) is called the distribution function of P . Theorem 2.6 [Correspondence between Distribution Functions and Probability Measures on R] If F is the distribution function of a countably-additive probability measure P on (R, B), then F is non-decreasing and right-continuous, with F () := limx F (x) = 0 and F () = 1. Conversely, any non-decreasing right-continuous function F : R [0, 1] with F () = 0 and F () = 1 denes a unique countably-additive probability measure P on (R, B) with P ((, x]) = F (x) for all x R. Proof. If F is the distribution function of P , then F (y) F (x) = P ((x, y]) 0 for all x y, while the countable-additivity of P implies F () = limx P ((, x]) = 0, F () = 1, and F (x + ) F (x) = P ((x, x + ]) 0 as 0. Conversely, if F is non-decreasing and right-continuous with F () = 0 and F () = 1, then we can dene a set function P on intervals of the form (x, y], with x y, by P ((x, y]) := F (y) F (x). Note that nite disjoint unions of such intervals (including ) form an algebra I, and P extends to a nitely-additive probability measure on I. Futhermore, we note that I generates (via countable union and countable intersection) open and closed intervals on R, and hence B is the -algebra generated by I. Therefore it only remains to show that P is countably-additive on I, so that we can then apply Caratheodory Extension Theorem to conclude that P extends uniquely to a probability measure on (R, B). Let An I with An . By Exercise 2.1, we need to show that P (An ) 0. First we claim that it suces to verify P (An (l, l]) 0 for any l > 0, which allows us to replace An by its truncation Al := An (l, l]. Indeed, note that n P (An ) P (Al ) + P ((, l]) + P ((l, )) = P (Al ) + F (l) + (1 F (l)), n n where we can rst send n and then make F (l) + (1 F (l)) arbitrarily small by picking l suciently large (possible because F () = liml F (l) = 0 and F () = 1). Given Al , suppose that P (Al ) > 0. We will derive a contradiction by constructing n n a decreasing sequence of non-empty closed subsets Dn Al . Since Al I, we can write Al n n n as the disjoint union of intervals kn (an,i , bn,i ]. Since the right-continuity of F implies that i=1 for any x R, P ((x, x + ]) = F (x + ) F (x) 0 as 0, we can choose en,i (an,i , bn,i ) such that Bn := kn (en,i , bn,i ] Al has P (Al \Bn ) /10n . Let En := n Bi . Then n n i=1 i=1 n Al = n (Bi (Al \Bi ) n Bi i=1 i i=1 i i=1 and hence
n

n (Al \Bi ), i=1 i

P (En ) =

P (n Bi ) i=1

P (n Al ) i=1 i

i=1

P (Al \Bi ) i

i=1

/10i > /2.

Therefore En = . Note that Bi Al , and hence Dn := En = n Bi Al is a decreasing n i i=1 D Al must contain at least one sequence of closed subsets of [l, l]. Therefore n=1 n n=1 n point, which contradicts our assumption. Remark 2.7 Similar to R, on Rd , nite disjoint unions of rectangles of the form (a1 , b1 ] (a2 , b2 ] (ad , bd ] forms an algebra which generates the Borel -algebra B on Rd . 3

Random Variable, Distribution and Expectation

In the previous section, we interpreted the probability space as the space of possible outcomes of an experiment. When we have multiple experiments with dierent spaces for their outcomes, its more instructive to use an abstract probability space (, F, P) from which all randomness originate. The experiments we perform are then realized as deterministic functions of the outcome of the experiment we perform on the abstract probability space (, F, P). This leads to the formulation of an experiment as Denition 3.1 [Random Variable] A real-valued random variable X is a measurable map from X : (, F) (R, B), i.e., for each Borel set A B, we have X 1 (A) F. Remark 3.2 A random variable X taking values in a general measurable space (E, G) (e.g., (Rd , B) or any complete separable metric space equipped with the Borel -algebra) is just a measurable map from (, F) to (E, G). Multiple measurable functions can be dened on (, F), leading to (generally dependent) random variables taking values in possibly dierent spaces. Exercise 3.3 Let X be a measurable map from (, F, P) to a measurable space (E, G), equipped with a -algebra G. Then the set function Q : G [0, 1] dened by Q(A) := (P X 1 )(A) = P(X 1 (A)), for all A G, is a probability measure on (E, G). Denition 3.4 [Distribution of a Random Variable] Let X : (, F, P) (E, G) be an E-valued random variable. The induced probability measure P X 1 on (E, G) is called the distribution of X under P. Remark 3.5 If X1 , , Xd : (, F, P) (R, B) are d real-valued random variables, then it can be shown that X := (X1 , , Xd ) : Rd is an (Rd , B(Rd ))-valued random variable. The induced probability measure P X 1 on (Rd , B(Rd )) is called the joint distribution of X1 , X2 , and Xd . Remark 3.6 The study of random variables on a nice enough measurable space (E, G) (in particular, complete separable metric space with Borel -algebra) can be reduced to the study of real-valued random variables. All we need to do is to apply to X a suciently large class of measurable test functions {fi }iI , with fi : (E, G) (R, B), so that we can determine the distribution of X from the joint distribution of {fi (X)}iI . Note that for measurable fi , fi (X) is a real-valued random variable. For a real-valued random variable X : (, F, P) (R, B), we need to dene the classic notion of expectation (or average) in our current measure-theoretic setting. This amounts to dening the integral X()P(d) of X on the probability space (, F, P), which calls for the theory of Lebesgue integration on a general measure space. Let us recall briey how Lebesgue integration is constructed. Firstly, for X of the form (called simple functions) X() := and ci R, we can dene the integral
k k i=1 ci 1Ai (),

with Ai F

X()P(d) :=
i=1

ci P(Ai ).

Note that linear combinations of simple functions are still simple, and the integral dened above is a linear operator on simple functions. Furthermore, the integral is a bounded operator on the space of simple function equipped with the supremum norm . More precisely, if X is simple, then X()P(d) X , where X := sup |X()|. Consequently, if Xn are simple functions with Xn X 0 for some limiting function X on (note that the limit of measurable functions is also measurable), then Xn P(d) must converge to a limit, which we dene to be XP(d). We then observe that every bounded measurable function X can be approximated in supremum norm by simple functions. Indeed, if we assume w.l.o.g. that X = 1, then we i can approximate X by Xn := n+1 i=n1 n 1An,i (), with An,i := { : X() [i/n, (i + 1)/n)}. Having constructed the integral for bounded measurable functions, we can then construct the integral for arbitrary non-negative measurable functions X by XP(d) := sup f ()P(d) : 0 f X, f < ,

and X is said to be integrable if XP(d) < . A general measurable function X is said to be integrable if its positive part X + := X 0 and negative part X = (X) 0 are both integrable, which is equivalent to |X| being integrable. In this case we then dene XP(d) := X + P(d) X P(d). |X|P(d), which is

The L1 (, F, P)-norm of a random variable X is dened by |X|1 := nite if and only if X is integrable.

For integrable random variables dened on the probability space (, F, P), we will introduce the notation E[X] to denote the integral of X over w.r.t. P, which is also called the expectation or mean of X. If we let denote the probability distribution of X on R under P, i.e., = P X 1 , then not surprisingly, one can show that Exercise 3.7 E[X] = R x (dx), and E[g(X)] = such that g is integrable w.r.t. on R.
R g(x)(dx)

for any g : (R, B) (R, B)

Convergence of Random Variables

For a sequence of random variables (Xn )nN dened on a probability space (, F, P), taking values in a metric space (E, G) with Borel -algebra, we have several notions of convergence. The rst is the notion of everywhere convergence, i.e., , X() := lim Xn () exists.
n

We leave it as an exercise to check that X is also a random variable. However, this notion of convergence is too strong because it is insensitive toward the probability measure P. A more sensible notion is Denition 4.1 [Almost Sure Convergence] A sequence of random variables (Xn )nN dened on (, F, P) is said to converge almost surely (abbreviated by a.s.) to a random variable X, if there exists a set o F with P(o ) = 1, such that Xn () X() for every o . 5

Almost sure convergence allows us to ignore what happens on a set of probability 0 w.r.t. P. A weaker notion is Denition 4.2 [Convergence in Probability] A sequence of random variables (Xn )nN dened on (, F, P) is said to converge in probability to a random variable X, if
n

lim P{ : |Xn () X()| } = 0

> 0.

Example 4.3 If we take (, F, P) := ([0, 1], B, ), the unit interval with Borel -algebra and Lebesgue measure, then Xn : [0, 1] R, dened by Xn () = n on [0, 1/n] and Xn () = 0 on (1/n, 1], is a sequence of random variables converging a.s. to X 0. If we dene instead Xn () := n on the interval ( n1 1/i, n 1/i] projected onto [0, 1] by identifying (k, k + 1] i=1 i=1 with (0, 1] for each k Z, and Xn () = 0 for other choices of , then Xn converges in probability (but not almost surely!) to X 0. Remark 4.4 [Convergence in Distribution] Both a.s. convergence and convergence in probability require the sequence of random variables Xn to be dened on the same probability space (, F, P). To phrase it in another way, we say that (Xn )nN are coupled. However, 1 what we are often interested in is rather the distribution of Xn , and whether n := P Xn converges in a suitable sense on the metric space (E, G) where (Xn )nN takes their values. This leads to the notion of convergence of Xn in distribution to X, or weak convergence of n to := P X 1 . The convergence of Xn to X in distribution is a statement about the distributions n and on (E, G), and has nothing to do with how (Xn )nN and X are coupled on (, F, P). However, if (Xn )nN and X are coupled in such a way that Xn X in probability (or even a.s.), then we can conclude that Xn converges to X in distribution. We will study in detail the notion of convergence in distribution for real-valued random variables before we study the Central Limit Theorem. We now collect some important results on the relation between the convergence of a sequence of real-valued random variables (Xn )nN , and the convergence of their expectations. We shall assume below that all random variables are real-valued and dened on a probability space (, F, P), with expectation denoted by E[]. Theorem 4.5 [Bounded Convergence Theorem] If (Xn )nN is a sequence of uniformly bounded random variables, and Xn converges in probability to X, then limn E[Xn ] = E[X]. Theorem 4.6 [Fatous Lemma] If (Xn )nN is a sequence of non-negative random variables and Xn X in probability, then E[X] lim inf n E[Xn ]. An easy way to remember the direction of the inequality above is to consider Example 4.3. Theorem 4.7 [Monotone Convergence Theorem] If (Xn )nN is a sequence of nonnegative random variables such that Xn X a.s., then limn E[Xn ] = E[X]. Theorem 4.8 [Dominated Convergence Theorem] If (Xn )nN is a sequence of random variables converging in probability to X, and there exists a random variable Y with E[|Y |] < and |Xn | Y a.s. for all n N, then limn E[Xn ] = E[X]. In practice, when we apply Theorems 4.54.8, its usually the case that Xn X in the stronger sense of a.s. convergence. Let us also recall here a very useful inequality: 6

Theorem 4.9 [Jensens Inequality] If X is an integrable random variable, and : R R is a convex function such that (X) is also integrable, then E[(X)] (E[X]). The proof of the above theorems can be found in any graduate textbook on analysis, or in any of the references below.

References
[1] R. Durrett. Probability: Theory and Examples, Duxbury Press. [2] S.R.S. Varadhan. Probability Theory, Courant Lecture Notes 7. [3] A. Klenke. Probability TheoryA Comprehensive Course, Springer-Verlag. [4] L. Breiman. Probability, Society for Industrial and Applied Mathematics. [5] W. Feller. An Introduction to Probability Theory and Its Applications, Vol II, John Riley & Sons, Inc. [6] K.L. Chung. A Course in Probability Theory, Academic Press.