Stochastic Process

Classes of stochastic processes: o White noise A continuous-time process is called white noise if for arbitrary n, sampling at arbitrary time instants t_1, t_2, ..., t_n, the resulting random variables, X_{t_1}, X_{t_2}, ..., X_{t_n} are independent, i.e., their joint pdf f(x_1, x_2, ..., x_n)= f(x_1)*f(x_2)*...*f(x_n). marginal distribution is enough to determine joint pdf of all orders. o Gaussian processes: used to model noise white Gaussian noise: marginal pdf is Gaussian. colored (and wide sense stationary) Gaussian noise: characterized by marginal distribution and autocorrelation R(\tau). heavily used in communication theory and signal processing, due to 1) Gaussian assumption is valid in many practical situations, and 2) easy to obtain close-form solutions with Gaussian processes. E.g., Q function and Kalman filter. o Poisson processes: used to model arrival processes heavily used in queueing theory, due to 1) Poisson assumption is valid in many practical situations, and 2) easy to obtain close-form solutions with Poisson processes. E.g., M/M/1 and Jackson networks. o Renewal processes used to model arrival processes heavily used in queueing theory, e.g., M/G/1, G/M/1, G/G/1 o Markov processes: the queue in M/M/1 is a Markov process. o Semi-Markov processes the queue in M/G/1 and G/M/1 is a semi-Markov process. o Random walk o Brownian motion o Wiener process o Diffusion process o Self similar process, long range dependence (LRD) process, short range dependence (SRD) process o Mixing processes: which characterizes asymptotic decay in correlation over time. -mixing process -mixing process: implies -mixing process -mixing process: implies -mixing process -mixing process: implies both -mixing process and -mixing process

Ergodic transformation
Let be a measure-preserving transformation on a measure space (X,,). An element A of is T-invariant if A differs from T 1(A) by a set of measure zero, i.e. if

where B.

denotes the set-theoretic symmetric difference of A and

The transformation T is said to be ergodic if for every T-invariant element A of , either A or X\A has measure zero. Ergodic transformations capture a very common phenomenon in statistical physics. For instance, if one thinks of the measure space as a model for the particles of some gas contained in a bounded recipient, with X being a finite set of positions that the particles fill at any time and the counting measure on X, and if T(x) is the position of the particle x after one unit of time, then the assertion that T is ergodic means that any part of the gas which is not empty nor the whole recipient is mixed with its complement during one unit of time. This is of course a reasonable assumption from a physical point of view. In other words, for any A where 0< (A)<1, mixing must happen after transformation T, i.e., . That is, the system state can change to any state in the sample space (with non-zero transition probability or non-zero conditional probability density) after transformation T; every state is reachable with nonzero probability measure after transformation T. Given state X(t) at time t, the next state X(t+1) = T(X(t)); X(t+1) can take any value x with (X(t+1)=x)>0, where x satisfies (X(t)=x)>0. If T is ergodic transformation, then X(t+1)=T(X(t)) can reach any state reachable by X(t). Ergodic transformation could be applied integer number of times (discrete time); ergodic transformation can be extended to the case of continuous time. A stochastic process created by ergodic transformation is called ergodic process. A process possesses ergodic property if the time/empirical averages converge (to a r.v. or deterministic value) in some sense (almost sure, in probability, and in pth norm sense). o Strong law of large numbers: the sample average of i.i.d. random variables, each with finite mean and variance, converges to their expectation with probability one (a.s.).

o Weak law of large numbers: the sample average of i.i.d. random variables, each with finite mean and variance, converges to their expectation in probability. o Central limit theorem: the normalized sum of i.i.d. random variables, each with finite mean and variance, converges to a Gaussian r.v. (convergence in distribution). Specifically, the central limit theorem states that as the sample size n increases, the distribution of the sample average of these random variables approaches the normal distribution with a mean and variance 2 / n , irrespective of the shape of the original distribution. In other words, converges to a Gaussian r.v. of zero mean, unit variance. An ergodic process may not have ergodic property. o For example: at the start of the process X(t), we flip a fair coin, i.e., 50% probability of having head and 50% probability of having tail. If head appears, the process X(t) will always take a value of 5; if tail appears, the process X(t) will always take a value of 7. So the time average will be either 5 or 7, not equal to the expectation, which is 6.

Similar to Probability theory, the theory of stochastic process can be developed with non-measure theoretic probability theory or measure theoretic probability theory. How to characterize a stochastic process: o Use n-dimensional pdf (or cdf or pmf) of n random variable at n randomly selected time instants. (It is also called nth-order pdf). Generally, the ndimensional pdf is time varying. If it is time invariant, the stochastic process is stationary in the strict sense. To characterize the transient behavior of a queueing system (rather than the equilibrium behavior), we use time-varying marginal cdf F(q,t) of the queue length Q(t). Then the steady-state distribution F(q) is simply the limit of F(q,t) as t goes to infinity. o Use moments: expectation, auto-correlation, high-order statistics o Use spectrum: power spectral density: Fourier transform of the second-order moment bi-spectrum: Fourier transform of the third-order moment tri-spectrum: Fourier transform of the fourth-order moment poly-spectrum. Limit Theorems: o Ergodic theorems: sufficient condition for ergodic property. A process possesses ergodic property if the time/empirical averages converge (to a

r.v. or deterministic value) in some sense (almost sure, in probability, and in p-th mean sense). Laws of large numbers Mean Ergodic Theorems in L^p space Necessary condition for limiting sampling averages to be constants instead of random variable: the process has to be ergodic. (not ergodic property) o Central limit theorems: sufficient condition for normalized time averages converge to a Gaussian r.v. in distribution. Laws of large numbers o Weak law of large numbers (WLLN) Sample means converge to a numerical value (not necessarily statistical mean) in probability. o Strong law of large numbers (SLLN) Sample means converge to a numerical value (not necessarily statistical mean) with probability 1. (SLLN/WLLN) If X1, X2, ... are i.i.d. with finite mean \mu, then sample means converge to \mu with probability 1 and in probability. (Kolmogorov): If {X_i} are i.i.d. r.v.'s with E[|X_i|]<infinity and E[X_i]= \mu, then sample means converge to \mu with probability 1. For {X_i} are i.i.d. r.v.'s with E[|X_i|]<infinity, E[X_i]= \mu, and Var(X_i)=infinity, then sample means converge to \mu with probability 1. But the variance of sample means does not converge to 0. Actually, the variance of sample means is infinity. This is an example that convergence almost sure does not imply convergence in mean square sense. Mean Ergodic Theorems: o Sample means converge to a numerical value (not necessarily statistical mean) in mean square sense. o A stochastic process is said to be mean ergodic if its sample means converge to the expectation. Central limit theorems (CLT) o Normalized sample means converge to a Gaussian random variable in distribution. Normalized by the standard deviation of the sample mean. Like lim_{x goes to 0}x/x=1 (the limit of 0/0 is a constant), CLT characterizes that as n goes to infinity, (S_n-E(X)/(\sigma/sqrt(n)) converges to a r.v. N(0,1), i.e., convergence in distribution. By abusing notation a bit, lim_{n goes to \infty}(S_n-E(X)/ (\sigma/sqrt(n)) = Y, the limit of 0/0 is a r.v. SLLN/WLLN is about the re-centered sample mean converging to 0. CLT is about the limit of 0/0.

(Lindeberg-Levy): If {x_i} are i.i.d. and have finite mean m and finite variance \sigma^2 (\neq 0), then the CDF of [(\sum_{i=1}^n x_i /n) - m]/(\sigma/\sqrt{n}) converges to a Gaussian distribution with mean 0 and unity variance.

Comments: WLLN/SLLN does not require finite variance but they obtain convergence with probability and in probability, respectively; stronger than convergence in distribution in CLT. Why? The difference is that in WLLN/SLLN, sample means converge to a deterministic value rather than a random variable as in CLT. Since CLT also requires finite variance, CLT gives a stronger result than WLLN/SLLN. That is, WLLN/SLLN only tell that sample means converge to a deterministic value but WLLN/SLLN do not tell how sample means converge to the deterministic value (in what distribution?). CLT tells that the sample mean is asymptotically Gaussian distributed. Implication of CLT: The aggregation of random effects follows Gaussian distribution. We can use Gaussian approximation/assumption in practice and enjoy the ease of doing math with Gaussian r.v.'s. Proof1, Proof2 What's the intuition of CLT? Why do we have this phenomenon (sample means converge to a Gaussain r.v.)? For i.i.d. r.v.s with a heavy-tail or sub-exponential distribution, as long as the mean and variance are finite, the sequence satisfies CLT. When the Gaussian approximation under/over estimates the tail probability? If the tail of the distribution decays slower than the Gaussian (e.g., heavy-tail distributions), the Gaussian approximation under-estimates the tail probability, i.e., the actual tail probability is larger than the Gaussian approximation. If the tail of the distribution decays faster than the Gaussian (e.g., close-to-deterministic distributions), the Gaussian approximation over-estimates the tail probability, i.e., the actual tail probability is smaller than the Gaussian approximation. Maximum likelihood estimators of mean and variance of i.i.d. Xi Y Z0 1 n 1 1 n 1 = X i = + 0 + o( ), = ( X i ) 2 = + n + o( n ) n i =1 n 1 i =1 n n where Y0 is distributed as N(0,1); Z0 is a Gaussian r.v.

Non-identical case: If {x_i} are independent but not necessarily identically distributed, and if each x_i << \sum_{i=1}^n x_i for sufficiently large n, then the CDF of [(\sum_{i=1}^n x_i /n) - m]/(\sigma/\sqrt{n}) converges to a Gaussian distribution with mean 0 and unity variance.

Non-independent case: There are some theorems which treat the case of sums of non-independent variables, for instance the m-dependent central limit theorem, the martingale central limit theorem and the central limit theorem for mixing processes. How to characterize the correlation structure of a stochastic process? o auto-correlation function R(t1,t2)=E[X(t1)X(t2)] For wide-sense (covariance) stationary process, R(\tau) = R(t1,t1+\tau) for all t1 \in R. If the process is a white noise with zero mean, R(\tau) is a Dirac delta function, the magnitude of which is the doublesided power spectrum density of the white noise. Note that the variance of a r.v. at any time in a white process is infinity. If R(\tau) is a Dirac delta function, then the r.v.'s at any two different instant are orthogonal. In discrete time, we have similar conclusions: If a random sequence consists of i.i.d. r.v.'s, R(n) is a Kronecker delta function, the magnitude of which is the second moment of a r.v. If R(n) is a Kronecker delta function, then the r.v.'s at any two different instant are orthogonal. R(t1,t2) characterizes orthogonality between a process' two r.v.'s at different instant. Cross-reference: temporal (time) autocorrelation function of a deterministic process (which is energy limited): R(\tau)= \int_{-infinity}^{+infinity} X(t)*X(t+\tau) dt Discrete time: R(n) = \sum_{i=-infinity}^{+infinity} X(i)*X(i+n) o auto-covariance function C(t1,t2)=E[(X(t1)-E[X(t1)])(X(t1)-E[X(t1)])] For wide-sense (covariance) stationary process, C(\tau) = C(t1,t1+\tau) for all t1 \in R. If the process is a white noise, C(\tau) is a Dirac delta function, the magnitude of which is the double-sided power spectrum density of the white noise. If C(\tau) is a Dirac delta function, then the r.v.'s at any two different instant are (linearly) uncorrelated. In discrete time, we have similar conclusions:
o

If a random sequence consists of i.i.d. r.v.'s, C(n) is a Kronecker delta function, the magnitude of which is the variance of a r.v. If C(n) is a Kronecker delta function, then the r.v.'s at any two different instant are uncorrelated. C(t1,t2) characterizes linear correlation between a process' two r.v.'s at different instant. C(t1,t2)>0: positively correlated C(t1,t2)<0: negatively correlated C(t1,t2)=0: uncorrelated o mixing coefficient: Why is Toeplitz matrix important? o The covariance matrix of any wide-sense stationary discrete-time process is Toeplitz. o A n-by-n Toeplitz matrix T = [t_{i,j}], where t_{i,j}=a_{i-j}, and a_{-(n1)}, a_{-(n-2)}, ... a_{n-1} are constant numbers. Only 2n-1 numbers are enough to specify a n-by-n Toeplitz matrix. o In a word, the diagonals of a Toeplitz matrix is constant (constant-alongdiagonals). o Toeplitz matrix is constant-along-diagonals; circulant matrix is constant-along-diagonals and symmetric along diagonals. Toeplitz matrix: "right shift without rotation"; circulant matrix: "right shift with rotation". Why is circulant/cyclic matrix important? o Circulant matrices are used to approximate the behavior of Toeplitz matrices. o A n-by-n circulant matrix C=[c_{i,j}], where each row is a cyclic shift of the row above it. Denote the top row by {c_0, c_1, ..., c_{n-1}}. Other rows are cyclic shifts of this row. Only n numbers are enough to specify a n-by-n circulant matrix. o Circulant matrices are an especially tractable class of matrices since their inverse, product, and sums are also circulants and it is straightforward to construct inverse, product, and sums of circulants. The eigenvalues of such matrices can be easily and exactly found. Empirical/sample/time average (mean) Borel-Cantelli theorem o The first Borel-Cantelli theorem If sum_{n=1}^{\infty} Pr{A_n}<\infty, then Pr{limsup_{n goes to \infty} A_n}=0. Intuition (using a queue with infinite buffer size): if the sum of probability of queue length=n is finite, then the probability that the actual queue length is infinite is 0, i.e., the actual/expected queue length is finite with probability 1. E[Q]=sum_{n=1}^{\infty} Pr{Q>=n}, this is another way to compute expectation. o The second Borel-Cantelli theorem

If {A_n} are independent and sum_{i=1}^n Pr{A_i} diverges, then Pr{limsup_{n goes to \infty} A_n}=1.

1. First-order result (about mean): WLLN/SLLN Question: Can we use the sample mean to estimate the expectation? WLLN studies the sufficient conditions for P{| S n E[ X ] |< } n1, > 0 or p S n E[X ] (in probability). Let E[ X i ] = E[ X ] . If WLLN is satisfied for every 1 n n E[ X ] R , the estimator is called a consistent estimator. var{ S n } = 2 i , j . n i =1 j =1 p (Markov) If lim var{S n } = 0 , then S n E[X ] ; i.e., convergence in mss implies
n

convergence in probability. o For wide sense stationary LRD process with var{ X i } < and i ,i + n n 0 , we have S n p E[X ] . So in this LRD case, sample mean is a consistent estimator. (Khintchin) If { X i } are iid random variables with E[ X i ] < (and possibly infinite variance), then S n p E[X ] . Note that here { X i } converges in probability even if { X i } does not converge in mss (infinite variance case). SLLN studies the sufficient conditions for S n wp1 E[ X ] (with probability 1). (Kolmogorov) If { X i } are iid random variables with E[| X i |] < (and possibly infinite variance), then S n wp1 E[ X ] . Note that here { X i } converges with probability 1 even if { X i } does not converge in mss (infinite variance case). 2. Second-order result (about variance): convergence in mss and CLT mss Mean ergodic theorem: S n E[ X ] lim var{S n } = 0 .
n

Mean ergodic theorem (wide sense stationary, WSS): Let { X i } be WSS with E[ X i ] < . If var{ X i } < and covariance i ,i + n n 0 , then S n mss E[ X ] . n o For LRD process with var{ X i } < and i ,i + n 0 , we have mss S n E[ X ] . So in this LRD case, as the number of samples increases, the variance of the estimator (sample mean) reduces. o LRD with covariance
1 1 matrix 1 1 1 1 1 1 1 1 1 1 0.5 1 0.5 0.5 1 1 1 1

n has var{ S n } =1, , i.e.,

the variance of the estimator does not reduce due to more samples. o LRD with covariance
1 0.5 matrix .5 0 0 .5 0.5 0.5 1 0.5 0.5 0.5 has 0.5 1

lim var{ S n } = 0.5 , i.e., the variance of the estimator does not reduce to 0
n

due to infinite samples.

o SRD process always has lim var{S n } = 0 since lim i ,i + j < n n

implies i ,i + n 0 . o Note that mean ergodic theorem only tells the condition for the variance of the sample mean to converge but does not tell the convergence rate. We expect a good estimator has a fast convergence rate, i.e., for a fixed n, it should have small variance; in terms of convergence rate, we know the relation iid>SRD>LRD holds, i.e., iid sequence has the highest convergence rate. CLT: o Motivation: WLLN tells the first-order behavior of the estimator, sample mean; i.e., sample means converge to the expectation and we have unbiased estimator since E[ S n ] = E[ X ]. Mean ergodic theorem tells the second-order behavior of the estimator; i.e., the variance of the sample means converges to 0. The next question is: what about the distribution of the estimator? This is answered by CLT. o (Lindeberg-Levy CLT): If { X i } are iid random variables with E[ X i ] < and var{ X i } < , then S n d Y , where Y is a normal random variable with mean E[ X i ] and variance var{ X i } / n . 2 d Denote S n ~ N ( , / n) . I.e., ( S n E[ X i ]) / var{ X i } / n Y0 , where Y0 is a normal random variable with zero mean and unity variance. o How to use CLT: given the normal distribution of the estimator (sample means), we can evaluate the performance of the estimator (e.g., confidence interval). That is, CLT provides approximations to, or limits of, performance measures (e.g., confidence interval) for the estimator, as the sample size gets large. So, we can compute the confidence interval according to the normal distribution. 2 2 Since S n ~ N ( , / n) , we also have ~ N ( S n , / n) . Then we can compute the confidence interval of . o CLT does not apply to self-similar processes since the randomness does not average out for self-similar processes. The sample means have the same (non-zero) variance; i.e., even if the number of samples goes to infinity, the variance of the sample mean does not go to zero. o CLT only use the expectation and variance of the sample mean since the sample mean is asymptotically normal. High order statistics of the sample mean are ignored. Why is the limiting distribution Gaussian? In the proof using moment-generating function, we see that the high-order terms O (n 2 ) are gone since first order and second order statistics dominate the value of the moment-generating function as n goes to infinity. Intuitively, first order and second order statistics can completely characterize the distribution in the asymptotic region, since the randomness is averaged out; we only need to capture the mean and variance (energy) of the sample mean in the asymptotic domain.
n

Gaussian distribution is maximum entropy distribution under the constraints on mean and variance. o What about LRD, subexponential distribution, heavy-tail distribution? Large deviation theory o Motivation: WLLN tells P{| S n E[ X ] | } n 0, > 0. But it does not tell how fast it converges to zero. Large deviation theory characterizes the convergence rate. Why is this important? Because in many applications, we need to compute the probability of a rare event like | S n E[ X ] | , where is large. o Large deviation principle (LDP) is about computing a small probability P{| S n E[ X ] | } for sufficiently large n, more specifically, n 1 1 lim sup log P{| S n E[ X ] | } or lim sup log P{ X i na}, for n n n n i =1 a>E[X]. LDP characterizes the probability of a large deviation of

X
i =1

(n in the order ) about the expectation n*E[X]; i.e., a large deviation of ( S n in the order 1) about the expectation E[X].
2 o In contrast, CLT is for computing lim P{( S n E[ X ]) / / n x} . It n

characterizes the probability of a small deviation in the order O ( n 1 / 2 ) about the expectation. As n goes to infinity, O ( n 1 / 2 ) goes to 0; so it is a small deviation with respect to a fixed expectation. On the other hand, this deviation O ( n 1 / 2 ) is in the same order of the variance of the sample mean, so that this deviation normalized by the variance is still finite. o For LRD process with var{ X i } < and i ,i + n n 0 , even though we n have WLLN, i.e., P{| S n E[ X ] | } 0, > 0 , the probability P{| S n E[ X ] | } does not satisfy large deviation principle (i.e., the convergence rate is not exponential). We need to compress the time axis to obtain large deviation principle. White Gaussian noise: The auto-correlation function: R ( ) = 2 ( ) . f The power spectral density: S ( f ) = 2 , ( ,+ ). The marginal PDF is Gaussian with zero mean, infinite variance. If the white Gaussian noise passes a filer with frequency response H(f) in frequency band [B,B], the resulting marginal PDF is Gaussian with zero mean, variance

| H ( f ) |

df .

Markov property: Markov property is good since you can reduce the dimensionality of the sufficient statistics using conditional independence. But there exists stationary sources that are not Markovian of any finite order; that is, there isnt conditional independence and you cannot reduce the dimensionality of the sufficient statistics; you have to use all the data to construct the sufficient statistics.

Classification of stochastic processes: Memoryless processes: Poisson process, Bernoulli process Short-memory processes: sum of the auto-correlations (actually, auto-covariance coefficients) is finite. (SRD) Markov processes (having memoryless property, i.e., conditional independence) possess short memory. E.g., AR(1) has a memory of 1 time unit. Long-memory processes: sum of the auto-correlations (actually, auto-covariance coefficients) is infinite. (LRD, self similar, sub-exponential distribution, heavytail distribution)