Scribd
Upload
97 views12 pages

18.445 Introduction To Stochastic Processes: Lecture 3: Markov Chains: Time-Reversal

This lecture discusses Markov chains and stochastic processes. It introduces the ergodic theorem, time reversal of Markov chains, and birth-and-death chains. It also covers stationary distributions, detailed balance equations, reversible Markov chains, and the total variation distance as a way to measure the difference between probability measures.

Uploaded by

AdityaPandhare
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
97 views12 pages

18.445 Introduction To Stochastic Processes: Lecture 3: Markov Chains: Time-Reversal

This lecture discusses Markov chains and stochastic processes. It introduces the ergodic theorem, time reversal of Markov chains, and birth-and-death chains. It also covers stationary distributions, detailed balance equations, reversible Markov chains, and the total variation distance as a way to measure the difference between probability measures.

Uploaded by

AdityaPandhare
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 12

18.

445 Introduction to Stochastic Processes


Lecture 3: Markov chains: time-reversal

Hao Wu

MIT

18 February 2015

Hao Wu (MIT) 18.445 18 February 2015 1 / 11


Recall
Consider a Markov chain with state space Ω and transition matrix P :

P[Xn+1 = y | Xn = x] = P(x, y ).

A probability measure π is stationary if π = πP.


If P is irreducible, there exists a unique stationary distribution.
Today’s goal
Ergodic Theorem
Time-reversal of Markov chain
Birth-and-Death chains
Total variation distance

Hao Wu (MIT) 18.445 18 February 2015 2 / 11


Ergodic Theorem

Theorem
Let f be a real-valued function defined on Ω. If (Xn )n is an irreducible
Markov chain with stationary distribution π, then for any starting
distribution µ, we have
n
1X
lim f (Xj ) = πf , Pµ − a.s.
n→∞ n
j=0

In particular,
n
1X
lim 1[Xj =x] = π(x), Pµ − a.s.
n→∞ n
j=0

Hao Wu (MIT) 18.445 18 February 2015 3 / 11


Detailed balance equations

Definition
Suppose that a probability measrue π on Ω satisfies

π(x)P(x, y ) = π(y )P(y , x), ∀x, y ∈ Ω.

These are called detailed balance equations.

Lemma
Any distribution π satisfying the detailed balance equations is
stationary for P.

Definition
A chain satisfying detailed balance equations is called reversible.

Hao Wu (MIT) 18.445 18 February 2015 4 / 11


Simple random walk on graph

Example Consider simple random walk on graph G = (V , E) (which is


connected). The measure

deg(x)
π(x) = , x ∈Ω
2|E|

satisfies detailed balance equations ; therefore the simple random walk


on G is reversible.

Hao Wu (MIT) 18.445 18 February 2015 5 / 11


Time-reversal of Markov chain

Theorem
Let (Xn ) be an irreducible Markov chain with transition matrix P and
stationary distribution π. Define P
b to be

π(y )P(y , x)
P(x,
b y) = .
π(x)
P
b is stochastic
Let (Xbn ) be a Markov chain with transition matrix P.
b Then π is also
stationary for P.
b
For any x0 , ..., xn ∈ Ω, we have
Pπ [X0 = x0 , ..., Xn = xn ] = Pπ [X
b0 = xn , ..., X
bn = x0 ].
We call X
b the time-reversal of X .

Remark If a chain with transition matrix P is reversible, then P


b =P
and X has the same law as X .
b
Hao Wu (MIT) 18.445 18 February 2015 6 / 11
Birth-and-Death chains

A birth-and-death chain has state space Ω = {0, 1, ..., N}.


The current state can be though of as the size of some population ; in a
single step of the chain there can be at most one birth or death. The
transition probabilities can be specified by {(pk , rk , qk )N
k =0 } where
pk + rk + qk = 1 for each k and
pk is the probability of moving from k to k + 1 when 0 ≤ k < N ;
pN = 0
qk is the probability of moving from k to k − 1 when 0 < k ≤ N ;
q0 = 0
rk is the probability of remaining at k when 0 ≤ k ≤ N.

Theorem
Every birth-and-death chain is reversible.

Hao Wu (MIT) 18.445 18 February 2015 7 / 11


Total variation distance

Definition
The total variation distance between two probability measures µ and ν
on Ω is defined by

||µ − ν||TV = max |µ(A) − ν(A)|.


A⊂Ω

Lemma
The total variation distance satisfies triangle inequality :

||µ − ν||TV ≤ ||µ − η||TV + ||η − ν||TV .

Hao Wu (MIT) 18.445 18 February 2015 8 / 11


Three ways to characterize the total variation distance

Lemma
1X
||µ − ν||TV = |µ(x) − ν(x)|.
2
x∈Ω

Lemma
1
||µ − ν||TV = sup{µf − νf : f satisfying max |f (x)| ≤ 1}.
2 x∈Ω

Hao Wu (MIT) 18.445 18 February 2015 9 / 11


Three ways to characterize the total variation distance

Definition
A coupling of two probability measures µ and ν is a pair of random
variables (X , Y ) defined on the same probability space such that the
marginal law of X is µ and the marginal law of Y is ν.

Lemma
||µ − ν||TV = inf{P[X =
6 Y ] : (X , Y ) is a coupling of µ, ν}.

Definition
6 Y ] = ||µ − ν||TV .
We call (X , Y ) the optimal coupling if P[X =

Hao Wu (MIT) 18.445 18 February 2015 10 / 11


Homework 1 due Feb. 23rd

Hao Wu (MIT) 18.445 18 February 2015 11 / 11

MIT OpenCourseWare
http://ocw.mit.edu

18.445 Introduction to Stochastic Processes


Spring 2015

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

You might also like