Scribd
Upload
12 views19 pages

Lecture 19 Compressed

The lecture focuses on Markov chains, their properties, and applications, including the gambler's ruin problem. Key concepts discussed include stationary distributions, periodicity, and the fundamental theorem of Markov chains. Examples illustrate the unique stationary distribution in various scenarios, such as shuffling cards and random walks on undirected graphs.

Uploaded by

chaimaalabyad647
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views19 pages

Lecture 19 Compressed

The lecture focuses on Markov chains, their properties, and applications, including the gambler's ruin problem. Key concepts discussed include stationary distributions, periodicity, and the fundamental theorem of Markov chains. Examples illustrate the unique stationary distribution in various scenarios, such as shuffling cards and random walks on undirected graphs.

Uploaded by

chaimaalabyad647
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

COMPSCI 614: Randomized Algorithms with

Applications to Data Science

Prof. Cameron Musco


University of Massachusetts Amherst. Spring 2024.
Lecture 19

1
Logistics

• I will send responses to project progress reports soon.

2
Summary

Last Week: Start on Markov Chains.

• Start on Markov chains and their analysis


• Markov chain based algorithms for satisfiability: ≈ n2 time for
2-SAT, and ≈ (4/3)n for 3-SAT.

Today: Markov Chains Continued

• The gambler’s ruin problem.


• Aperiodicity and stationary distribution of a Markov chain.
• The fundamental theorem of Markov chains.

3
Markov Chain Review

• A discrete time stochastic process is a Markov chain if is it


memoryless:
Pr(Xt = at |Xt−1 = at−1 , . . . , X0 = a0 ) = Pr(Xt = at |Xt−1 = at−1 )

• If each Xt can take m possible values, the Markov chain is


specified by the transition matrix P ∈ [0, 1]m×m with
Pi,j = Pr(Xt+1 = j|Xt = i).

• Let qt ∈ [0, 1]1×m be the distribution of Xi . Then qt+1 = qt P.

4
Markov Chain Review

Often viewed as an underlying state transition graph. Nodes


correspond to possible values that each Xt can take.

The Markov chain is irreducible if the underlying graph consists of


single strongly connected component.

5
Gambler’s Ruin

5
Gambler’s Ruin

• You and ‘a friend’ repeatedly toss a fair coin. If it hits heads, you
give your friend $1. If it hits tails, they give you $1.
• You start with $ℓ1 and your friend starts with $ℓ2 . When either of
you runs out of money the game terminates.
• What is the probability that you win $ℓ2 ?

6
Gambler’s Ruin Markov Chain

Let X0 , X1 , . . . be the Markov chain where Xt is your profit at


step t. X0 = 0 and:
P−ℓ1 ,−ℓ1 = Pℓ2 ,ℓ2 = 1
Pi,i+1 = Pi,i−1 = 1/2 for − ℓ1 < i < ℓ2

• ℓ1 and ℓ2 are absorbing states.


• All i with −ℓ1 < i < ℓ2 are transient states. I.e.,
Pr[Xt′ = i for some t′ > t | Xt = i] < 1.
Observe that this Markov chain is also a Martingale since
E[Xt+1 |Xt ] = Xt .
7
Gambler’s Ruin Analysis

Let X0 , X1 , . . . be the Markov chain where Xt is your profit at step t.


X0 = 0 and:
P−ℓ1 ,−ℓ1 = Pℓ2 ,ℓ2 = 1
Pi,i+1 = Pi,i−1 = 1/2 for − ℓ1 < i < ℓ2

We want to compute q = limt→∞ Pr[Xt = ℓ2 ].

By linearity of expectation, for any i, E[Xi ] = 0. Further, for


q = limt→∞ Pr[Xt = ℓ2 ], since −ℓ1 , ℓ2 are the only non-transient states,

lim E[Xt ] = ℓ2 q + −ℓ1 (1 − q) = 0.


t→∞

Solving for q, we have q = ℓ1 +ℓ2 .


ℓ1

8
Gambler’s Ruin Thought Exercise

What if you always walk away as soon as you win just $1. Then
what is your probability of winning, and what are your
expected winnings?

9
Stationary Distributions

9
Stationary Distribution

A stationary distribution of a Markov chain with transition matrix


P ∈ [0, 1]m×m is a distribution π ∈ [0, 1]m such that π = πP.
I.e. if Xt ∼ π, then Xt+1 ∼ πP = π.

Think-pair-share: Do all Markov chains have a stationary


distribution?

10
Claim (Existence of Stationary Distribution)
Any Markov chain with a finite state space, and transition
matrix P ∈ [0, 1]m×m has a stationary distribution π ∈ [0, 1]m
with π = πP.

Follows from the Brouwer fixed point theorem: for any


continuous function f : S → S, where S is a compact convex
set, there is some x such that f(x) = x.

11
Periodicity

The periodicity of a state i is defined as:


T = gcd{t > 0 : Pr(Xt = i | X0 = i) > 0}.

The state is aperiodic if it has periodicity T = 1.


A Markov chain is aperiodic if all states are aperiodic.
12
Periodicity

Claim
If a Markov chain is irreducible, and has at least one
self-loop, then it is aperiodic.

13
Fundamental Theorem

Theorem (The Fundamental Theorem of Markov Chains)


Let X0 , X1 , . . . be a Markov chain with a finite state space and
transition matrix P ∈ [0, 1]m×m . If the chain is both irreducible and
aperiodic,

1. There exists a unique stationary distribution π ∈ [0, 1]m with


π = πP.
2. For any states i, j, limt→∞ Pr[Xt = i|X0 = j] = π(i). I.e., for any
initial distribution q0 , limt→∞ qt = limt→∞ q0 Pt = π.
3. π(i) = E[min(t:X1t =i)|X0 =i] . I.e., π(i) is the inverse of the average
expected return time from state i back to i.

In the limit, the probability of being at any state i is independent of


the starting state.
14
Stationary Distribution Example 1

Shuffling Markov Chain: Given a pack of c cards. At each step draw


two random cards, swap them and repeat.

• What is the state space of this chain?


• What is the transition probability Pi,j ? How does it compare to
Pj,i ?
• This Markov chain is symmetric and thus its stationary
distribution is uniform, π(i) = c!1 .

Letting m = c! denote the size of the state space,


∑ ∑ 1 ∑ 1
πP:,i = π(j)Pj,i = π(j)Pi,j = Pi,j = = π(i).
m m
j j j

Once we have exhibited a stationary distribution, we know that it is


unique and that the chain converges to it in the limit!

15
Stationary Distribution Example 2

Random Walk on an Undirected Graph: Consider a random


walk on an undirected graph. If it is at node i at step t, then it
moves to any of i’s neighbors at step t + 1 with probability d1 .
i

• What is the state space of this chain?


• What is the transition probability Pi,j ?
• Is this chain aperiodic?
• If the graph is not bipartite, then there is at least one odd
cycle, making the chain aperiodic.

16
Stationary Distribution Example 2

Random Walk on an Undirected Graph: Consider a random


walk on an undirected graph. If it is at node i at step t, then it
moves to any of i’s neighbors at step t + 1 with probability d1 .
i

Claim: When the graph is not bipartite, the unique stationary


di
distribution of this Markov chain is given by π(i) = 2|E| .

∑ ∑ dj 1 ∑ 1 d
πP:,i = π(j)Pj,i = · = = i = π(i).
2|E| dj 2|E| 2|E|
j j j

I.e., the probability of being at a given node i is dependent


only on the node’s degree, not on the structure of the graph in
any other way.

17

You might also like