Scribd
Upload
75 views4 pages

Stochastic Processes and Time Series Markov Chains - III: 1 Markov Chain: Classification of States

The document discusses Markov chains and classifies states as either recurrent or transient. Recurrent states will be visited infinitely often with probability 1, while transient states will only be visited a finite number of times. The expected number of visits to a recurrent state is infinite, while the expected number of visits to a transient state is finite.

Uploaded by

Biju Angalees
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
75 views4 pages

Stochastic Processes and Time Series Markov Chains - III: 1 Markov Chain: Classification of States

The document discusses Markov chains and classifies states as either recurrent or transient. Recurrent states will be visited infinitely often with probability 1, while transient states will only be visited a finite number of times. The expected number of visits to a recurrent state is infinite, while the expected number of visits to a transient state is finite.

Uploaded by

Biju Angalees
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Stochastic Processes and Time Series

Module 3
Markov Chains - III
Dr. Alok Goswami, Professor, Indian Statistical Institute, Kolkata

1 Markov Chain: Classification of States


Consider a MC {Xn }n≥0 on state space S with transition matrix P = (( pxy )). We classify the
states into two types − they will play very different roles in the long-run behaviour of the chain.
Towards this end, we start with some definitions:

1.1 First Hitting Time and Number of Visits of a State:


For x ∈ S, define:
Tx = min{n ≥ 1 : Xn = x} and Nx = #{n ≥ 1 : Xn = x}
[In the definition of Tx , follow the convention that min ∅ = ∞]

• Tx = time of first visit (not counting time 0) to state x,


Nx = total number of visits (again not counting time 0) to x.

• Both Tx and Nx are random variables, with Tx taking values in {1, 2, . . . , ∞} and Nx taking
values in {0, 1, 2, . . . , ∞}.

• Events {Tx = ∞} and {Nx = 0} are same events; by taking complements, {Tx < ∞} and
{Nx ≥ 1} are same events.

For x, y ∈ S (not necessarily distinct), denote:


ρxy = Px (Ty < ∞)
ρxy represents the probability that the chain ever visits the state y (at some time ≥ 1), given that
it starts from state x.
(n) P (n)
• If we denote ρxy = Px (Ty = n), n ≥ 1, then ρxy = n ρxy . (The events {Ty = n}, for
different n, are disjoint!)

• The event {Ty = n} implies (but may not be implied by) the event {Xn = y}; so, one has
(n) (n)
ρxy ≤ pxy

1
1.2 Recurrent and Transient States: Definition
A state x ∈ S is called “recurrent” if ρxx = 1 and is called “transient” if ρxx < 1.
Thus a state x is recurrent if the chain starting from x is sure to return to x, while x is transient
if, for the chain starting at x, there is a positive probability of never returning to x.

• If a state x is recurrent, then the definition says that a chain given to start at x is sure to
return to x at least once.

• However, it turns out (and we will prove this) that something stronger is actually true.

• If x is a recurrent state, then a chain starting at x makes an infinite number of returns


to x, with probability one!!

• Seems surprising? Think about it carefully and you would see that this emerges as a natural
consequence of Markov property (carefully used!)

We prove a basic result which would not only give the above result, but lead to a clearer picture
of the difference between recurrent and transient states in the long-term evolution of the chain.

Proposition: Let x, y ∈ S. Then for any m ≥ 1, Px (Ny ≥ m) = ρxy · ρm−1 yy


Trivially for m = 1: Px (Ny ≥ 1) = Px (Ty < ∞) = ρxy .
Heuristic for m ≥ 2: Consider, for example, m = 2.
{Ny ≥ 2}: there is a 1st visit to y, followed by at least 1 more visit. The probability of there being
a first visit to y, starting from x is ρxy , and, conditioning on the history upto the time of the first
visit to y, the probability of there being at least one more visit to y should be (markov property?)
same as the probability for a chain starting at y to make at least one visit to y, which is ρyy .
Why is this NOT a proof? The time of the first visit to y is a random time, while in our definition
of markov property, we can only condition on history upto a non-random time n.
The actual proof takes care of this by decomposing according to the time when the first visit to y
occurs.
Proof: As seen already, there is nothing to prove for m = 1.
Consider m = 2: The event {Ny ≥ 2} means a 1st visit to y, followed by at least 1 more visit.
Decomposing according to the time when the first visit to y happens, we can write
∞
[
{Ny ≥ 2} = {Ty = n, Xn+k = y for some k ≥ 1}
n=1
Since {Ty = n}, n = 1, 2, . . . are disjoint events,
X∞
Px (Ny ≥ 2) = Px (Ty = n, Xn+k = y for some k ≥ 1)
n=1
Recaling Px (A) = P(A | X0 = x) and applying property (CP3), we get, for each n ≥ 1,
Px (Ty = n, Xn+k = y for some k ≥ 1) = P(Ty = n | X0 = x)
×P(Xn+k = y for some k ≥ 1 | X0 = x, Ty = n)
(n)
The 1st factor on the rhs equals ρxy , by definition.

Expanding the event {Ty = n}, the 2nd factor on the rhs equals
P(Xn+k = y for some k ≥ 1 | X0 = x, Xj 6= y, 1 ≤ j ≤ n − 1, Xn = y), which, by the assumed Markov

2
property, equals
P(Xk = y for some k ≥ 1 | X0 = y) = Py (Ny ≥ 1) = ρyy
∞
P (n)
Thus, Px (Ny ≥ 2) = ρxy · ρyy = ρxy ρyy , as was to be proved. Proof for general m can be
n=1
completed by induction (Exercise!).

1.3 Interesting Consequences:


• y recurrent: For any x, Px (Ny ≥ m) = ρxy ∀ m ≥ 1
{Ny ≥ m} ↓ {Ny = ∞} as m → ∞ ⇒ Px (Ny = ∞) = ρxy
In particular, Py (Ny = ∞) = 1 (as claimed earlier!)

• y transient: Since ρyy < 1, Px (Ny ≥ m) = ρxy ρm−1


yy → 0.
Thus, for any x, Px (Ny = ∞) = 0;
Or equivalently, for any state x, Px (Ny < ∞) = 1

Main Takeaways:

• y recurrent: If the chain starts from y, it will return to y infinitely often with probability 1.
In general, if the chain starts from any state x, then the probability of y being visited infinitely
often is the same as the probability of at least one visit, namely, ρxy .

• y transient: No matter which state x the chain starts from, it will, with probability 1, visit y
only a finite number of times (the number of visits is random, of course!)

Next consider, Expected Number of Visits: Ex (Ny ) for x, y ∈ S.

y recurrent: Not much to do! As already seen, given that the chain starts at a state x, Ny takes
only two values, 0 and ∞, with probabiliities ρxy and 1 − ρxy respectively. It follows that
Ex (Ny ) = ∞ or 0, according as, ρxy > 0 or ρxy = 0;
In particular, Ey (Ny ) = ∞
y transient: We know, for any x, Px (Ny < ∞) = 1, that is, Ny is a random variable taking only
non-negative integer values.
∞
P
Recall: For non-negative integer-valued Z, E(Z) = P(Z ≥ m)
m=1
Using this and ρyy < 1, one gets that, for any x,
∞ ∞
X X ρxy
Ex (Ny ) = Px (Ny ≥ m) = ρxy ρm−1
yy = <∞
1 − ρyy
m=1 m=1
A different way of looking at Ex (Ny ): For y ∈ S, consider indicator random variables {ξn }n≥1 ,
defined as:
ξn = 1 or 0 according as Xn = y or Xn 6= y
P
Clearly, Ny = n≥1 ξn and so, for any x,
X X X
Ex (Ny ) = Ex (ξn ) = Px (ξn = 1) = p(n)
xy
n≥1 n≥1 n≥1
P
(ξn are non-negative, so interchange of Ex and n≥1 is justified! )

3
y transient: We already know Ex (Ny ) < ∞, for all x.
P (n)
By what we just proved, this is equivalent to pxy < ∞, for all x.
n≥1
(n)
In particular, pxy → 0 as n → ∞, for any x.

P (n)
y recurrent: We know Ey (Ny ) = ∞, which means pyy = ∞.
n≥1

We get a new characterization of recurrence/transience:

1.4 Theorem
P (n)
(a) A state y is recurrent or transient according as the infinite series pyy diverges or converges.
n≥1
P (n)
(b) If y is transient, then the infinite series pxy converges for all x and, in particular, for all x,
n≥1
(n)
lim pxy = 0.
n→∞

You might also like