Scribd
Upload
25 views3 pages

Topic10 DTMC LimitingDistribution

1. The document discusses limiting behavior in discrete-time Markov chains, including finding the n-step transition probability matrix and the Chapman-Kolmogorov equations. 2. It provides an example of a weather forecast Markov chain and shows how the transition probabilities converge to the same value as n increases. 3. For an irreducible and ergodic Markov chain, there is a unique limiting or stationary probability distribution that the process will be in each state after a large number of transitions, and this does not depend on the initial state.

Uploaded by

Jane Hanger
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
25 views3 pages

Topic10 DTMC LimitingDistribution

1. The document discusses limiting behavior in discrete-time Markov chains, including finding the n-step transition probability matrix and the Chapman-Kolmogorov equations. 2. It provides an example of a weather forecast Markov chain and shows how the transition probabilities converge to the same value as n increases. 3. For an irreducible and ergodic Markov chain, there is a unique limiting or stationary probability distribution that the process will be in each state after a large number of transitions, and this does not depend on the initial state.

Uploaded by

Jane Hanger
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

1

Probabilistic Operations Research (POR)


Topic # 10 Limiting Behavior in Discrete-Time Markov Chains
(Chapter 6.2 and 6.4 of Cassady and Nachlas textbook)

Finding the n-step Transition Probability
The one-step transition probability matrix (tpm) can be used to examine the dynamic behavior of
the system.
Consider the discrete-time Markov chain {() } with state space and tpm . Then,
the n-step transition probability from state to state for all and
is expressed as

()
(( ) () )
The collection of all the n-step transition probabilities is the n-step transition probability
matrix,
()
.

Chapman-Kolmogorov Equations
There can be several feasible ways to make an n-step transition and we can consider the
transition to an intermediate state from state when transitioning to state in n steps using the
Chapman-Kolmogorov equations shown below.


for all .

) (

) (



In general, the summation of products on the right hand side is a matrix multiplication, then

()

()

()

for all .
A useful special case is:
()

()
for all .
2

Limiting behavior
Consider the Weather Forecast example in Topic 8:
Suppose that whether or not it rains tomorrow depends only on todays weather condition. If it
rains today, it will rain tomorrow with probability . If it does not rain today, it will rain
tomorrow with probability .
The tpm is
[


]
When we find the n-step tpm for n=2, 4, 8, and 16:

()
[


]
()
[


]

()
[


]
()
[


]
Note that the transition probabilities

converge to the same value as n and is the same


for all .
There seems to be a limiting probability that the process will be in state after a large number of
transitions and this probability is independent of the initial state .

In order to determine the step-dependent probability distribution on the identity of the state
occupied at any point in time, we need to know the distribution on state occupancy at the
beginning of the analysis.
Initial occupancy probability:

)
The vector of initial occupancy probabilities:


n-step occupancy probability:

()
(

)
The vector of n-step occupancy probabilities:
()

()

()

Note that

()

()

for all n = 1, 2, and

()

()
for all n = 1, 2,


3

If a DTMC is irreducible and has a finite state space, the initial state becomes irrelevant as the
number of transitions increases. This steady-state behavior is represented by the limiting
(stationary) probabilities,

.
For an irreducible and ergodic (positive recurrent and aperiodic) Markov chain,

()
for all .
Equivalently,

()
for all .
The vector of limiting probabilities:



The vector is the unique non-negative solution to the set of linear equations


for all (or equivalently, ) and

You might also like