Scribd
Upload
32 views2 pages

Week 2 Problem Set: DTMCS: K+1 K+M 1 K

This document contains a problem set on discrete-time Markov chains (DTMCs) with 5 questions: 1. Prove that the probability of being in a state at a future time depends only on the most recent state, not on earlier states. 2. Check if 3 DTMCs with given transition matrices are irreducible or aperiodic. 3. Note that the stationary distribution of an irreducible, aperiodic DTMC with a doubly stochastic transition matrix will be uniform. 4. Calculate probabilities related to a Markovian rain process based on transition probabilities between rainy and dry days. 5. Determine the limiting probability of heads for a process of repeatedly flipping biased coins until tails occurs.

Uploaded by

Sruthi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
32 views2 pages

Week 2 Problem Set: DTMCS: K+1 K+M 1 K

This document contains a problem set on discrete-time Markov chains (DTMCs) with 5 questions: 1. Prove that the probability of being in a state at a future time depends only on the most recent state, not on earlier states. 2. Check if 3 DTMCs with given transition matrices are irreducible or aperiodic. 3. Note that the stationary distribution of an irreducible, aperiodic DTMC with a doubly stochastic transition matrix will be uniform. 4. Calculate probabilities related to a Markovian rain process based on transition probabilities between rainy and dry days. 5. Determine the limiting probability of heads for a process of repeatedly flipping biased coins until tails occurs.

Uploaded by

Sruthi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

EE 621 Spring 2021

Week 2 problem set: DTMCs

1 DTMC definition

Consider the DTMC {Xn }n≥0 over countable state space S. Prove that for any i5 , i3 , i1 ∈ S,

P (X5 = i5 | X3 = i3 , X1 = i1 ) = P (X5 = i5 |X3 = i3 ) .

Note: From the above proof, you should be able to convince yourself that the following more general result
holds. Given 0 ≤ n1 < n2 < · · · < nk < nk+1 < · · · < nk+m , and i1 , i2 , · · · , ik+m ∈ S, we have

P Xnk+1 = ik+1 , · · · , Xnk+m = ik+m | Xn1 = i1 , · · · , Xnk = ik

= P Xnk+1 = ik+1 , · · · , Xnk+m = ik+m | Xnk = ik .

2 Reviewing the basics

Consider DTMCs with the following transition probability matrices. In each case, check if the DTMC is irre-
ducible. If irreducible, also check if the chain is aperiodic.

 
1 1 1
3 3 3
1.  0 0 1 
1 0 0
 
0 1 0
2.  0 0 1 
1 0 0
 1 2

3 0 3
3.  12 12 0 
0 0 1

3 Doubly stochastic transition matrix

A doubly stochastic matrix is one in which each row and column sum equals 1. Consider a finite, irreducible,
aperiodic DTMC with a doubly stochastic probability transition matrix. What can you say about the stationary
distribution of this DTMC?

1
4 Markovian rains1

Suppose the probability of rain today is 0.6 if it rained yesterday, but only 0.2 if it did not, where these proba-
bilities do not depend on the weather on previous days.

1. Given rain fell today, what is the probability that it will rain day-after tomorrow?
2. What is the average duration (number of days) of a rainy period?
3. What is the limiting probability of a rainy day?

5 Markovian coin-flips

You have two biased coins. Coin 1, when flipped comes up heads with probability 0.6, and Coin 2, when flipped
comes up heads with probability 0.5.
A coin is flipped repeatedly until it comes up tails, at which point we set that coin aside, and start flipping the
other coin. This process is repeated indefinitely.
Let p(n) denote the probability that the nth flip comes up heads. What is the limiting value of p(n) as n → ∞?

1
This is Problem 3.13 in “Stochastic Modeling and the Theory of Queues” by R. W. Wolff

You might also like