STAT 333 - Problem Set 6

Exercises marked with an asterisk ∗ are considered to be recommended exercises. The non-
asterisk exercises are intended to serve as extra practice exercises.

Section 3.3
Exercise 3.3.1.∗ In Example 3.2, convince yourself that by taking several higher powers of
P n,
0 1 2
 
0 0.47619 0.238095 0.285714
(n)
lim P = 1 0.47619 0.238095 0.285714 .
n→∞
2 0.47619 0.238095 0.285714
Verify that the conditions of the BLT hold true for this DTMC, and show that the limiting
probabilities obtained using the BLT agree with the above limiting TPM.

Exercise 3.3.2. For the DTMC in Example 3.10, show that there exists an infinite number
of stationary distributions.

(n)
Exercise 3.3.3. For the DTMC in Example 3.12, determine expressions for P1,j , j = 0, 1, 2,
and use them to show that
(n) 2 (n) (n) 1
lim P1,0 = , lim P1,1 = 0, and lim P1,2 = .
n→∞ 3 n→∞ n→∞ 3
Exercise 3.3.4. Suppose that the TPM of a two-state DTMC is given by

0 1
 
0 α 1−α
P = ,
1 1−β β

where 0 < α < 1 and 0 < β < 1.

(a) Assuming that α = β = p, show by mathematical induction that

0 1
"1 1 1 1
#
0 2 + 2 (2p − 1)n 2
− 2
(2p − 1)n
P (n) = .
1 12 − 21 (2p − 1)n 1
2
+ 1
2
(2p − 1)n

(b) In part (a), what happens to P (n) as n → ∞?

(c) Find the limiting probability distribution of this DTMC.

1
 (1) (n)
(d) Show that the first return time distribution to state 0 is given by f0,0 = α and f0,0 =
(1 − α)(1 − β)β n−2 , n = 2, 3, 4, . . . .

(e) Use part (d) to calculate m0 , the mean recurrent time for state 0, and then verify that
π0 = 1/m0 where π0 represents the long-run mean fraction of time the DTMC is in
state 0.

Let {Xi }∞
Exercise 3.3.5. P i=1 be an iid sequence of BERN(p) rvs, where p ∈ (0, 1). For
+ n
n ∈ Z , let Yn = i=1 Xi .

(a) Justify why the stochastic process {Yn , n ∈ Z+ } is a DTMC.

(b) Determine the TPM of this DTMC.

(c) Specify the communication classes of this DTMC. For each class of this DTMC, deter-
mine whether it is recurrent or transient. Justify your response.

(d) Does a limiting distribution for this DTMC exist? If so, what is the limiting distribu-
tion? Justify your response.

(e) For each k ∈ Z+ , let Tk = min{n ∈ Z+ : Yn = k}. Determine the pmf of the rv Tk .

Exercise 3.3.6.∗ Consider a DTMC {Xn , n ∈ N} with TPM

0 1 2 3
 
0 1 − p0 p0 0 0
1 1 − p1 0 p1 0
P =  .
2 1 − p2 0 0 p2 
3 1 − p3 0 0 p3

(a) Suppose that p0 = p1 = p2 = 1 but 0 < p3 < 1. Does the DTMC have a limiting
probability distribution? If so, find it.

(b) Suppose now that (p0 , p1 , p2 , p3 ) = (1/2, 1/3, 1/3, 1/2). Does the DTMC have a limiting
probability distribution? If so, find it.

(c) Suppose once again that (p0 , p1 , p2 , p3 ) = (1/2, 1/3, 1/3, 1/2), and that the initial con-
ditions are governed by the stationary distribution of the DTMC. Assuming that
n = 2, 3, 4, . . . , calculate P (Xn−2 = 1|Xn = 0).

Exercise 3.3.7.∗ Consider a DTMC with state space S = {0, 1, 2, 3, 4}. Let P0,4 = 1, and
suppose that when the chain is in state i, i > 0, the next state to be visited is equally likely
to be any of the states 0, 1, . . . , i − 1. Find the limiting probabilities of this DTMC.

2
Exercise 3.3.8.∗ An individual begins his day each morning by wanting to buy a latte at
the local coffee shop. As he prepares to go out, he is equally likely to leave either from the
front or back door of his house. The individual owns m pairs of shoes which he removes at
whichever door he happens to enter back through. In a similar fashion, he is equally likely
to come back either through the front or back door. However, if there are no shoes at the
door from which he decides to leave from, he simply strolls over barefoot to his neighbour’s
house and enjoys a complimentary tea with the neighbour on his outdoor patio.

(a) Describe how you might model this scenario as a DTMC and specify its associated
TPM.

(b) What is the long-run proportion of the time the individual enjoys a tea with his neigh-
bour?

Exercise 3.3.9.∗ Consider a dam with a maximum capacity of h ∈ Z+ units of water. Let
Xn represent the amount of water in the dam at the end of the nth day. In addition, suppose
that the following facts are known:

ˆ The daily inputs to the dam are independent of each other, and on a given day,

P (input is j units) = 2−(j+1) , j = 0, 1, 2, . . . .

ˆ Any overflow (when the total amount of water exceeds the maximum capacity of h
units) is regarded as lost.

ˆ Provided the dam is not empty, one unit of water is released at the end of the day.

ˆ The value of Xn is the amount of water in the dam following the one unit release of
water at the end of the day.

As a result, {Xn , n ∈ Z+ } is a discrete-time stochastic process with state space S =

(a) Justify why the stochastic process {Xn , n ∈ Z+ } is a DTMC.

(b) Prove that the DTMC is irreducible and aperiodic.

(c) Assuming that h = 5, determine the associated TPM and solve for the limiting prob-
abilities of the DTMC.