University of Pretoria

WST312 Stochastic Processes

Past Questions Memo

Chapter 4
Question 1 Suppose {X t : t  0} is a Poisson process that models the cumulative number
of arrivals of insurance claims. The average number of claims per hour is 0.83.
a) Write down the transition rates for this process.

b) State the distribution of each X t .

c) Calculate P[ X 3  2] and P[ X 5  X 2  3] . (Hint: If Y ~ exp( ) , then

e   y
f Y ( y)  .)
y!

d) Define the first waiting time W0 , and state it’s distribution.

e) Show that P[W0  s  t | W0  s]  P[W0  t ]. (Hint: Use X t  s and X s .)

1
f) Given that the process is in state 0 after four and a half hours, calculate the probability that
the remaining time until the first jump is at least two hours. (Hint: If Y ~ exp( ) , then
FY ( y)  1  e y .)

Question 2 A particular machine is in constant use (i.e. 24 hours a day). Regardless of how
long it has been since the last repair, it tends to break down once a day and on average it takes
the repairman 6 hours to fix but can still be used for it’s purpose. We can model the machine’s
status as a time-homogeneous Markov jump process { X t : t  0} , where t is in days, with two
states, namely, 0: being repaired and 1: working.

(i) State the two main assumptions that we make in applying the model and discuss briefly how
you could test if each of them holds. (Hint: How are the waiting times distributed?

(ii) What are the distributions of the waiting times?

(iii) What is the transition rate matrix A for this process?

(iv) State Kolmogorov’s forward equation for the probability P00 (t ) , that the machine

2
 is being repaired at time t , given that it is being repaired at time 0.

1 4  5t
(v) Show that P00 (t )   e satisfies the equation obtained in (iv).
5 5

Question 3 Consider a Markov Jump process sickness and death model which has four states
S  H (healthy), S (sick ),T ( terminally ill), D(dead) . From a healthy state transitions are possible to
states S and D , each at a rate of 0.05 per year. A sick person recovers his health at rate 1.0 per year;
other possible transitions are to D and T , each with rate 0.1 per year. Only one transition is possible
from the terminally ill state, and that is to state D with transition rate of 0.4 per year.
a) Set up the transition rate matrix for this process.

b) Let P(t )  P(0, t ) . What property does this assumption imply holds?

c) State the Kolmogorov forward equation satisfied by P(t ) and also the associated equation for
PHD (t ) , making sure you use your result in (a).

d) Calculate the probability of being healthy for at least 10 uninterrupted years given that you are
healthy now.

3
e) Let d j denote the probability that a life which is currently in state j will never suffer a terminal
1 1
illness. By considering the first transition from state H , show that d H   d S and deduce similarly
2 2
1 5
that d S   d H . Hence, find d H and d S . (Hint: Recall how we obtain the transition
12 6
 ij
probabilities from the transition rate matrix i.e. Pij (t )  ).
  ii

f) Find the expected duration of a terminal illness, starting from the moment of the first transition
into state T . Use the result of (e) to deduce the expectation of the future time spent terminally ill
by an individual who is currently healthy.

Question 4 Two types of buses arrive at a certain bus stop: the number 45 and the ‘direct X45’.
The buses arrive in accordance with independent Poisson processes with rates 5 per hour and 1 per
hour respectively.
a) Prove that the resulting process is a Poisson process with parameter the sum of the individual
parameters.
We have that Z 0  X 0  Y0  0  0  0 .
We also have that X t  X s is independent of { X u : 0  u  s} and of {Yu : 0  u  s} since the
processes are independent. Similarly Yt  Ys is independent of {Yu : 0  u  s} and of { X u : 0  u  s} .
Hence Z t  Z s  X t  X s  Yt  Ys is independent { X u : 0  u  s} and {Yu : 0  u  s} and therefore
independent of {Z u  X u  Yu : 0  u  s} i.e. the Z t 's have independent increments.
Furthermore Pi ,i u (s, t )  P[Z t  Z s  u]  P[ X t  X s  Yt  Ys  u]. But X t  X s and Yt  Ys are
independent Poisson random variables with parameters 1 (t  s) and 2 (t  s) respectively and

4
therefore Z t  Z s is a Poisson random variable with parameter 1 (t  s)  2 (t  s)  (1  2 )(t  s)
.
b) What is the resulting parameter?
5+1=6
c) Calculate the probability of the following events, if you initially have no preference between buses:

(i) exactly six 45’s and no X45’s arriving in the next hour,
Let U represent the 45 buses and V the X45 buses. Then since the two are
e 5 56 1
independent, P[U  6]P[V  0]  e  0.0538 .
6!
(ii) having to wait more than 15 minutes for a bus, if you’ve just missed one.
15min = ¼ hour thus P[T  1 / 4]  e 6 / 4  0.2231
(iii) you get on an X45, when only the 45 goes to your destination.
One in 6 buses is an X45 therefore 1/6.

  3 3 0
Question 5 Consider a Markov Jump process with A   1  1 0 . If the distribution amongst
 0 0 0
1 1 1
the states is   at time 0, what is the distribution after 2 years and 3 months?
4 2 4
We have P31 (t )  0, P32 (t )  0 and P33 (t )  1 so that the rows sum to 1 in P(t). (1/2 mark)
Then since P[W0  u | X 0  i]  e iiu (1/2 mark), P22 (t )  e t and then P21 (t )  1  e t .(1/2
mark)
Similarly, P11 (t )  e 3t and then P12 (t )  1  e 3t . (1/2 mark)
 e 3t 1  e 3t 0
 
So P(t )  1  e t e t 0 .
 0 0 1

3
The distribution at time 2  2.25 is given by
12
1 1
P(2.25)  0.05299 0.69701 0.25 (1 mark)
1
4 4 
 2

Question 6 An internet service provider (ISP) is modeling the capacity requirements for its
network. It assumes that if a customer is not currently connected to the internet (‘offline’) the
probability of connecting in the short time interval (t , t  t ) is 0.2t  o(t ) . If the customer is
connected to the internet (‘online’) then it assumes the probability of disconnecting in the time
interval is given by 0.8t  o(t ) . The probabilities that the customer has been online and offline for
a period t are PON (t ) and POFF (t ) respectively.

5
a) Explain why the status of an individual customer can be considered as a Markov Jump process.


b) Derive the Kolmogorov Forward equations for POFF (t ) .
t

c) (i) For the general case where Pij (t )  Pij (s, s  t ) , state and prove the Chapman-Kolmogorov
equations.

(ii) If we take the derivative with respect to the first argument of Pij ( s, s  t ) in the Chapman-
Kolmogorov equations, state and prove the result that then follows.

6
Question 7 Consider a time-homogeneous Markov Jump process with states 1 and 2 and with
   
transition rate matrix A    . Use (c) to find an approximation for the transition
    
probability matrix for this process by assuming the terms in A k for k  3 are negligible.
(t  s ) k     (t  s )1     (t  s ) 2    
k 2

(t  s ) k k 
P ( s, t )  e (t  s ) A
 A  I 
k 0 k! k 0 k!   
  1!      2!     
   
2
 2  
 2   
and since    we have
       
   2  2

(t  s)1     (t  s) 2  2    2   
P ( s, t )  I    
1!      2!     2    2 
 1   (t  s )  1 (t  s) 2 (2  ) 1   (t  s )  1 (t  s) 2 (2  )
 2 2 
1   (t  s )  2 (t  s) (   ) 1   (t  s )  2 (t  s) (   ) 
1 1
2 2 2 2

Question 8 Consider a Markov Jump process {Yt , t  0} with transition probability matrix
0.6  0.4e 5t 0.4  0.4e 5t 
P(t )   5t  and state space S  {1,2} .
0.6  0.6e 0.4  0.6e 5t 
a) Verify that P(t ) is a valid stochastic matrix.
0.6  0.4e 5t  0.4  0.4e 5t  1 and 0.6  0.6e 5t  0.4  0.6e 5t  1
b) Is this process time-homogeneous? Explain?
Yes, since P(t ) depends only on the distance between times i.e. t and not the actual times i.e. s to
s  t for example.
c) Compute the probability P[Y2.4  1, Y3.8  2, Y4.2  2 | Y0  1] . State which result you make use of.
2 .4 3.8  2.4 1.4 4 .2  3 . 8  0 . 4
The transition path is 1 1  2  2 so the probability is equal to
  
0.6  0.4e 5( 2.4) 0.4  0.4e 5(1.4) 0.4  0.6e 5( 0.4)  0.115383 
P[ X t1  i1 , X t2  i2 ,..., X tn  in X t0  i0 ]  pi(0t,0i1,t1 ) pi(1t,1i,2t2 ) pi(2t,2i3,t3 ) ... pi(ntn1,1i,ntn ) .

d) Find the transition rate matrix for this process.

d  0.4(5)e 5t  0.4(5)e 5t   2 2 
A  P(t )      
dt t 0  0.6(5)e
5t
0.6(5)e 5t  t 0  3  3
e) By finding four differential equations for Pij (t ) , find the limiting distribution for this process.

By the Kolmogorov forward equations P(t )  P(t ) A we have four differential equations:
t

7
 
P11 (t )  2 P11 (t )  3P12 (t )
t

P12 (t )  2 P11 (t )  3P12 (t )
t

P21 (t )  2 P21 (t )  3P22 (t )
t

P22 (t )  2 P21 (t )  3P22 (t )
t
 Pij ( s, t  h)  Pij ( s, t ) Pij ( s, t  h)  Pij ( s, t )
Now lim Pij ( s, t )  lim lim  lim lim  lim 0  0
t  t t  h 0 h h 0 t  h h 0

provided the interchange of limits is in order.

Let lim Pin ( s, t )  p n .
t 

So we get 0  2 p1  3 p 2 as the only unique equation. Using the fact that p1  p 2  1 we get
2 5 3 2
that p1  p1  p1  1 thus p1  and p 2  .
3 3 5 5

Question 9 A study is undertaken on marriage patterns of women in a certain country. A sample

of women, and the start and end of all their marriage(s) (they are not permitted to be married to more
than one man at a time) is available. The reason for the end of a marriage - death or divorce - is also
available. The study is aimed at estimating the rate of first marriage for all women.
a) Draw a diagram of the process with 4 states ‘Never married’, ‘Married’, ‘Widowed’ and
‘Divorced’. Indicate between the states the transition rate directions.

b) Explain why this process should be modeled as a Markov jump process and not as a Markov chain.

c) Using suitable symbols for the various transition rates, set up the time-inhomogeneous transition
rate matrix.

d) Derive the Kolmogorov forward equation for the probability of leaving the ‘Never married’ state.

8
Question 10 Consider the following transition rate matrix for a Markov jump process:
0 0 0
A  1  1 0  .

1 1  2
a) Is this process time-homogeneous? Explain.

b) Prove that P[Wt  u | X t  i]  P[W0  u | X 0  i]  e iiu where Wt is the length of time that the
process remains in the state being occupied at time t .

c) If the distribution at time 2.2 is 13 1

3
1
3
 , find the distribution at time 2.5. You may assume, for
d
simplicity, that the solution of the differential equation f (t )   f (t )  e  2t is f (t )  1  e 2t  e 3t .
dt

9
Question 11 Suppose an insurance company models its claim arrivals as a Poisson process with
parameter  .
a) Show that if there is exactly one claim in the interval [0, t1 ] , the time of the claim is uniformly
distributed on [0, t1 ] . Hint: Show that the (conditional) cumulative distribution function for this time is
a uniform cumulative distribution function and make use of the properties of the Poisson process.
xa
Recall that if X ~ U (a, b) then FX ( x)  .
ba

b) State the joint distribution of the times between claims.

Question 12 The probability of a dishwasher breaking down is assumed to follow an exponential

distribution with parameter  where
1 if the machine has never broken down
 10
   15 if the machine has broken down once before . Assume that at a breakdown
1
 2
if the machine has broken down two or more times before
the repair is done immediately and the machine returned to use, initially the machine has never broken
down, and a maximum of three breakdowns is allowed. Let P0i (t ) be the probability of i breakdowns
by time t and let the state space for the process be S  {0,1,2,3} .
a) Set up a transition rate matrix for this process.

b) Derive differential equations for the probabilities P00 (t ), P01 (t ) and P02 (t ) .

10
  10
1
 10
1
 15 t
c) Show that the solutions for the first two equations in (c) are P00 (t )  e and P01 (t )  e e
t t
.

d) If in addition the solution P02 (t ) 

3

4  101 t 1t 1t

e  3e 5  2e 4 is known, calculate the average number of
claims over the first two time units.

 0.1t 0.1t 
Question 13 Consider a Markov jump process with transition rate matrix A(t )   
 0.05t  0.05t 
with state space S  {1,2} . Make use of the integrated form of the Kolmogorov equations to derive a
formula for the probability P12 ( s, t ).

11
Question 14 Consider the following transition rate matrix for a Markov jump process:
0 0 0
A  1  1 0  . The state space of the process is S  {1, 2,3} .

1 1  2

a) Explain why the process is time-homogenous.

b) Explain why Pii (t )  Pii C (t ) for i  1, 2,3.

c) Prove that P[Wt  u | X t  i]  P[W0  u | X 0  i]  e iiu where Wt is the length of time that the
process remains in the state being occupied at time t .

d) Find pii (t ) for t  1, 2 and 3 , with explanation.

12
e) By using the Kolmogorov forward equations, or otherwise, find the matrix P(t ) . You may also
d
assume that the solution of the differential equation f (t )   f (t )  e  2t is f (t )  1  e 2t  e 3t .
dt

f) If the distribution at time t  2.2 is 13 1

3
1
3
 , find the distribution at time t  2.5 .

Question 15 Consider a Markov jump process state space S  {1, 2,3} , transition rate matrix
 0.5t 0.125t 0.375t  0.7 0.1 0.2 
A(t )  0.125t 0.125t 0  and transition probability matrix P (t )   0.5 0.5 0  .

 0 0 0   0 0 1 

Use the integrated Kolmogorov equations to calculate P13 (0,5) .

13
Question 16 Consider a stochastic process with state space S  {1,2,3} and transition rate matrix
 5 3 2
A   0  4 4 .
 0 0 0

a) Using the definition of the Markov property discuss whether this process has the
Markov property.

b) Is this process time-homogeneous? Explain.

c) Provide a formula for P13 (2.3) if it is known the process is in state 2 after 1.1 time units.

d) Discuss the implication of the result Pij (h)  PijC (h)  o(h) .

Question 17 Consider the Poisson process with parameter  .

a) Provide distributions for the following:
(i) Nt ~
(ii) Nt  N s ~
(iii) Ti ~
n
(iv) T   Ti ~
i 1

14
b) Let X t be the number of events of some type that takes place in the interval [0, t ] and let
the state space be S  {0,1,2,...} . If the following assumptions hold:
(1.) X 0  0
(2.) The number of events in disjoint intervals are independent random variables.

(3.) The probability of an event taking place in the interval (t , t  h] only depends on
the length of the interval.
(4.) The probability that one event occurs in the interval (t , t  h] is h  o(h) .
(5.) The probability that no events occur in the interval (t , t  h] is 1  h  o(h) .

(6.) The probability that more than one event takes place in the interval (t , t  h] is o(h)
(i) Derive a differential equation for Pij (t ) where j  i .

(ii) Provide a solution for the differential equation derived in (i) and show it is a solution.

c) Suppose that { X t : t  [0, )} and {Yt : t  [0, )} are two independent Poisson processes
with parameters 1 and 2 respectively. Let Z t  X t  Yt . Prove that {Z t : t  [0, )} is a
Poisson process with parameter 1  2 .

15
 Question 18 a) State and prove the Kolmogorov Backward equation for a time-homogeneous
Markov Jump process.

b) State and prove a property of the rows of a transition rate matrix of a time-homogeneous
Markov Jump process.

16
Question 19 Consider a physical phenomenon in which there is no increase in the population
size once the process started. Given an initial population size N  0 individuals "die" at a
certain rate eventually reducing the size to zero. Let this be a Markov Jump process { X t : t  0}
and state space S  {0,1,2,3,..., N } .
a) Provide the 6 assumptions for this process.

b) If the rates i for the process are all identical, derive differential equation(s) for
PNn (t ),0  n  N .

17
   5 5 0
Question 20 Consider a Markov Jump process with transition rate matrix A   6  6 0
 0 0 0
and state space S  {1,2,3} . Derive the limiting probabilities for this process.

18
Question 21 Consider a time-homogeneous Markov Jump process with states 1 and 2 and with
   
transition rate matrix A    . Use the matrix exponential solution of the Kolmogorov
 0 0
Forward equation to find an approximation for the transition probability matrix for this process
by assuming the terms in A k for k  3 are negligible.

Question 22 Consider a time-homogeneous Markov Jump process.

a) Define the transition rate of going from state i to state j in the vicinity of time s .

 6 0 6 
b) Suppose the transition rate matrix for this process is A   0 0 0  . Derive the limiting
 4 0  4
probabilities for this process.

19
Question 23 a) Define a Poisson process with parameter  .

b) Derive a differential equation for Pii (t ) for i  0 for a Poisson process with parameter  .

20
c) Provide a solution for the differential equation derived in (b) and show that it is a solution.

d) Complete the following for the Yule process:

(i) Pii (t , t  h)  _________________________________
(ii) Pi ,i 1 (t , t  h)  _______________________________
(iii) For j  i  1, Pij (t , t  h)  ______________________
(iv) For j  i, Pij (t , t  h)  _________________________
Explain how this differs from the assumptions of the Poisson process and why this is useful.

e) If for the Yule process P1n (t )  e t 1  e t  , compare P11 (t ) for the Yule with P00 (t ) for
n 1

the Poisson process respectively. Also compare, lim P11 (t ) and lim P00 (t ) for the two processes,
t  t 

providing explanations for the answers.

Question 24 Let {X t : t  0} be a Markov jump process with transition rate matrix

A(t )  [ ij (t )] .
P[ X s  w  h  j , w  Ws  w  h | X s  i ]
a) Using the fact that lim  f X  ,W | X ( j , w | i ) derive the
h 0 h s s s

joint probability mass/density function of X s and Ws .

sw

b) Show that f Ws | X s ( w | i)   ii ( s  w)e s

 ii ( u ) du
.
 ij (s  w)
c) Now show that P[ X s  j | X s  i,Ws  w]  .
  ii ( s  w)
d) Explain why X s and Ws are not independent? Explain.

21
 e) If the process is time-homogeneous, show that X s and Ws are independent.
 6 0 6 
f) Consider a Markov Jump process with transition rate matrix A   0 0 0  . Use the
 4 0  4
results proven above to determine the transition probability matrix P(t ) .

Question 24
a) Complete the following table with the correct notation for the two cases time-homogeneous
and time-inhomogeneous:
Transition probability Transition rate Initial condition
matrix (with the matrix
element notation as (with the element
well) notation as well)
Time-homogeneous
Case

Time-
Inhomogeneous
Case

C
b) (i) Provide o(h) relations for Pij (h) and Pij (h) for a general Markov process.
(ii) Discuss the implication of the result in b(i).
(iii) Discuss why P ii (t )  Pii (t )t for the Yule process.
C

22
Question 24 (a) Provide random variables notation for x and y in the picture below for both
the time-homogeneous and time-inhomogeneous cases.

 ij (s  w)
(b) Let { X t : t  0} be a Markov jump process. Prove that P[ X s  j | X s  i,Ws  w]  .
  ii ( s  w)
You may use without proof that P[ A  B | C ]  P[ B | C ]P[ A | B  C ] .

23
(c) How does the result in (b) change if the process is time-homogeneous?

Question 25 Consider the following transition rate matrix for a Markov jump process with
S  {1,2,3} :
 1 1 0
A   0  2 2 .
 0 0 0
(a) Explain why Pii (t )  PiiC (t )i for this process.

(b) Is this process time-homogeneous? Explain.

(c) Determine with reasoning the transition probability matrix P(t ) .

24
(d) Determine the limiting probabilities for this process using two different methods.

Question 26 Consider the following transition rate matrix for a Markov jump process with
S  {1,2,3} :
 1 1 0
A   0  2 2 .
 0 0 0
Use (a) to derive P23 (1.6) and P23 (10.5) . Comment on the two answers in relation to the
progression of the process.

25
Question 27 a) Define the transition rate  ii .

b) Complete the following table for a Markov jump process:

c) Provide an o(h) expression for PiiC (t , t  h) and compare to that of Pii (t , t  h) .

Question 28 Consider a Markov jump process with transition rate matrix

 2 2 0 0 0
 1 1 0 0 0 

A 0 0 0 0 0  and state space S  {1,2,3} . Derive the limiting probabilities for
 
0 0 1 1 0 
 0 0 3 0  3
this process.

26
Question 29 Consider the following transition rate matrix for a Markov jump process with
state space S  {1,2,3} :
0 0 0
A  1  1 0 

0 2  2
a) Explain why Pii (t )  PiiC (t ) t .

b) Use (i) and (ii to get a formula for Pii (t ) for i  1,2,3 .
(i) The density function of Wt is given by   iie iiu .
(ii) Let X Wt be the state to which the process jumps when the first jump occurs. Then t
Wt and X W are independent random variables and the instantaneous transition
t

 ij
probability is given by P[ X Wt  j | X t  i]  for i  j , if you are given that
  ii
P[ X t u h  j, u  Wt  u  h | X t  i]  e u Pij (h) .
ii

27
c) Use the integrated form of the Kolmogorov equation to derive a formula for P32 (t ) .

Question 30 Consider a birth and death process. You are given

P00 (t , t  h)  1  0 h  o(h) ,
P01 (t , t  h)  0 h  o(h) ;
and for n  1 Pn,n1 (t , t  h)  n h  o(h), Pn,n1 (t , t  h)  n h  o(h) , and Pnj (t , t  h)  o(h)
for j  n  1, n or n  1 .
a) Derive the remaining o(h) expression for Pn,n (t , t  h) .

28
 b) Set up the transition rate matrix for this process and further derive a differential equation for
Pin (t ), n  0 if n  n and  n  n .

c) Determine, with reasoning, lim P1n (t ) using the differential equation in (b).
t 

29
 Question 31 Consider the following transition rate matrix for a Markov jump process with
state space S  {1,2,3} :
 1 1 0
A   0  2 2
 0 0 0
a) Explain why Pii (t )  PiiC (t ) t .

b) Find a formula for Pii (t ) for i  1,2,3 .

c) Use the integrated form of the Kolmogorov equation to derive a formula for P12 (t ) .

30