Stochastic Models

MA 310

Dr. Sudipta Das

Outline I

1 Renewal Process
Introduction
Elementary Renewal Theorem

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 2 / 32

Introduction I

Definition:-
Given a sequence of mutually independent nonnegative real
valued random variables, Xk , k ≥ 1, with the random variables
Xk , k ≥ 2, also being identically distributed,
this sequence of random variables is called the sequence of
life-times of the renewal process,
for k ≥ 1,
Xk
Zk = Xi
i=1

is the k th renewal instant; define Z0 = 0.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 3 / 32

Introduction II

For t ≥ 0,
M(t) = sup{k ≥ 0 : Zk ≤ t},
the number of renewals in [0, t], is called the renewal process,
and, for t ≥ 0,
m(t) = E(M(t))
is called the renewal function.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 4 / 32

Introduction III

Remarks:-
In general, Zk is a discrete parameter random process that takes
nonnegative real values.
Also, M(t) is a continuous time random process that takes values in
{0, 1, 2, 3, · · · }.
On the other hand m(t) is just a nonnegative, real valued,
non-decreasing, (deterministic) function of time.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 5 / 32

Introduction IV

The sample paths of M(t) are non-decreasing step functions that

are right continuous at the jumps.
It is possible to have multiple jumps at an instant; for example, it is
possible that Zk = Zk +1 in which case

M(Zk +1 ) = M(Zk ),

and
M(Zk +1 ) − M(Zk − ) = 2.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 6 / 32

Introduction V

Several interesting related processes can be defined, and some

relationships between the processes can be observed.
ZM(t) is the instant of the last renewal in the interval [0, t].
Notice M(t) is a random time (or random index) for the process Zk ,
and ZM(t) is the value of the process Zk at the random time M(t).
Hence, for a sample path ω, ZM(t) means ZM(t,ω) (ω).
By definition, ZM(t) ≤ t.
M(t) + 1 is the index of the first renewal after t.
Hence, ZM(t)+1 is the first renewal instant (strictly) after t, i.e., the first
renewal instant in (t, ∞).
We have ZM(t)+1 > t.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 7 / 32

Introduction VI

For t ≥ 0,
Y (t) = ZM(t)+1 − t
is the residual life-time at time t.
Thus Y (0) = X1 , and Y (t) decreases at 1 unit per unit time until Z1 ,
then Y (t) jumps up by X2 , and so on.
The sample paths of Y (t) are right continuous at the jump instants
Zk , k ≥ 1, and Y (Zk − ) = 0, k ≥ 1.
For t ≥ 0, define the age process U(t) by

U(t) = t − ZM(t) .

If M(t) = 0, since, by definition, Z0 = 0, we have U(t) = t.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 8 / 32

Introduction VII

Notice that, for all n ≥ 1, and t ≥ 0,

(Zn ≤ t) = (M(t) ≥ n),

i.e.,
the event that the nth renewal occurs at or before t is the same as
the event that there are at least n renewals in [0, t].

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 9 / 32

Introduction VIII

We now see how the renewal function, m(t), can be expressed in

terms of the life-time distributions.
Let A(·) be the distribution of X1 and F (·) that of {X2 , X3 , . . . , }. It
then follows that
P(Z1 ≤ t) = A(t),
Z t
P(Z2 ≤ t) = F (t − u)dA(u) = (A ∗ F )(t),
0

P(Zn ≤ t) = (A ∗ F ∗ F ∗ · · · ∗ F )(t) = (A ∗ F (n−1) )(t),

where, ∗ denotes the convolution of the distributions A and F .

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 10 / 32

Introduction IX

Thus we conclude that

P(M(t) ≥ n) = P(Zn ≤ t) = (A ∗ F (n−1) )(t).

Hence we can write the renewal function as follows

∞
X ∞
X
m(t) = E[M(t)] = P(M(t) ≥ n) = (A ∗ F (n−1) )(t).
n=1 n=1

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 11 / 32

Introduction X

Lemma:-
If E(Xj ) > 0, j ≥ 2, then m(t) < ∞ for all t.
Remark:-
Note that for the nonnegative random variables Xj , E(Xj ) > 0 is
equivalent to the statement P(Xj > 0) > 0, or Fj (0+ ) < 1.
Observe that this result implies that, under the hypotheses,
M(t) < ∞ with probability 1.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 12 / 32

Introduction XI

Proof:-
We first observe that, for any n ≥ 1, and 0 ≤ m ≤ n,
Z t
F (n) (t) = F (n−m) (t − u)dF (m) (u)
0
Z t
≤ F (n−m) (t) dF (m) (u)
0

≤ F (n−m) (t)F (m)

(t)

Thus,
F (nr +k ) (t) ≤ F ((n−1)r +k ) (t)F (r ) (t)
≤ (F (r ) (t))n F (k ) (t)

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 13 / 32

Introduction XII

Hence, for any r ≥ 1,

∞
X
m(t) = (A ∗ F (n−1) )(t)
n=1
∞
X
≤ F (n) (t)
n=0
r −1
∞ X
X
= F (mr +k ) (t)
m=0 k =0
X∞
(mr )
≤ rF (t)
m=0
X∞
≤ r (F (r ) (t))m
m=0

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 14 / 32

Introduction XIII

Now, for each t choose r such that F (r ) (t) < 1; then we see, from
the last expression, that m(t) < ∞.
Such a choice of t is possible by virtue of the hypothesis that
E(X2 ) > 0.
For then there exists  > 0 such that F () < 1.
Now, for every

n, (1 − F (n) (n)) ≥ (1 − F ())n > 0;

i.e., F (n) (n) < 1. It follows that

t t
F d  e (d e) < 1.

Hence,
t
F d  e (t) < 1.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 15 / 32

The Elementary Renewal Theorem I

The Elementary Renewal Theorem (ERT):-

Given a sequence of mutually independent nonnegative random
variables (life-times) Xk , k ≥ 1, with Xk , k ≥ 2, being identically
distributed, such that,
for k ≥ 1, P(Xk < ∞) = 1
(all the life-time random variables are proper),
0 ≤ E(X1 ) ≤ ∞
(we allow the mean of the first life-time to be 0 and also ∞),
and,
0 < E(Xk ) ≤ ∞ for k ≥ 2,
(the mean life-times after the first are positive, and possibly infinite,
and, of course, identical to E(X2 )).
Defining E(X2 ) = µ1 , and this hypothesis is equivalent to

0 ≤ µ < ∞.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 16 / 32

The Elementary Renewal Theorem II

Then the following conclusions hold

M(t)
1 lim t = µ, almost surely.
t→∞
 
M(t)
i.e., P lim = µ = 1.
t→∞ t
m(t)
2 lim t = µ.
t→∞

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 17 / 32

The Elementary Renewal Theorem III

Wald’s Lemma:-
Let N be a stopping time for an infinite sequence of mutually
independent random variables Xi , i ≥ 1. If
1 E(N) < ∞
2 E(|Xn |) < B, a constant, for all n ≥ 1, and
3 E(Xn ) = E(X1 ) for all n ≥ 1,
then " N #
X
E Xn = E[X1 ]E[N]
n=1

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 18 / 32

The Elementary Renewal Theorem IV

Remarks:-
Let us also observe that a seemingly obvious calculation does not
always work.
Let pk = P(N = k ), k ≥ 1, and define SN := Nn=1 Xn .
P

Now
" " N ## ∞
" N #
X X X
E[SN ] = E E Xn |N = pk E Xn |N = k
n=1 k =1 n=1
hP i
N
Now if we could write E n=1 Xn |N = k = kE(X1 ) then it would
immediately follow that E(SN ) = E(X1 )E(N).
However, in general, this step is incorrect, because conditioning on
N = k may change the joint distribution of Xi , 1 ≤ i ≤ k .
In fact, given that N = k the random variables Xi , 1 ≤ i ≤ k , may also
have become dependent.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 19 / 32

The Elementary Renewal Theorem V

As an illustration of the previous remark, consider a renewal

process with i.i.d. life-times Xi , i ≥ 1, and, for given t > 0, define the
random time N = M(t).
PM(t)
Now look at n=1 Xn , and observe that, given M(t) = k , it must be
that
k
X
Xn ≤ t,
n=1

so that the random variables X1 , X2 , . . . , Xk are conditionally

dependent and also are conditionally bounded between 0 and t.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 20 / 32

The Elementary Renewal Theorem VI

Proof of Wald’s Lemma:-

We can write
" N #
X
E [SN ] = E Xn
n=1
"∞ #
X
= E Xn I{n≤N}
n=1
∞
DCT
X  
= E Xn I{n≤N}
n=1

Now,

I{n≤N} = 1 − I{N≤n−1}
N is stopping time
= f (X1 , . . . , Xn−1 )

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 21 / 32

The Elementary Renewal Theorem VII

It therefore follows that Xn is independent of I{n ≤ N}, and we

obtain
∞
X  
E(SN ) = E[Xn ]E I{n≤N}
n=1
∞
X
= E[X1 ]P [N ≥ n]
n=1
∞
X
= E[X1 ] P [N ≥ n]
n=1
= E[X1 ]E[N]

and the result is proved.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 22 / 32

The Elementary Renewal Theorem VIII

Result 1:-
Let M(t) be a renewal process with i.i.d. life-times Xi , i ≥ 1, and
with 0 < E(X1 ) < ∞. Then M(t) is not a stopping time of the
process.
Proof:-
For n ≥ 1,

I{M(t)≤n} = 1 − I{M(t)≥n+1}
= 1 − I{Zn+1 ≤t}
= 1 − I{Pn+1 Xi ≤t}
i=1

= f (X1 , X2 , . . . , Xn+1 )

Hence, to determine if M(t) ≤ n we need to look at Xn+1 as well.

Thus, M(t) is not a stopping time.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 23 / 32

The Elementary Renewal Theorem IX

Result 2:-
Let M(t) be a renewal process with i.i.d. life-times Xi , i ≥ 1, and
with 0 < E(X1 ) < ∞. Then M(t) + 1 is a stopping time of the
process.
Proof:-
For n ≥ 1,

I{M(t)+1≤n} = 1 − I{M(t)≥n}
= 1 − I{Zn ≤t}
= 1 − I{Pni=1 Xi ≤t}
= f (X1 , X2 , . . . , Xn )

Hence, to determine if M(t) + 1 ≤ n we need to look till Xn .

Thus, M(t) + 1 is a stopping time.

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 24 / 32

The Elementary Renewal Theorem X

Result 3:-
Let M(t) be a renewal process with i.i.d. life-times Xi , i ≥ 1, and
with 0 < E(X1 ) < ∞. Then
 
E ZM(t)+1 = E(X1 )(m(t) + 1).

Proof:-
We apply Wald’s lemma
 
M(t)+1
  X
E ZM(t)+1 = E  Xn 
n=1

= E(X1 )E(M(t) + 1)
= E(X1 )(m(t) + 1)

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 25 / 32

Problems I

Exercise 1:-
Find the probability P(M(t) ≥ k ) in a renewal process having
lifetime density

ρe−ρ(x−δ) for x > δ


f (x) = .
0 for x ≤ δ

where δ > 0 is fixed.

Exercise 2:-
Show that the renewal function corresponding to the lifetime
density
f (x) = λ2 xe−λx , x ≥ 0,
is
1 1 
m(t) = λt − 1 − e−2λt .
2 4

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 26 / 32

Problems II

Exercise 3:-
Consider a renewal process M(t), t ≥ 0, for which the sequence of
the life-times X1 , X2 , . . . are i.i.d exponential with rate λ.
Show that the

(λt)k −λt
P(M(t) = k ) = e for k = 0, 1, . . . .
k!
Find the distribution of nth renewal instant Zn .

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 27 / 32

Problems III

Exercise 4:-
Cars arrive at a gate. Each car is of random length L having
distribution function F (ζ). The first car arrives and parks against
the gate. Each succeeding car parks behind the previous one at a
distance that is random according to a uniform distribution [0, 1].
Consider the number of cars Nx that are lined up within a total
distance x of the gate. Determine

lim E[Nx ]/x,

x→∞

for
for F (ζ) a degenerate distribution of length c,
and also for the case F (ζ) = 1 − e−ζ .

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 28 / 32

Problems IV

Exercise 5:-
The weather in a certain locale A consists of rainy spells
alternating with spells when the sun shines. Suppose that the
number of days of each rainy spell is Poisson distributed with
parameter 2 and a sunny spell is distributed according to a
geometric distribution with mean 7 days. Assume that the
successive random duration of rainy and sunny spells are
statistically independent variables. In the long run, what is the
probability on a given day that it will be raining?

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 29 / 32

Problems V

Exercise 6:-
Consider a renewal process for which the life-times X1 , X2 , . . . are
discrete random variables having the Poison distribution with
mean λ. That is

λk −λ
P(Xn = k ) = e for k = 0, 1, . . .
k!

Determine P(M(t) = k ).
Find the distribution of nth renewal instant Zn .

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 30 / 32

Problems VI

Exercise 7:-
Consider a DTMC Yn , n ≥ 0, on S = 0, 1, 2, . . . , with Y0 = i and
the state j is given to be recurrent (i.e., fjj = 1 and fij = 1.)
Notice that, the visits to the state j will define a renewal process,
with life-times Xj , j ≥ 1. (Note that here all the lifetimes are integer
valued random variables.)
Let E(X2 ) = νj , be the mean return time to the state j. Then show
that
n 
1 X (k ) 0 if j is null recurrent
lim pij = γj
n→∞ n > 0 if j is positive recurrent
k =1

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 31 / 32

Problems VII
Exercise 8:-
{Xi , i ≥ 1} is a sequence of i.i.d. random variables with
P(Xi = 1) = P(Xi = −1) = 1/2. {Sn , n ≥ 0} is a process defined
as follows.
Xn
S0 = −1 and Sn = S0 + Xi .
i=1

Let N = min{j ≥ 1 : Sj = 1} i.e., N is the hitting time of the state 1.

Show that N is a stopping time for the process {Sn , n ≥ 0}.

Clearly,
XN
SN = −1 + Xi .
i=1

Show that an anomaly arises if we “apply" Wald’s Lemma to get

E(SN ). Explain the anomaly. (Hint: a hypothesis of Wald’s Lemma
fails.)

Sudipta Das (RKMVERI) Stochastic Models MCS-PSP-2021 32 / 32