Lecture 6-7: Continuous times Markov chains.

Poisson Process.
Birth and Death process.

Continuous Time Markov Chains

➢ In this lecture, we will discuss Markov Chains in continuous time.
➢ Continuous time Markov Chains are used to represent
• population growth,
• epidemics,
• queueing models,
• reliability of mechanical systems, etc.

➢ In Continuous time Markov Process, the time is perturbed by exponentially distributed

holding times in each state while the succession of states visited still follows a discrete time
Markov chain.

we have been discussing discrete time and discrete state-space stochastic. However, there are
important stochastic processes for which S is discrete but the indexing set is continuous. These
processes may change their values at any instant of time rather than at specified times. In other
words, the stochastic process can change instantaneously
An important process of this type is called the Poisson process.
➢ The Poisson process is one of the most widely-used counting processes. It is usually used in
scenarios where we are counting the occurrences of certain events that appear to happen at a
certain rate, but completely at random.
➢ For example, suppose that from historical data, we know that earthquakes occur in a certain area
with a rate of 22 per month. Other than this information, the timings of earthquakes seem to be
completely random. Thus, we conclude that the Poisson process might be a good model for
earthquakes. In practice, the Poisson process or its extensions have been used to model:
−− the number of car accidents at a site or in an area;
−− the location of users in a wireless network;
−− the requests for individual documents on a web server;
−− the outbreak of wars;
−− photons landing on a photodiode.
A stochastic process {N(t),t ≥ 0} is called a counting process if N(t) represents the total number of events
that occur by time t.
Definition: Poisson process
The Poisson process is a continuous-time process counting events taking place at random times,
such as e.g. customers arriving at a desk. Its definition follows.
Definition A continuous-time stochastic process (Nt , t ∈ R+) is a Poisson process with intensity
λ > 0 if

➢ N(t) represent the number of events that occured up to time t.

➢ The first condition N(0)=0, means the number of events occured at time 0 is 0.
As time increases, the number of events can only increase.
➢ The second condition says that, suppose there are n events occurs in the interval [0,t], then the
probability that there will be m events occuring in the interval (t,t+h] is given by the formula in
(b).
➢ o(h) term signifies that if h is small, then the probability that there will be more than one event
occuring in the interval (t,t+h] is very small.
➢ The third condition means that for any two disjoint intervals, then number of events occured in
one interval is independent with number of events occured in the other interval.
Theorem 1

(1)
 (2)
Thus, equation (1) becomes

(3)

By equation (3)

(4)
Equation (4) implies

(5)
Take the limit as h→0 on both side of (5), we have

(6)

(7)

(8)
 (9)

We start with (8) that is,

(10)
Thus, we have

(11)

(12)

(13)

(14)

(15)

(16)

(17)
From (6) and the assumption, we have

(18)

(19)
Again multiply eλt on both side of (19) we get

(20)

(21)

(22)
➢ Theorem (1) suggests that there is a deep connection between the
• Poisson process and
• the p.m.f. of Poisson distribution.
➢ The number of event occuring between (0,t] follows a Poisson distribution with parameter λt. As
the length of the interval increases, the mean of the Poisson distribution also increases linearly.

The Poisson Process as a Birth Process

Many processes in nature may change their values at any instant of time rather than at certain times only.
Depending of the underlying random mechanism, these processes may or may not satisfy a Markov
property. Before attempting to study a general theory of continuous Markov Processes we explore a one
simple in detail.
The Pure Birth Markov Processes

➢ a birth process or a pure birth process is a special case of a continuous-time Markov

process and a generalisation of a Poisson process.
➢ It defines a continuous process which takes values in the natural numbers and can only
increase by one (a "birth") or remain unchanged. This is a type of birth–death
process with no deaths.
➢ A pure birth process is a type of Markov process where the population size can only increase by
discrete integer values over time. The rate of increase (birth rate) may depend on the current
population size.
➢ The lecture provides the assumptions and governing equations of a pure birth process.
➢ It derives the difference-differential equations that describe how the probability of being in each
population state changes over time. Several special cases of pure birth processes are also defined.

➢ A pure birth process is a continuous time, discrete state Markov process for which state
transitions are either the process remains at the current state or the state increases by one unit. The
process cannot move from higher state to a lower state since there is no death. Specifically, we
deal with a family of random variables {X(t);t ≥ 0} where the possible values of X (t) are non
negative integers representing the population size t time t.

➢ Suppose the process is at time t; then when a birth occurs, the process goes from state n to state
n+1. If no birth occurs, the process remains at state n. The birth process is characterized by the
birth rate λn which varies according to the state n.

Assumptions of process:
The process {X(t);t ≥ 0} satisfies the following assumptions: during time interval [t,t+δt] where δt is a
sufficiently small time interval
(i)Pr[X (t + δt) − X (t) = 1|X (t) = n] = λnδt+ o(δt)
(ii) Pr[X(t + δt) − X(t) = 0|X(t) = n] = 1 − λnδt + 0(δt)
(iii) Pr[X (t + δt) − X (t) ≥ 2|X (t) = n] = o(δt)
Where
o(𝛿𝑡)
lim =0
δt→0 𝛿𝑡
We use these assumptions to define system of equations that are useful in deriving properties of
population size at given time t.
Simple birth process

A Poisson process is a special case of a birth process.

A Poisson process is a birth process where the birth rates are constant, i.e λn= λ for some λ positive

A simple birth process, where birth rates are equal to the size of the current population.

Linear Birth
A simple linear birth process is a birth process with rates λn= nλ. It models a population in which each
individual gives birth repeatedly and independently at rate λ.
The number of births in time t from a simple linear birth process of population n is given by:
𝑛
𝑃𝑛,𝑛+𝑚 (𝑡) = ( ) (𝜆t)𝑚 (1 − 𝜆t)𝑛−𝑚
𝑚
In exact form, the number of births is the negative binomial distribution . (visit the text on pages 66-69)

The Death Markov Process

When a birth occurs, the process goes from state n to n + 1. When a death occurs, the process goes from
state n to state n − 1. The process is specified by positive birth rates λn and positive death rates μi.

➢ Individuals persist only until they die and there are no births.
➢ We assume initially, there are n0 number of individuals at time t = 0.
Linear Death

(visit the text on pages 69 -72)

Birth-and-Death process: An Introduction

The birth-death process is a special case of continuous time Markov process, where the states (for
example) represent a current size of a population and the transitions are limited to birth and death. When a
birth occurs, the process goes from state i to state i + 1. Similarly, when death occurs, the process goes
from state i to state i − 1. It is assumed that the birth and death events are independent of each other,

Definition

(23)
(23) tells us that if the number of events happend in [0,t] is i, then how many events will occur in (t,t+h].

Note:
➢ There is a basic difference between the birth and death process and the Poisson process.
 • in the Poisson process, the number of events can only increase over time, while
• in the birth and death process, the number of events can also decrease.
• When the number of events increase, we call it a birth process and
• when the number of events decrease, we call it a death process.

For example, if you have up to time t, i events occurred. Then between t to t+h, there can be a
birth, a death, or nothing happens, with probability given by the first three equations in (23) .
➢ λi is called the birth rate and
➢ μi is called the death rate.

The last equation in (23) suggests that the probabilities that there will be more than one birth or
more than one death will be very small if the time interval is small.

Application: Here are some typical situations where birth and death process is used to model the
data.
1. Demography: how population of a particular community evolves over time.
2. Queuing theory: in a counter, look at the number of people in the queue. A person is
served and leaves the queue is called a “death”, and a person joins the queue is called a
“birth”.
3. Mathematical biology: to model the evolotion of bacteria.

(24)

(25)

(26)
(visit the text on pages 72 -80)

Get more special cases on birth and death process on pages 80-86