Scribd
Upload
132 views19 pages

Chapter3 Part1

Uploaded by

s230005074
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
132 views19 pages

Chapter3 Part1

Uploaded by

s230005074
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

1

Markov Chains
4.1 Introduction
2

 Markov chain:
 A discrete time process {Xn, n = 0, 1, 2, . . .} with discrete state
Space X n {0,1,2,…,} is a Markov chain if it has the Markov
property:

In words, for a Markov chain, the conditional distribution of any future


state Xn+1, given the past states X0,X1, . . . , Xn−1 and the present state Xn,
is independent of the past states and depends only on the present state.
4.1 Introduction
3

We consider homogeneous Markov chains for which

Define

The value Pij represents the probability that the process will, when
in state i, next make a transition into state j.
Since probabilities are nonnegative and since the process must
make a transition into some state, we have that
4.1 Introduction
4

Let P = [Pij ] denote the (possibly infinite) transition matrix of


the one-step transition probabilities so that

n step transition probability matrix P(n) = (p(n)ij)


pij(n) = P{Xn = j | X0 = i}
Example 4.1.1 (A Gambling model)
5

A gambler either wins 1 EUR with probability p or loses 1 EUR


w.p. 1 − p. The gambler quits if he either goes broke or attains a
fortune N EUR. Then the gambler’s fortune is a Markov chain
with state space {0, 1, . . . ,N} having transition probabilities:

Such Markov chain is a random walk with barriers (states 0 and N).
States 0 and N are called absorbing states since once entered they
are never left.
Example 4.1.1. The simple Random
Walk
6
Example 4.1. The simple
7
Random Walk

Proof.
Example 4.1. The simple Random
Walk
8

Proof. (cont.)
Example 4.1. The simple Random
Walk
9

Proof. (cont.)
Example 4.1 The simple Random
Walk
10

• From the proposition, it follows upon conditioning on whether


Sn = +i or −i that
4.2. Chapman-Kolmogorov
Equations
11
4.2 Chapman-Kolmogorov Equations
12
4.2 Chapman-Kolmogorov Equations
13
Example 4.2.1 (Forecasting the
weather)
14

Suppose that the chance of rain tomorrow depends on previous


conditions only through whether or not it is raining today.

• If it rains today, it will rain tomorrow with probability α.


• If it does not rain today, it will rain tomorrow with probability
β

The process is in state 0 if it rains and in state 1 if it does not


rain. Then we have a two-state Markov chain whose transition
probabilities are given by
Example 4.2.1 (Four-days weather
forecast)
15

Still consider the Example: State 0: rain, State 1: no rain

Take α=0.7, β=0.4. Given it its raining today, what is the chance
that it will rain after 4 days from today?
Solution:
Example 4.2.1 (Four-days weather forecast)
16

State 0: rain, State 1: no rain


4.2 Chapman-Kolmogorov Equations
17

Example 4.2.2 An urn always contains 2 balls. Ball colors are red
and blue. At each stage a ball is randomly chosen and then replaced
by a new ball, which with probability 0.8 is the same color, and
with probability 0.2 is the opposite color, as the ball it replaces. If
initially both balls are red, find the probability that the fifth ball
selected is red.

Solution: Let us define Xn to be the number of red balls in the urn


after the nth selection and subsequent replacement. Then Xn, n ≥ 0,
is a Markov chain with states 0, 1, 2 and with transition probability
matrix P given by
4.2 Chapman-Kolmogorov Equations
18

Example 4.2.2 (cont.)

To determine the probability that the fifth selection is red, condition on


the number of red balls in the urn after the fourth selection. This yields

To calculate the preceding we compute P4. Doing so yields

giving the answer P(fifth selection is red) = 0.7048.


Another weather example
19

Solution: Pnij is the probability that the state at time n is j given that the initial
state at time 0 is i.
p=matrix(c(0.4,.2,0.1,0.6,0.5,0.7,0,0.3,0.2),nrow=3)
pp=0
for (i in 2:3)
for (j in 2:3)
for (k in 2:3)
pp=pp+p[3,i]*p[i,j]*p[j,k]

You might also like