The first step analysis

1 Some important definitions.

Let Xn , n = 0, 1, 2, ... be a sequence random variables takin values in S = (1, 2, ...m).
Such sequences are often called random processes (RP). S is called the state space.

Definition 1.1
A RP Xn , n = 0, 1, 2, ... is called a Markov chain (MC) if

P {Xn+1 = j | Xn = i, Xn−1 = in−1 , . . . , X0 = i0 } = P {Xn+1 = j | Xn = i} (1)

Relation (??) is called the Markov property. It implies that once Xn is given the
future development of the MC does not depend on what has happened before time n.
This is sometimes stated as “the future and the past are independent if the present
is known”.
The probabilities P {Xn+1 = j | Xn = i} are called the transition probabilities. If these
probabilities do not depend on n then the MC Xn is said to be homogeneous (or more
precisely, homogeneous in time). In this course we shall study only homogeneous
def
MCs. A homogeneous MC is basically described by its transition probabilities pij =
P {Xn+1 = j | Xn = i} and it is convenient and useful to present the collection of
these probabilities as a transition matrix of the MC Xn :
 
p11 p12 . . . p1m
 p21 p22 . . . p2m 
P = (pij ) =  ..
 
.. .. .. 
 . . . . 
pm1 pm2 . . . pmm

2 A reminder
We recall the following general formula for an expectation of a random variable known
from Probability II. Let ξ be a randomTvariable and let events B1 , B2 , ..., Bm be such
that P (B1 ) + ... + P (Bm ) = 1 and Bj Bk = ∅ if j 6= k. Then
m
X
E(ξ) = P (Bj )E(ξ | Bj ). (2)
j=1

In particular, if η is a random variable defined on a set of trajectories of a MC then

m
X
E(η | X0 = i) = pij E(η | X0 = i, X1 = j). (3)
j=1

Note that (??) is obtained from (??) with Bj = {X1 = j, | X0 = i} and ξ = (η | X0 =

i).

1
3 First step analysis (FSA)
We say that a state i of a MC Xn is absorbing if pii = 1.
Suppose that a MC Xn has absorbing states and let T be the time at which it reaches
(is absorbed by) one of these states. If the MC eventually reaches one of the absorbing
states (or, equivalently, if T is finite) then we say that Xn is an absorbing MC.
Remark. Strictly speaking, Xn is an absorbing MC if T is finite with probability 1.
However, this level of rigor is beyond the technical means of our course.
Let f (i) be a function on S taking real values and set
T
X
F = f (X0 ) + f (X1 ) + ... + f (XT ) ≡ f (Xj ).
j=0

FSA is a technique which allows one to find E(F | X0 = i) by establishing a relation

between this quantity and the one into which this expectation is transformed after
the MC makes its first step, namely the E(F | X0 = i, X1 = j). Set

wi = E(f (X0 ) + f (X1 ) + ... + f (XT ) | X0 = i) ≡ E(F | X0 = i).

The FSA allows one to prove the following

Theorem 3.1
Suppose that Xn is an absorbing MC. Then
(
wi = f (i) if i is absorbing
(4)
wi = f (i) + m
P
j=1 pij wj if i is not absorbing

and these equation have a unique solution.

We shall prove that equations (??) hold. The existence and uniqueness of their
solutions will not be proved.
Proof. The proof of the first equation is immediate since if i is absorbing then
(F | X0 = i) = f (i) and hence wi = E(F | X0 = i) = f (i).
To prove the second equation we start with formula (??) with η = F . Observe that
then E(η | X0 = i, X1 = j) becomes

E(F | X0 = i, X1 = j) = f (i) + E(f (X1 ) + f (X2 ) + ... + f (XT ) | X1 = j)

= f (i) + E(f (X0 ) + f (X1 ) + ... + f (XT ) | X0 = j).

The last step in this formula is due to the fact that we deal with a homogeneous MC
and therefore the distribution of (f (X1 ) + f (X2 ) + ... + f (XT ) | X1 = j) is the same
as that of (f (X0 ) + f (X1 ) + ... + f (XT ) | X0 = j). Hence

E(f (X1 )+f (X2 )+...+f (XT ) | X1 = j) = E(f (X0 )+f (X1 )+...+f (XT ) | X0 = j) ≡ E(F | X0 = j)

and thus
E(F | X0 = i, X1 = j) = f (i) + wj .

2
Substituting this expression into the r.h.s. of (??) and replacing the l.h.s. of (??) by
wi we obtain
Xm m
X m
X
wi = pij (f (i) + wj ) = f (i) pij + pij wj . (5)
j=1 j=1 j=1
Pm
Since j=1 pij = 1, we see that (??) implies (??) and this finishes the proof of the
theorem. 

4 Examples.
The examples ( presented below differ by the choice of the function f .
0 if i is an absorbing state
1) Set f (i) =
1 if i is not an absorbing stae
Then F = f (X0 ) + f (X1 ) + ... + f (XT ) = f (X0 ) + f (X1 ) + ... + f (XT −1 ) = T . Indeed,
XT is an absorbing state by the definition of T whereas Xr , 0 ≤ r ≤ T − 1, is not.
Hence f (XT ) = 0 and f (X0 ) = f (X1 ) = ... = f (XT −1 = 1. We recover the equations
def
for vi = E{T | X0 = i} (derived directly from the definition of T in one of previous
lectures), namely
(
vi = 0 if i is absorbing
(6)
ii = 1 + m
P
j=1 pij vj if i is not absorbing

(
1 if i = k
2) For a fixed non-absorbing state k set f (i) = Then
0 in all other cases

F = f (X0 )+f (X1 )+...+f (XT ) = f (X0 )+f (X1 )+...+f (XT −1 ) = the number of visits to k.
Indeed, f (XT ) = 0 since XT is an absorbing state and for f (Xr ) = 1 if and only if
Xr = k. Hence the equality.
def
Let us denote by a the number of visits to k and set bi = E{a | X0 = i}. Then bi can
be found from the following equations
(
bi = 0 if i is absorbing
(7)
bi = δik + m
P
j=1 pij bj if i is not absorbing
(
def 1 if i = k
(Remember that δik = )
0 if i 6= k
(
1 if i = k
3) For a fixed absorbing state k set f (i) = Then
0 in all other cases
(
1 if XT = k
F = f (X0 ) + f (X1 ) + ... + f (XT ) = f (XT ) =
0 if XT 6= k.

3
Indeed, f (X0 ) = f (X1 ) = ... = f (XT −1 = 0 because X0 , ..., XT −1 are not absorbing
states and f (XT ) = 1 if and only if XT = k.
Note next that in this case ui = E(F | X0 = i) = E{XT | X0 = i} = P {XT = k, | X0 =
i}. Once again, we recover the equations for ui (derived directly from the definition
of ui in one of previous lectures), namely
(
ui = δik if i is absorbing
Pm (8)
ui = j=1 p ij uj if i is not absorbing

4) If a game is described by our MC, f (i) is sometimes interpreted as a reward

for visiting state i (which can perfectly well be a negative number) and thus F =
P T
k=0 f (Xk ) is the reward paid out at the end of the game. The wi is then the
expectation (the mean value) of the reward given that the starting point of the process
is i. If the initial P
distribution π = (p1 , ..., pm ) of the chain is known then we can also
find the E(F ) = m i=1 pi wi .
In some cases it is natural to say that a game is fair if E(F ) = 0. Solving equations
(??) enables one to find out whether or not a particular game is fare.

We shall consider much more concrete examples in lectures.