Chapter 10. Markov chains.

Manual for SOA Exam MLC.

Chapter 10. Markov chains.
Section 10.2. Markov chains.

2008.
c Miguel A. Arcones. All rights reserved.

Extract from:
”Arcones’ Manual for SOA Exam MLC. Fall 2009 Edition”,
available at http://www.actexmadriver.com/

1/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Markov chains

Definition 1
A discrete time Markov chain {Xn : n = 0, 1, 2, . . . } is a
stochastic process with values in the countable space E such that
for each i0 , i1 , . . . , in , j ∈ E ,

P{Xn+1 = j|X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in } (1)

= P{Xn+1 = j|Xn = in }.

2/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Markov chains

Definition 1
A discrete time Markov chain {Xn : n = 0, 1, 2, . . . } is a
stochastic process with values in the countable space E such that
for each i0 , i1 , . . . , in , j ∈ E ,

P{Xn+1 = j|X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in } (1)

= P{Xn+1 = j|Xn = in }.

Since Xn takes values in the countable set E , Xn has a discrete

distribution.

3/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Markov chains

Definition 1
A discrete time Markov chain {Xn : n = 0, 1, 2, . . . } is a
stochastic process with values in the countable space E such that
for each i0 , i1 , . . . , in , j ∈ E ,

P{Xn+1 = j|X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in } (1)

= P{Xn+1 = j|Xn = in }.

Since Xn takes values in the countable set E , Xn has a discrete

distribution.
The set E in the previous definition is called the state space.
Usually, E = {0, 1, 2, . . . } or E = {1, 2, . . . , m}. We will assume
that E = {0, 1, 2, . . . }. Each element of E is called a state. If
Xn = k, where k ∈ E , we say that the Markov chain {Xn }∞ n=0 is at
state k at stage n. 4/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

For a Markov chain the conditional distribution of any future state

Xn+1 given the past states X0 , X1 , . . . , Xn−1 and the present state
Xn is independent of the past values and depends only on the
present state. Having observed the process until time n, the
distribution of the process after time n on depends only on the
value of the process at time n. The interpretation of

P{Xn+1 = j|X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in }

=P{Xn+1 = j|Xn = in }.

is that given the present the future is independent of the past. In

order words, the evolution of the process depends only on the
present and not in the past.

5/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Definition 2
Given events A, B and C such that P{C } > 0, we say that A and
B are independent given C if

P{A ∩ B|C } = P{A|C }P{B|C }.

Theorem 1
Given events A, B and C such that P{B ∩ C } > 0, we have that A
and B are independent given C if and only if

P{A|B ∩ C } = P{A|C }.

Proof: A and B are independent given C if and only if

P{A∩B∩C } } P{B∩C }
P{C } = P{A∩C
P(C } P{C } . P{A|B ∩ C } = P(A|C } if and only if
P{A∩B∩C } P{A∩C }
P{B∩C } = P{C } .

6/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

By the previous theorem, for a Markov chain

{Xn : n = 0, 1, 2, . . . }, for each i0 , . . . , in+1 ∈ E , given that
Xn = in , (X0 , X1 , . . . , Xn−1 ) = (i0 , . . . , in−1 ) and Xn+1 = in+1 are
independent.

7/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 1
A fair coin is thrown out repeatedly. Let Xn be the total number of
heads obtained in the first n throws, n = 0, 1, 2 . . . . Notice that
X0 = 0. Xn has a binomial distribution with parameters n and 21 .
The state space is E = {0, 1, 2, . . . , }. {Xn }∞
n=0 is a Markov chain
because for each i0 , i1 , . . . , in , j ∈ E ,

P{Xn+1 = j|X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in }

=P{Xn+1 = j|Xn = in },

1
 2 if j = in ,

= 12 if j = in + 1,

0 else.


8/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Let αn (i) = P{Xn = i}, i ∈ E .

(α0 (i))i∈E is called the initial distribution of the Markov chain.
(αn (i))i∈E is called the distribution
P of the Markov chain at time n.
Notice that αn (i) ≥ 0 and i∈E αn (i) = 1.
We will denote to the row vector (αn (i))i∈E by αn . For example, if
E = {0, 1, . . . , k},
αn = (αn (0), αn (1), . . . , αn (k)).

9/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Let αn (i) = P{Xn = i}, i ∈ E .

(α0 (i))i∈E is called the initial distribution of the Markov chain.
(αn (i))i∈E is called the distribution
P of the Markov chain at time n.
Notice that αn (i) ≥ 0 and i∈E αn (i) = 1.
We will denote to the row vector (αn (i))i∈E by αn . For example, if
E = {0, 1, . . . , k},
αn = (αn (0), αn (1), . . . , αn (k)).

Theorem 2
X
αn (i) ≥ 0 and αn (i) = 1.
i∈E

10/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Let αn (i) = P{Xn = i}, i ∈ E .

(α0 (i))i∈E is called the initial distribution of the Markov chain.
(αn (i))i∈E is called the distribution
P of the Markov chain at time n.
Notice that αn (i) ≥ 0 and i∈E αn (i) = 1.
We will denote to the row vector (αn (i))i∈E by αn . For example, if
E = {0, 1, . . . , k},
αn = (αn (0), αn (1), . . . , αn (k)).

Theorem 2
X
αn (i) ≥ 0 and αn (i) = 1.
i∈E

Proof: αn (i) = P{Xn = i} ≥ 0. We also have that

X X
αn (i) = P{Xn = i} = P{Xn ∈ E } = 1.
i∈E i∈E
11/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Let Qn (i, j) = P{Xn+1 = j|Xn = i}, where i, j ∈ E . Qn (i, j) is

called the one–step transition probability from state i into state
j at stage n. We have that (Qn (i, j))i,j∈E is a matrix. If
E = {0, 1, , . . . , k}, then
 
Qn (0, 0) Qn (0, 1) · · · Qn (0, k)
 Qn (1, 0) Qn (1, 1) · · · Qn (1, k) 
(Qn (i, j))i,j∈E =   ···

··· ··· ··· 
Qn (k, 0) Qn (k, 1) · · · Qn (k, k)

12/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

The row of the matrix Qn for the i entry is formed by conditional

probabilities given Xn = i, the departing state. The column of the
matrix Qn for the j entry consists of conditional probabilities for
Xn = j, the arriving state. If we write the states in the matrix Qn ,
we have
arriving states
0 1 ··· k
Qn (0, 0) Qn (0, 1) · · · Qn (0, k)
 
0
 Qn (1, 0) Qn (1, 1) · · · Qn (1, k) 
1  
·  · · · · 
departing states  
· 
 · · · · 

·  · · · · 
k Qn (k, 0) Qn (k, 1) · Qn (k, k)

To find Qn (i, j) in this matrix, we need to look for i in the rows

and j in the columns.
13/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 2
Consider a Markov chain with E = {0, 1, 2} and
 
0.2 0.3 0.5
Q6 = 0.3
 0.5 0.2 .
0.1 0.1 0.8

Find:
(i) P{X7 = j|X6 = 0}, j = 0, 1, 2.
(ii) P{X7 = 1|X6 = j}, j = 0, 1, 2.
Solution: (i) We have that
P{X7 = 0|X6 = 0} = 0.2, P{X7 = 1|X6 = 0} = 0.3
and P{X7 = 2|X6 = 0} = 0.5.
(ii) We have that
P{X7 = 1|X6 = 0} = 0.3, P{X7 = 1|X6 = 1} = 0.5
and P{X7 = 1|X6 = 2} = 0.1.
14/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

In the previous example, P{X7 = j|X6 = 0}, j = 0, 1, 2, are all the

transition probabilities from state 0 at time 6 to another state at
time 7. In the previous problem, P{X7 = j|X6 = 0}, j = 0, 1, 2, are
all the transition probabilities from time 6 to time 7 arriving at
state 1.

15/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Theorem 3
The one–step transition probabilities Qn (i, j) satisfy that
X
Qn (i, j) ≥ 0 and Qn (i, j) = 1,
j∈E

i.e. the sum of the elements in each row is one.

16/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Theorem 3
The one–step transition probabilities Qn (i, j) satisfy that
X
Qn (i, j) ≥ 0 and Qn (i, j) = 1,
j∈E

i.e. the sum of the elements in each row is one.

Proof.
Since Qn (i, j) is a conditional probability, Qn (i, j) ≥ 0. We also
have that
X X
Qn (i, j) = P{Xn+1 = j|Xn = i} = P{Xn+1 ∈ E |Xn = i} = 1.
j∈E j∈E

17/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 3
Which of the following are legitime one-step transition probability
matrices  
−1 2
(i) Q0 = .
0.5 0.5
 
0.3 0.7
(ii) Q0 = .
0.4 0.7

18/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 3
Which of the following are legitime one-step transition probability
matrices  
−1 2
(i) Q0 = .
0.5 0.5
 
0.3 0.7
(ii) Q0 = .
0.4 0.7
Solution: (i) The matrix is not a legitime transition probability
because the entry (1, 1) is negative.
(ii) The matrix is not a legitime transition probability because the
elements of the second row do not add to one.

19/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

(k) (k)
We define Qn (i, j) = P{Xn+k = j|Xn = i}. Qn (i, j) is called the
k–step transition probability from state i into state j at time n.
Theorem 4
(k) (k)
X
Qn (i, j) ≥ 0 and Qn (i, j) = 1.
j∈E

20/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Lemma 1
Successive conditioning rule. For each B, A1 , A2 , . . . , An ⊂ Ω,

P{A1 ∩ A2 ∩ · · · ∩ An |B}
=P{A1 |B}P{A2 | B ∩ A1 } · · · P{An | B ∩ A1 ∩ A2 ∩ · · · ∩ An−1 }.

21/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Lemma 1
Successive conditioning rule. For each B, A1 , A2 , . . . , An ⊂ Ω,

P{A1 ∩ A2 ∩ · · · ∩ An |B}
=P{A1 |B}P{A2 | B ∩ A1 } · · · P{An | B ∩ A1 ∩ A2 ∩ · · · ∩ An−1 }.

Proof:

P{A1 |B}P{A2 | B ∩ A1 } · · · P{An | B ∩ A1 ∩ A2 ∩ · · · ∩ An−1 }

P{B ∩ A1 } P{B ∩ A1 ∩ A2 } P{B ∩ A1 ∩ A2 ∩ · · · ∩ An−1 ∩ An }
= ···
P{B} P{B ∩ A1 } P{B ∩ A1 ∩ A2 ∩ · · · ∩ An−1 }
P{B ∩ A1 ∩ A2 ∩ · · · ∩ An−1 ∩ An }
=
P{B}
=P{A1 ∩ A2 ∩ · · · ∩ An |B}.

22/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Corollary 1
For each A1 , A2 , . . . , An ⊂ Ω,

P{A1 ∩ A2 ∩ · · · ∩ An }
=P{A1 }P{A2 | A1 } · · · P[An | A1 ∩ A2 ∩ · · · ∩ An−1 }.

23/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Corollary 1
For each A1 , A2 , . . . , An ⊂ Ω,

P{A1 ∩ A2 ∩ · · · ∩ An }
=P{A1 }P{A2 | A1 } · · · P[An | A1 ∩ A2 ∩ · · · ∩ An−1 }.

Proof: By Lemma 1,

P{A1 ∩ A2 ∩ · · · ∩ An |B}
=P{A1 |B}P{A2 | B ∩ A1 } · · · P{An | B ∩ A1 ∩ A2 ∩ · · · ∩ An−1 }.

Taking B = Ω, we get

P{A1 ∩ A2 ∩ · · · ∩ An }
=P{A1 }P{A2 | A1 } · · · P{An | A1 ∩ A2 ∩ · · · ∩ An−1 }.

24/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 1
From a deck of 52 cards, you withdraw three cards one after
another. Find the probability that the first two cards are spades
and the third one is a club.

25/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 1
From a deck of 52 cards, you withdraw three cards one after
another. Find the probability that the first two cards are spades
and the third one is a club.
Solution: Let A1 = {first card is a spade}, let
A2 = {second card is a spade} and let
A3 = {third card is a club}. We have that

P{A1 ∩ A2 ∩ A3 } = P{A1 }P{A2 |A1 }P{A3 |A1 ∩ A2 }

13 12 13
= = 0.01529411765.
52 51 50

26/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Theorem 5
(Basic theorem for Markov chains)
(i)

P{X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in }

=α0 (i0 )Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in )

(ii)

αn (in ) = P{Xn = in }
X
= α0 (i0 )Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in ).
i0 ,i1 ,...,in−1 ∈E

27/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Proof: (i) Using Corollary 1 and the Markov property,

P{X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in }

=P{X0 = i0 }P{X1 = i1 |X0 = i0 }P{X2 = i2 |X0 = i0 , X1 = i1 }
· · · P{Xn = in |X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 }
=P{X0 = i0 }P{X1 = i1 |X0 = i0 }P{X2 = i2 |X1 = i1 }
· · · P{Xn = in |Xn−1 = in−1 }
=α0 (i0 )Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in ).

28/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

(ii) Notice that

{Xn = in }
= ∪i0 ,i1 ,...,in−1 ∈E {X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in },

where the union is over disjoint events. Hence,

αn (in ) = P{Xn = in }
X
= P{X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in }
i0 ,i1 ,...,in−1 ∈E
X
= α0 (i0 )Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in ).
i0 ,i1 ,...,in−1 ∈E

29/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

X
αn (in ) = α0 (i0 )Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in )
i0 ,i1 ,...,in−1 ∈E

implies that
αn = α0 Q0 Q1 · · · Qn−1 .
Matrix multiplication is used in the previous formula. For example,
if E = {0, 1, . . . , k},
α0 Q0
 
Q0 (0, 0) Q0 (0, 1) ··· Q0 (0, k)
 Q0 (1, 0) Q0 (1, 1) ··· Q0 (1, k) 
= (α0 (0), α0 (1), . . . , α0 (k)) 
 ···

··· ··· ··· 
Q0 (k, 0) Q0 (k, 1) ··· Q0 (k, k)
k k k
!
X X X
= α0 (i)Q0 (i, 0), α0 (i)Q0 (i, 1), . . . , αn (i)Q0 (i, k))
i=0 i=0 i=0
=α1 ,
30/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

α0 Q0 Q1 = α1 Q1
k k
!
X X
= α0 (i)Q0 (i, 0), . . . , α0 (i)Q0 (i, k))
i=0 i=0
 
Q1 (0, 0) · · · Q1 (0, k)
 Q1 (1, 0) · · · Q1 (1, k) 
× ···

··· ··· 
Q1 (k, 0) · · · Q1 (k, k)
 
Xk X k k
X
= αn (i)Q0 (i, j)Q1 (j, 0), . . . , αn (i)Q0 (i, j)Q1 (j, k)
i=0 j=0 i=0

=α2 .

31/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

32/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(i) P{X0 = 1}.

33/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(i) P{X0 = 1}.

Solution: (i) P{X0 = 1} = 0.4.

34/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(ii) P{X1 = 1}.

35/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(ii) P{X1 = 1}.

Solution: (ii) We have that
2
X
P{X1 = 1} = α0 (j)Q1 (j, 1) = (0.3)(0.3) + (0.4)(0.5) + (0.3)(0.1),
j=0

= 0.32, which is the second entry in the vector

 
0.2 0.3 0.5
(0.3, 0.4, 0.3) 0.3 0.5 0.2 = (0.21, 0.32, 0.47).
0.1 0.1 0.8 36/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(iii) P{X2 = 1}.

37/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(iii) P{X2 = 1}.

Solution: (iii) We have that
  
0.2 0.3 0.5 0.1 0.1 0.8
(0.3, 0.4, 0.3) 0.3 0.5 0.2 0.3 0.5 0.2
0.1 0.1 0.8 0.3 0.5 0.2
 
0.1 0.1 0.8
=(0.21, 0.32, 0.47) 0.3 0.5 0.2
0.3 0.5 0.2
=(0.258, 0.416, 0.326), and P{X2 = 1} = 0.416.
38/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(iv) P{X0 = 1, X1 = 2}.

39/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(iv) P{X0 = 1, X1 = 2}.

Solution: (iv) 1 7→ 2,

P{X0 = 1, X1 = 2} = α0 (1)Q0 (1, 2) = (0.4)(0.2) = 0.08.

40/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(v) P{X0 = 1, X1 = 0}.

41/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(v) P{X0 = 1, X1 = 0}.

Solution: (v) 1 7→ 0,

P{X0 = 1, X1 = 0} = α0 (1)Q0 (1, 0) = (0.4)(0.3) = 0.12.

42/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(vi) P{X0 = 1, X1 = 2, X2 = 2}.

43/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(vi) P{X0 = 1, X1 = 2, X2 = 2}.

Solution: (vi)

P{X0 = 1, X1 = 2, X2 = 2} = α0 (1)Q0 (1, 2)Q1 (2, 2)

=(0.4)(0.2)(0.2) = 0.016.

44/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(vii) P{X0 = 2, X1 = 1, X2 = 0}.

45/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(vii) P{X0 = 2, X1 = 1, X2 = 0}.

Solution: (vii) 2 7→ 1 7→ 0,

P{X0 = 2, X1 = 1, X2 = 0} = α0 (2)Q0 (2, 1)Q1 (1, 0)

=(0.3)(0.1)(0.3) = 0.009.

46/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(viii) P{X0 = 1, X2 = 2}.

47/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 4
Consider a Markov chain with E = {0, 1, 2}, α0 = (0.3, 0.4, 0.3),
   
0.2 0.3 0.5 0.1 0.1 0.8
Q0 = 0.3 0.5 0.2 , Q1 = 0.3 0.5 0.2 .
0.1 0.1 0.8 0.3 0.5 0.2

(viii) P{X0 = 1, X2 = 2}.

Solution: (viii)
2
X
P{X0 = 1, X2 = 2} = P{X0 = 1, X1 = j, X3 = 2}
j=0

=(0.4)(0.3)(0.8) + (0.4)(0.5)(0.2) + (0.4)(0.2)(0.2) = 0.152.

48/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Suppose that the annual effective rate of interest is i. Consider the

cashflow
payments C1 C2 ··· Cn
Time (in years) t1 t2 ··· tn
The present value of the former cashflow at time t is
n
X
Cj (1 + i)t−tj .
j=1

If the time at which the present value is omitted, we assume that

the time is time zero. The present value of the former cashflow is
n
X
Cj (1 + i)−tj .
j=1

The annual interest factor is 1 + i.

The annual discount factor is ν = (1 + i)−1 .
i
The annual effective rate of discount is d = 1 − ν = 1+i . 49/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Suppose that the payments in a cashflow happen with certain

probability. We have

Probability that a payment is made p1 p2 ··· pn

payments C1 C2 ··· Cn
Time (in years) t1 t2 ··· tn

Then, the actuarial present value of the former cashflow is

n
X
Cj pj (1 + i)−tj .
j=1

50/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 5
An actuary models the life status of an individual with lung cancer
using a non–homogenous Markov chain model with states: State
1: life; and State 2: dead. The transition probability matrices are
     
0.6 0.4 0.4 0.6 0.2 0.8
Q0 = , Q1 = , Q2 = ,
0 1 0 1 0 1
 
0 1
Q3 =
0 1

Suppose that changes in state occur at the end of the year. A

death benefit of 100000 is paid at the end of the year of death.
The annual effective rate of interest is 5%. The insured is alive at
the beginning of year zero. Calculate the actuarial present value of
this life insurance.

51/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Solution: Since at the beginning the individual is alive,

α0 = (1, 0). We have that
 
0.6 0.4
α0 = α0 Q0 = (1, 0) = (0.6, 0.4),
0 1
 
0.4 0.6
α1 = α0 Q0 Q1 = (0.6, 0.4) = (0.24, 0.76),
0 1
 
0.2 0.8
α2 = α0 Q0 Q1 Q2 = (0.24, 0.76) = (0.048, 0.952),
0 1
 
0 1
α3 = α0 Q0 Q1 Q2 Q3 = (0.048, 0.952) = (0, 1).
0 1

The probability that an individual dies in the n–th year is

P{Xn = 2} − P{Xn−1 = 2}.

52/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Hence,
dead 1 2 3 4
happens
at the end
of year
Probab. 0.4 0.76 − 0.4 0.952 − 0.76 1 − 0.952
0.4 = 0.36 = 0.192 = 0.048

The actuarial present value of this life insurance is

100000(0.4)(1.05)−1 + 100000(0.36)(1.05)−2
+ 100000(0.192)(1.05)−3 + 100000(0.048)(1.05)−4
=91282.95309.

53/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Theorem 6
(i)

P{X1 = i1 , . . . , Xn−1 = in−1 , Xn = in |X0 = i0 }

=Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in ).

(ii)
(n)
X
Q0 (i0 , in ) = Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in ).
i1 ,...,in−1 ∈E

54/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Proof.
(i) Using Theorem 5, we have that

P{X1 = i1 , . . . , Xn−1 = in−1 , Xn = in |X0 = i0 }

P{X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = in }
=
P{X0 = i0 }
α0 (i0 )Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in )
=
α0 (i0 )
=Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in ).

(ii) Noticing that

{Xn = in } = ∪i1 ,...,in−1 ∈E {X1 = i1 , . . . , Xn−1 = in−1 , Xn = in },

where the union is over disjoint events, we get that

(n)
X
Q0 (i0 , in ) = Q0 (i0 , i1 )Q1 (i1 , i2 ) · · · Qn−1 (in−1 , in ).
i1 ,...,in−1 ∈E
55/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Using matrix notation, Theorem 6 (ii) states that

(n)
Q0 = Q0 Q1 · · · Qn−1 . (2)

This equation is one of the Kolmogorov–Chapman equations of

a Markov chain.
(2)
For example, Q0 = Q0 Q1 . Notice that

Q0 Q1
  
Q0 (0, 0) · · · Q0 (0, k) Q1 (0, 0) · · · Q1 (0, k)
 Q0 (1, 0) · · · Q0 (1, k)   Q1 (1, 0) · · · Q0 (1, k) 
=
 ···
 
··· ···  ··· ··· ··· 
Q0 (k, 0) · · · Q0 (k, k) Q1 (k, 0) · · · Q1 (k, k)
 Pk Pk 
i=0 Q0 (0, i)Q1 (i, 0) ... i=0 Q0 (0, i)Q0 (i, k))
= ··· ··· ··· 
Pk Pk
i=0 Q0 (k, i)(i)Q1 (i, 0) . . . i=0 αn (i)Q0 (k, k))
(2)
=Q0 .
56/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 6
Consider a Markov chain with E = {1, 2},
   
0.5 0.5 0.2 0.8
Q0 = , Q1 = .
0.3 0.7 0.6 0.4

Suppose that X0 = 1.

57/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 6
Consider a Markov chain with E = {1, 2},
   
0.5 0.5 0.2 0.8
Q0 = , Q1 = .
0.3 0.7 0.6 0.4

Suppose that X0 = 1.
(i) Find the probability that at stage 2 the chain is in state 2.

58/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 6
Consider a Markov chain with E = {1, 2},
   
0.5 0.5 0.2 0.8
Q0 = , Q1 = .
0.3 0.7 0.6 0.4

Suppose that X0 = 1.
(i) Find the probability that at stage 2 the chain is in state 2.
Solution: (i) The Markov chain can be at stage 2 in state 1, if any
of the following transitions occur

1 7→ 1 7→ 2, 1 7→ 2 7→ 2.

The probabilities of the previous occurrences are

(0.5)(0.8) = 0.4 and (0.5)(0.4) = 0.2.

The probability that at stage 2 the chain is in state 1 is 0.4 + 0.2 =

0.6. 59/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 6
Consider a Markov chain with E = {1, 2},
   
0.5 0.5 0.2 0.8
Q0 = , Q1 = .
0.3 0.7 0.6 0.4

Suppose that X0 = 1.
(i) Find the probability that at stage 2 the chain is in state 2.
(2)
Solution: (i) We need to find Q0 (1, 2) = P{X2 = 1|X0 = 1},
which is the element (1, 2) of the matrix:
    
0.5 0.5 0.2 0.8 0.4 0.6
Q0 Q 1 = =
0.3 0.7 0.6 0.4 0.48 0.52

The answer is 0.6.

60/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 6
Consider a Markov chain with E = {1, 2},
   
0.5 0.5 0.2 0.8
Q0 = , Q1 = .
0.3 0.7 0.6 0.4

Suppose that X0 = 1.
(ii) Find the probability that the first time the chain is in state 2 is
stage 2.

61/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 6
Consider a Markov chain with E = {1, 2},
   
0.5 0.5 0.2 0.8
Q0 = , Q1 = .
0.3 0.7 0.6 0.4

Suppose that X0 = 1.
(ii) Find the probability that the first time the chain is in state 2 is
stage 2.
Solution: (ii) If the first time the chain is in state 1 is stage 2, then
the Markov chain does 1 7→ 1 7→ 2, which happens with probability

P{X1 = 1, X2 = 2|X0 = 1} = Q0 (1, 1)Q1 (1, 2) = (0.5)(0.8) = 0.4

62/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Theorem 7
(i)

P{Xn = in , Xn+1 = in+1 , . . . , Xn = in+m }

=αn (in )Qn (in , in+1 )Qn+1 (in+1 , in+2 ) · · · Qn+m−1 (in+m−1 , in+m ).

(ii)

αn+m (in+m )
X
= αn (in )Qn (in , in+1 ) · · · Qn+m−1 (in+m−1 , in+m ).
in ,in+1 ,...,in+m−1 ∈E

63/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Proof: (i) Using Theorem 5,

P{Xn = in , Xn+1 = in+1 , . . . , Xn = in+m }

X
= P{X0 = i0 , . . . , Xn−1 = in−1 , Xn+1 = in+1 , . . . , Xn = in+m }
i0 ,i1 ,...,in−1
X
= α0 (i0 )Q0 (i0 , i1 ) · · · Qn−1 (in−1 , in )
i0 ,i1 ,...,in−1

× Qn (in , in+1 )Qn+1 (in+1 , in+2 ) · · · Qn+m−1 (in+m−1 , in+m )

=αn (in )Qn (in , in+1 )Qn+1 (in+1 , in+2 ) · · · Qn+m−1 (in+m−1 , in+m ).

(ii) follows noticing that

{Xn+m = in+m }
= ∪in ,in+1 ,...,in+m−1 ∈E {Xn = in , Xn+1 = in+1 , . . . , Xn+m = in+m },

where the union is over disjoint events.

64/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

In matrix notation

αn+m (in+m )
X
= αn (in )Qn (in , in+1 ) · · · Qn+m−1 (in+m−1 , in+m ).
in ,in+1 ,...,in+m−1 ∈E

says that
αn+m = αn Qn Qn+1 · · · Qn+m−1 .

65/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 7
Consider a Markov chain with E = {1, 2}, α3 = (0.2, 0.8),
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:

66/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 7
Consider a Markov chain with E = {1, 2}, α3 = (0.2, 0.8),
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(i) P{X3 = 2, X4 = 1}.

67/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 7
Consider a Markov chain with E = {1, 2}, α3 = (0.2, 0.8),
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(i) P{X3 = 2, X4 = 1}.
Solution: (i)

P{X3 = 2, X4 = 1} = α3 (2)Q3 (2, 1) = (0.8)(0.3) = 0.24.

68/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 7
Consider a Markov chain with E = {1, 2}, α3 = (0.2, 0.8),
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(ii) P{X3 = 1, X4 = 1, X5 = 2}.

69/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 7
Consider a Markov chain with E = {1, 2}, α3 = (0.2, 0.8),
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(ii) P{X3 = 1, X4 = 1, X5 = 2}.
Solution: (ii)

P{X3 = 1, X4 = 1, X5 = 2} = α3 (1)Q3 (1, 1)Q4 (1, 2)

=(0.2)(0.6)(0.8) = 0.096.

70/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 7
Consider a Markov chain with E = {1, 2}, α3 = (0.2, 0.8),
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(iii) P{X5 = 2}.

71/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 7
Consider a Markov chain with E = {1, 2}, α3 = (0.2, 0.8),
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(iii) P{X5 = 2}.
Solution: (iii) We have that
  
0.6 0.4 0.2 0.8
α5 = α3 Q3 Q4 = (0.2, 0.8)
0.3 0.7 0.7 0.3
 
0.2 0.8
=(0.36, 0.64) = (0.52, 0.48).
0.7 0.3

Hence, P{X5 = 2} = α5 (2) = 0.48.

72/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Next theorem shows that for a Markov chain, given the present,
the future is independent of the past, where future means events
involving Xn+1 , . . . , Xn+m , present means events involving Xn , and
past means events involving X0 , . . . , Xn−1 .
Theorem 8
Let {Xn : n = 0, 1, 2, . . . } be Markov chain such that for each
i0 , i1 , . . . , in , j1 , . . . , jm ∈ E ,

P{Xn+1 = j1 , Xn+2 = j2 , . . . , Xn+m = jm |X0 = i0 , . . . , Xn = in }

=Qn (in , in+1 )Qn+1 (in+1 , in+2 ) · · · Qn+m−1 (in+m−1 , in+m )
=P{Xn+1 = j1 , Xn+2 = j2 , . . . , Xn+m = jm |Xn = in }.

73/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Proof: By Lemma 1 and the definition of Markov chain, we have

that

P{Xn+1 = j1 , Xn+2 = j2 , . . . , Xn+m = jm |X0 = i0 , . . . , Xn = in }

=P{Xn+1 = j1 |X0 = i0 , X1 = i1 , . . . , Xn = in }
× P{Xn+2 = j2 |X0 = i0 , X1 = i1 , . . . , Xn = in , Xn+1 = j1 }
× ···
× P{Xn+m = jm |X0 = i0 , X1 = i1 , . . . , Xn+m−1 = jm−1 }
=P{Xn+1 = j1 |Xn = in }P{Xn+2 = j2 |Xn+1 = j1 } · · ·
× P{Xn+m = jm |Xn+m−1 = jm−1 }
=Qn (in , in+1 )Qn+1 (in+1 , in+2 ) · · · Qn+m−1 (in+m−1 , in+m ).

74/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

By Theorem 7,

P{Xn+1 = j1 , Xn+2 = j2 , . . . , Xn+m = jm |Xn = in }

P{Xn = in , Xn+1 = j1 , Xn+2 = j2 , . . . , Xn+m = jm }
=
P{Xn = in }
αn (in )Qn (in , in+1 )Qn+1 (in+1 , in+2 ) · · · Qn+m−1 (in+m−1 , in+m )
=
αn (in )
=Qn (in , in+1 )Qn+1 (in+1 , in+2 ) · · · Qn+m−1 (in+m−1 , in+m ).

75/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Previous theorem implies that:

Theorem 9

P{Xn+m = jm |Xn = in }
X
= Qn (in , j1 )Qn+1 (j1 , j2 ) · · · Qn+m−1 (jm−1 , jm ).
j1 ,...,jm−1 ∈E

Proof:

P{Xn+m = jm |Xn = in }
X
= P{Xn+1 = j1 , Xn+2 = j2 , . . . , Xn+m = jm |Xn = in }
j1 ,...,jm−1 ∈E
X
= Qn (in , j1 )Qn+1 (j1 , j2 ) · · · Qn+m−1 (jm−1 , jm ).
j1 ,...,jm−1 ∈E

76/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Previous theorem in matrix notation says that:

Theorem 10
(m)
Qn = Qn Qn+1 · · · Qn+m−1 .

Proof.
We have that
(m)
Qn (i, j) = P{Xn+m = j|Xn = i}
X
= Qn (i, j1 )Qn+1 (j1 , j2 ) · · · Qn+m−1 (jm−1 , j).
j1 ,...,jm−1 ∈E

(m)
So, Qn = Qn Qn+1 · · · Qn+m−1 .

77/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Equation
(m)
Qn = Qn Qn+1 · · · Qn+m−1
is one of the Kolmogorov–Chapman equations.

78/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 8
Consider a Markov chain with E = {1, 2},
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:

79/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 8
Consider a Markov chain with E = {1, 2},
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(i) P{X4 = 2|X3 = 1}.

80/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 8
Consider a Markov chain with E = {1, 2},
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(i) P{X4 = 2|X3 = 1}.
Solution: (i) P{X4 = 2|X3 = 1} = Q3 (1, 2) = 0.4.

81/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 8
Consider a Markov chain with E = {1, 2},
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(ii) P{X4 = 2, X5 = 1|X3 = 1}.

82/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 8
Consider a Markov chain with E = {1, 2},
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(ii) P{X4 = 2, X5 = 1|X3 = 1}.
Solution: (ii)

P{X4 = 2, X5 = 1|X3 = 1} = Q3 (1, 2)Q4 (2, 1) = (0.4)(0.7) = 0.28.

83/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 8
Consider a Markov chain with E = {1, 2},
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(iii) P{X5 = 1|X3 = 1}.

84/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 8
Consider a Markov chain with E = {1, 2},
   
0.6 0.4 0.2 0.8
Q3 = , Q4 = .
0.3 0.7 0.7 0.3

Find:
(iii) P{X5 = 1|X3 = 1}.
Solution: (iii) We have that
    
(2) 0.6 0.4 0.2 0.8 0.40 0.60
Q 3 = Q3 Q4 = =
0.3 0.7 0.7 0.3 0.55 0.45

and
(2)
P{X5 = 1|X3 = 1} = Q3 (1, 1) = 0.4.

85/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

For a Markov chain, given the present, the future is independent of

the past, where present means Xn , past means events involving
X0 , . . . , Xn−1 and future means events involving Xn+1 , . . . , Xn+m .
Theorem 11
Let {Xn : n = 0, 1, 2, . . . } be Markov chain. Then, for each A ∈ Rn
and B ∈ Rm

P{(Xn+1 , . . . , Xn+m ) ∈ B|(X0 , . . . , Xn−1 ) ∈ A, Xn = in }

=P{(Xn+1 , . . . , Xn+m ) ∈ B|Xn = in }

The proof of the previous theorem is in Arcones’ manual.

86/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 9
Consider a Markov chain with E = {1, 2},
   
0.4 0.6 0.3 0.7
Q7 = , Q8 = .
0.1 0.8 0.6 0.4

Find P{X9 = 1|X5 = 2, X7 = 1}.

87/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 9
Consider a Markov chain with E = {1, 2},
   
0.4 0.6 0.3 0.7
Q7 = , Q8 = .
0.1 0.8 0.6 0.4

Find P{X9 = 1|X5 = 2, X7 = 1}.

Solution: We have that
    
(2) 0.4 0.6 0.3 0.7 0.48 0.52
Q 7 = Q7 Q8 = =
0.1 0.8 0.6 0.4 0.51 0.39

and
(2)
P{X9 = 1|X5 = 2, X7 = 1} = P{X9 = 1|X7 = 1} = Q7 (1, 1) = 0.48.

88/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

A Markov chain satisfying that P(Xn+1 = j|Xn = i) is independent

of n is called an homogeneous Markov chain. We define
P(i, j) = P{Xn+1 = j|Xn = i}. P(i, j) is called the one-step
transition probability from state i into state j. Notice that P(i, j)
does not depend on n.
We denote P = (P(i, j))i,j∈E the matrix consisting of the one-step
transition probabilities. The matrix P must satisfy that
(i) for each i, j ∈ E ,PP(i, j) ≥ 0.
(ii) for each i ∈ E , j∈E P(i, j) = 1.

89/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

We define P (n) (i, j) = P{Xk+n = j|Xk = i}. P(i, j) is called the

n–step transition probability from state i into state j. Notice that
P(i, j) does not depend on k. We denote P (n) = (P (n) (i, j))i,j∈E
the matrix consisting of the n–step transition probabilities. We
have that P (1) = P. The matrix P (n) must satisfy that:
(i) for each i, j ∈ E ,PP (n) (i, j) ≥ 0.
(ii) for each i ∈ E , j∈E P (n) (i, j) = 1.

90/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Theorem 12
For an homogeneous Markov chain, we have
(i)

P{Xn = in , Xn+1 = in+1 , . . . , Xn+m−1 = in+m−1 , Xn+m = in+m }

=αn (in )P(in , in+1 )P(in+1 , in+2 ) · · · P(in+m−1 , in+m ).

(ii)

P{Xn+1 = in+1 , . . . , Xn+m−1 = in+m−1 , Xn+m = in+m |Xn = in }

=P(in , in+1 )P(in+1 , in+2 ) · · · P(in+m−1 , in+m ).

(iii)
P (n) = P n
(iv)
αm+n = αm P n
91/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

92/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(i) P (2) .

93/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(i) P (2) .
Solution: (i)
  
1/3 2/3 0 1/3 2/3 0
P (2) = 1/2 0 1/2 1/2 0 1/2
1/4 1/4 1/2 1/4 1/4 1/2
 
0.4444444 0.2222222 0.3333333
= 0.2916667 0.4583333 0.2500000
0.3333333 0.2916667 0.3750000 94/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(ii) P (3) .

95/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(ii) P (3) .
Solution: (ii)
  
1/3 2/3 0 0.4444444 0.2222222 0.3333333
P (3) = 1/2 0 1/2 0.2916667 0.4583333 0.2500000
1/4 1/4 1/2 0.3333333 0.2916667 0.3750000
 
0.3425926 0.3796296 0.2777778
= 0.3888889 0.2569444 0.3541667
0.3506944 0.3159722 0.3333333 96/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(iii) P{X2 = 2}.

97/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(iii) P{X2 = 2}.
Solution: (iii)
 
0.4444444 0.2222222 0.3333333
α0 P (2) = (1/2, 1/3, 1/6) 0.2916667 0.4583333 0.2500000
0.3333333 0.2916667 0.3750000
=(0.375, 0.3125, 0.3125).

So, P{X2 = 2} = 0.3125. 98/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(iv) P{X0 = 1, X3 = 3}.

99/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(iv) P{X0 = 1, X3 = 3}.
Solution: (iv)

P{X0 = 1, X3 = 3} = P{X0 = 1}P{X3 = 3|X0 = 1}

=α0 (1)P (3) (1, 3) = (0.5)(0.2777778) = 0.1388889.

100/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(v) P{X1 = 2, X2 = 3, X3 = 1|X0 = 1}.

101/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(v) P{X1 = 2, X2 = 3, X3 = 1|X0 = 1}.
Solution: (v)

P{X1 = 2, X2 = 3, X3 = 1|X0 = 1}
=P(1, 2)P(2, 3)P(3, 1) = (2/3)(1/2)(1/4) = 1/12.

102/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(vi) P{X2 = 3|X1 = 3}.

103/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(vi) P{X2 = 3|X1 = 3}.
Solution: (vi)

P{X2 = 3|X1 = 3} = P(3, 3) = 1/2.

104/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(vii) P{X12 = 1|X5 = 3, X10 = 1}.

105/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(vii) P{X12 = 1|X5 = 3, X10 = 1}.
Solution: (vii)

P{X12 = 1|X5 = 3, X10 = 1} = P{X (12) = 1|X (10) = 1}

=P (2) (1, 1) = 0.444444.

106/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(viii) P{X3 = 3, X5 = 1|X0 = 1}.

107/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(viii) P{X3 = 3, X5 = 1|X0 = 1}.
Solution: (viii)

P{X3 = 3, X5 = 1|X0 = 1} = P{X3 = 3|X0 = 1}P{X5 = 1|X3 = 3}

=P (3) (1, 3)P (2) (3, 1) = (0.277778)(0.3333) = 0.09258341.

108/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(ix) P{X3 = 3|X0 = 1}.

109/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.
 Chapter 10. Markov chains. Section 10.2. Markov chains.

Example 10
Suppose that an homogeneous Markov chain has state space
E = {1, 2, 3}, transition matrix
 
1/3 2/3 0
P = 1/2 0 1/2 ,
1/4 1/4 1/2

and initial distribution α0 = (1/2, 1/3, 1/6). Find:

(ix) P{X3 = 3|X0 = 1}.
Solution: (ix)

P{X3 = 3|X0 = 1} = P (3) (1, 3) = 0.277778.

110/110

2008.
c Miguel A. Arcones. All rights reserved. Manual for SOA Exam MLC.