1.

Steady state Distribution of a Markov Chain (with Markov Chain matrix properties)

1. A salesman’s territory consists of three cities A, B and C. He never sells in the same city on
successive days. If he sells in A, then the next day he sells in city B. However, if he sells in either B or
C, then the next day he is twice as likely to sell in city A as in the other city. In the long run, how often
does he sell in each of the cities?

A = [0, 1, 0; 2/3, 0, 1/3; 2/3, 1/3, 0];

A = A';
B = A^50

B = 3×3
0.4000 0.4000 0.4000
0.4500 0.4500 0.4500
0.1500 0.1500 0.1500

disp('In the long run, how often does he sell in each of the cities..')

In the long run, how often does he sell in each of the cities..

disp(B(:, 1));

0.4000
0.4500
0.1500

2.

A = [0, 1; 1/2, 1/2];

A = A';
B = A^50;
disp('Steady state distribution of chain')
disp(B(:, 1))

3. The three- state Markov chain is given by the transition probability matrix P= [ / /; / /; / / ]. Find the
steady state distribution of the chain

A = [0, 2/3, 1/3; 1/2, 0, 1/2; 1/2, 1/2, 0];

A = A';
B = A^50;
disp('Steady state distribution of chain')
disp(B(:, 1))

2. Generation of random state sequences

1
1. Three Boys A, B, C are throwing a ball each other. A always throws the ball to B and B always throws
the ball to C but C is just as likely to throw the ball to B as to A. Show the process is Markovian. Find
the transition probability matrix and generate 700 the state sequence. Also reestimate the parameters.

clc;clear all;

disp(['The person receiving the ball depends ONLY on the person ' ...
'holding the ball in the previous state or previously'])

The person receiving the ball depends ONLY on the person holding the ball in the previous state or previously

disp('Hence, Process is a markovian')

Hence, Process is a markovian

Find the transition probability matrix

T=zeros(3,3);
T(1,1)=0; T(1,2)=1; T(1,3)=0;
T(2,1)=0; T(2,2)=0; T(2,3)=1;
T(3,1)=0.5; T(3,2)=0.5; T(3,3)=0;

disp('Transition probability matrix')

Transition probability matrix

T = 3×3
0 1.0000 0
0 0 1.0000
0.5000 0.5000 0

disp('All Rows have a sum equal to 1')

All Rows have a sum equal to 1

disp('It is a stochastic matrix')

It is a stochastic matrix

generate 700 the state sequence

s(1)=randi(3,1); % randomly choose a state

n=700;

for i = 1:n-1
cs = s(i);
r = rand();
if r<T(cs, 1)
s(i+1)=1;
elseif r<T(cs,1)+T(cs,2)
s(i+1)=2;
else
s(i+1)=3;
end

2
 end

disp('Part of state sequence')

Part of state sequence

display(s(1:12))

Columns 1 through 4

3 2 3 2

Columns 5 through 8

3 1 2 3

Columns 9 through 12

2 3 1 2

Reestimate the parameters

% Counting transition
A=zeros(3,3);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end

disp('Estimated transition probabilities')

Estimated transition probabilities

rowsum=sum(A,2);
div=repmat(rowsum,[1,3]);
A=A./div;
disp(A)

Columns 1 through 2

0 1.0000
0 0
0.4823 0.5177

Column 3

0
1.0000
0

disp('Original transition probabilities')

Original transition probabilities

disp(T)

Columns 1 through 2

3
 0 1.0000
0 0
0.5000 0.5000

Column 3

0
1.0000
0

2. Consider the random experiment of rolling a die once. Generate 800 random state sequences for the
appearance on the die and reestimate the parameters. (you can determine the Markov chain on your
own).

clear all; clc;

% Transition probability matrix or Markov Chain
T = [3/15, 4/15, 2/15, 2/15, 1/15, 3/15;
2/15, 1/15, 3/15, 4/15, 3/15, 2/15;
2/15, 2/15, 3/15, 3/15, 3/15, 2/15;
3/15, 4/15, 2/15, 1/15, 1/15, 4/15;
1/15, 3/15, 2/15, 1/15, 4/15, 4/15;
3/15, 3/15, 5/15, 2/15, 1/15, 2/15];

Generate 800 random state sequences for the appearance on the die

s(1)=randi(6,1); % randomly choose a state

n=800;

for i = 1:n-1
cs = s(i); % current state
r = rand();
if r<T(cs, 1)
s(i+1)=1;
elseif r<T(cs,1)+T(cs,2)
s(i+1)=2;
elseif r<T(cs,1)+T(cs,2)+T(cs,3)
s(i+1)=3;
elseif r<T(cs,1)+T(cs,2)+T(cs,3)+T(cs,4)
s(i+1)=4;
elseif r<T(cs,1)+T(cs,2)+T(cs,3)+T(cs,4)+T(cs,5)
s(i+1)=5;
else
s(i+1)=6;
end
end

disp('Part of state sequence')

display(s(1:12))

Reestimate the parameters

% Counting transition
A=zeros(6,6);
n=length(s);
for i=1:n-1

4
 k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end

disp('Estimated transition probabilities')

rowsum=sum(A,2);
div=repmat(rowsum,[1,6]);
A=A./div;
disp(A)
disp('Original transition probabilities')
disp(T)

3. In the case of tossing a coin once, generate 2000 random state sequences for the appearance on
the die. Display any subset of the sequence with length 30 and reestimate its parameters. (you can
determine the Markov chain on your own).

% Transition probability matrix or Markov Chain

T = [0.3, 0.7;
0.6, 0.4];

generate 2000 random state sequences for the appearance on the die. Display any subset of the sequence with
length 30

s(1)=randi(2,1); % randomly choose a state

n=2000;

for i = 1:n-1
cs = s(i);
r = rand();
if r<T(cs, 1)
s(i+1)=1;
else
s(i+1)=2;
end
end

disp('Part of state sequence')

r = randi(1900, 1)
display(s(r+1:r+30))

Reestimate its parameters

% Counting transition
A=zeros(2,2);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end

disp('Estimated transition probabilities')

5
 rowsum=sum(A,2);
div=repmat(rowsum,[1,2]);
A=A./div;
disp(A)
disp('Original transition probabilities')
disp(T)

3. Generate Symbol and State sequences

1. Generate observation sequence and state sequence of the climate problem which we have discussed
in the class. Also re estimate the parameters.

%HMM data Generation

clear all;
T=[0.5, 0.3, 0.2; 0.4, 0.2, 0.4; 0, 0.3, 0.7]

T = 3×3
0.5000 0.3000 0.2000
0.4000 0.2000 0.4000
0 0.3000 0.7000

state_rainy.sad=0.9;
state_rainy.happy = 0.1;
state_cloudy.sad = 0.6;
state_cloudy.happy = 0.4;
state_sunny.sad = 0.2;
state_sunny.happy = 0.8;

x=rand();
if x<1/3
start_state=1;
elseif x<2/3
start_state=2;
else
start_state=3;
end
current_state=start_state;
num_of_observations = 200;
% s = zeros([1, num_of_samples])
s(1)=current_state;
Obs='';

for i=1:num_of_observations
x=rand();
if current_state==1
if x<state_rainy.sad
Obs(i)='S';
else
Obs(i)='H';
end
elseif current_state == 2
if x<state_cloudy.sad

6
 Obs(i)='S';
else
Obs(i)='H';
end
else
if x<state_sunny.sad
Obs(i)='S';
else
Obs(i)='H';
end
end
x=rand();
if x<T(current_state,1)
current_state=1;
elseif x<T(current_state,1)+T(current_state,2)
current_state=2;
else
current_state=3;
end
s(end+1)=current_state;
end
Obs

Obs =
'HHSSSSSHSSSHHHSHSHSSSHHHSHHHHSHHSSSSSHHSSHHHHHSSSSSSSHSSSSSSHHSSSHHSSSSHSHHHSHHSSHHHSHSHHSSHSSHHHHSHSHHSHSSSHHHHHSH

s=s(1:end-1);
% s=s(1:end-1);
state_rainy_computed.sad=0;
state_rainy_computed.happy=0;
state_cloudy_computed.sad=0;
state_cloudy_computed.happy=0;
state_sunny_computed.sad=0;
state_sunny_computed.happy=0;
for i=1:length(s)
e=Obs(i);
if and(s(i)==1, e=='S')
state_rainy_computed.sad = state_rainy_computed.sad+1;
elseif and(s(i)==1, e=='H')
state_rainy_computed.happy = state_rainy_computed.happy+1;
elseif and(s(i)==2, e=='S')
state_cloudy_computed.sad = state_cloudy_computed.sad+1;
elseif and(s(i)==2, e=='H')
state_cloudy_computed.happy = state_cloudy_computed.happy+1;
elseif and(s(i)==3, e=='S')
state_sunny_computed.sad = state_sunny_computed.sad+1;
else
state_sunny_computed.happy = state_sunny_computed.happy+1;
end
end
total=state_rainy_computed.sad+state_rainy_computed.happy+...
state_cloudy_computed.sad+state_cloudy_computed.happy+...
state_sunny_computed.sad+state_sunny_computed.happy;

7
state_rainy_computed_sad = state_rainy_computed.sad/(state_rainy_computed.happy+state_rainy_com

state_rainy_computed_sad = 0.8333

state_rainy_computed_happy = state_rainy_computed.happy/(state_rainy_computed.happy+state_rainy

state_rainy_computed_happy = 0.1667

state_cloudy_computed_sad = state_cloudy_computed.sad/(state_cloudy_computed.happy+state_cloudy

state_cloudy_computed_sad = 0.6591

state_cloudy_computed_happy = state_cloudy_computed.happy/(state_cloudy_computed.happy+state_cl

state_cloudy_computed_happy = 0.3409

state_sunny_computed_sad = state_sunny_computed.sad/(state_sunny_computed.happy+state_sunny_com

state_sunny_computed_sad = 0.1944

state_sunny_computed_happy = state_sunny_computed.happy/(state_sunny_computed.happy+state_sunny

state_sunny_computed_happy = 0.8056

%% Reestimation of the parameters

% Counting transition
A=zeros(3,3);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end
% Estimated transition probabilities
rowsum=sum(A,2);
div=repmat(rowsum,[1,3]);
A=A./div;
disp('Estimated transition probabilities')

Estimated transition probabilities

disp(A)

Columns 1 through 2

0.5417 0.2708
0.5000 0.2045
0 0.2056

Column 3

0.1875
0.2955
0.7944

8
2. Generate observation sequence and state sequence of the climate problem (which I already sent you
in teams). Also re estimate the parameters.

3. Generate 5 observation of the model which we have discussed today’s class.

clear all; clc;

%HMM data Generation
T=[0.7, 0.3; 0.2, 0.8];

state1.head = 0.8;
state1.tail = 0.2;
state2.head = 0.3;
state2.tail = 0.7;

x=rand();
if x<0.5
start_state = 1;
else
start_state = 2;
end
current_state = start_state;
num_of_observations = 5;
% s = zeros([1, num_of_samples])
s(1)=current_state;
Obs='';

for i=1:num_of_observations
x=rand();
if current_state==1
if x < state1.head
Obs(i)='H';
else
Obs(i)='T';
end
else
if x < state2.head
Obs(i)='H';
else
Obs(i)='T';
end
end
x=rand();
if x<T(current_state,1)
current_state=1;

9
 else
current_state=2;
end
s(end+1)=current_state;
end
Obs

Obs =
'HTTTT'

s=s(1:end-1)

s = 1×5
2 2 2 2 2

4. Generate 6 observation of the model which we have discussed today’s class. Also re estimate the
parameters.

clear all; clc;

%HMM data Generation
T=[0.7, 0.3; 0.2, 0.8]

T = 2×2
0.7000 0.3000
0.2000 0.8000

state1.head = 0.8;
state1.tail = 0.2

state1 = struct with fields:

head: 0.8000
tail: 0.2000

state2.head = 0.3;
state2.tail = 0.7

state2 = struct with fields:

head: 0.3000
tail: 0.7000

x=rand();
if x<0.5
start_state = 1;
else
start_state = 2;
end
current_state = start_state;
num_of_observations = 6;
% s = zeros([1, num_of_samples])
s(1)=current_state;
Obs='';

for i=1:num_of_observations

10
 x=rand();
if current_state==1
if x < state1.head
Obs(i)='H';
else
Obs(i)='T';
end
else
if x < state2.head
Obs(i)='H';
else
Obs(i)='T';
end
end
x=rand();
if x<T(current_state,1)
current_state=1;
else
current_state=2;
end
s(end+1)=current_state;
end
Obs

Obs =
'HHTHTT'

s=s(1:end-1);
% s=s(1:end-1);
state1_computed.head = 0;
state1_computed.tail = 0;
state2_computed.head = 0;
state2_computed.tail = 0;

for i=1:length(s)
e=Obs(i);
if and(s(i)==1, e=='H')
state1_computed.head = state1_computed.head+1;
elseif and(s(i)==1, e=='T')
state1_computed.tail = state1_computed.tail+1;
elseif and(s(i)==2, e=='H')
state2_computed.head = state2_computed.head+1;
else
state2_computed.tail = state2_computed.tail+1;
end
end
total=state1_computed.head+state1_computed.tail+...
state2_computed.head+state2_computed.tail;

state1_computed_head = state1_computed.head/(state1_computed.head+state1_computed.tail)

state1_computed_head = 0.7500

state1_computed_tail = state1_computed.tail/(state1_computed.head+state1_computed.tail)

11
 state1_computed_tail = 0.2500

state2_computed_head = state2_computed.head/(state2_computed.head+state2_computed.head)

state2_computed_head = NaN

state2_computed_tail = state2_computed.tail/(state2_computed.head+state2_computed.tail)

state2_computed_tail = 1

%% Reestimation of the parameters

% Counting transition
A=zeros(2,2);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end
% Estimated transition probabilities
rowsum=sum(A,2);
div=repmat(rowsum,[1,2]);
A=A./div;
disp('Estimated transition probabilities')

Estimated transition probabilities

disp(A)

0.7500 0.2500
0 1.0000

5. Generate all possible state sequence and display nth row of it.(n is the last two digit of your
registration number)

n=64;
Seq=[];
for i=0:n-1
x=dec2base(i,2,6);
x= strrep( x , '0' , '2' );
Seq=[Seq;x];
end
Seq(45, :)

ans =
'121122'

4) Finding the probability of Symbol sequence, State sequence

1. Generate transition probability matrix of brand(soap) switching problem and find its joint probability.
Also determine P(sym_sequence,state_sequence).

12
disp('Transition Probability Matrix')

Transition Probability Matrix

T = [0.8, 0.1, 0.1;

0.2, 0.6, 0.2;
0.1, 0.2, 0.7]

T = 3×3
0.8000 0.1000 0.1000
0.2000 0.6000 0.2000
0.1000 0.2000 0.7000

% Find P(sym_sequence,state_sequence)
% It is joint probability
% Let us take T's elements as [ T11 T12 ; T21 T22]
% actual transition prob is transpose of T
T=[0.5 0.5; 0.4 0.6];
state1.h=0.7;state1.t=0.3;state2.h=0.6;
state2.t=0.4;
x='HTTTH';
n=length(x); % length of symbol sequence
N=2^n;
Seq=[];
for i=0:N-1
x=dec2base(i,2,5);
x= strrep( x , '0' , '2' );
Seq=[Seq;x];
end
Seq;
% Get sym_sequence probabiity
prodx=1;
stat_seq=Seq(10,:); % take 10th row
for i=1:length(x)
if stat_seq(i)==1
if x(i)=='H'
prodx=prodx*state1.h;
else
prodx=prodx*state1.t;
end
else
if x(i)=='H'
prodx=prodx*state2.h;
else
prodx=prodx*state2.t;
end
end
end
% Get trans_sequence probabiity
prods=1;
for i=1:n-1

13
 j=str2num(stat_seq(i));
k=str2num(stat_seq(i+1));
prods=prods*T(j,k);
end
reqd_prob=vpa(prodx*prods)

reqd_prob =

2. P(x = HTTTH, = 12121)

3. Generate x with length 128 and Find P(X = row corresponds to your last two digit of your reg.
number | Pi = )

14