J . SGI. HIROSHIMA UNI V. SER.

A I
28 (1964), 8993
Equilibrium in a Stochastic n Person Game
A. M. Fink
(Received November 20, 1963)
Heuristically, a stochastic game is described by a sequence of states
which are determined stochastically. The stochastic element arises from a
set of transition probability measures. The determination of the particular
transition probability measure to be used at a move of the game is controlled
in part by each of the n players and it is this determination scheme which
gives rise to the strategies.
One might consider the following economics problem. We have n firms
competing for a market. Each will make a strategy decision periodically.
During each period the n firms play an ?zperson game. Across the infinite
horizon then we have a sequence of games. The economic situation behaves
in such a way that the game played in a given period depends on the game
played in the previous period and the strategies used in this period. The
dependence is not deterministic but is stochastic. A player's strategy will
reflect a concern both for the game now being played and for the situation
that will be probably confronted in the next period. The relative strength
of these two concerns will be in a geometric ratio, the socalled discounting
over the infinite horizon. We shall call the outcome of each game a cost to
each player. Negative cost is thus a gain.
More precisely we are considering a game which is described by a finite
set of states / . A play is a sequence of states {/}=
0
> &'
6
1 the game is
in the state /, each player h may choose an alternative j
h
eJ
h
( ). This choice
is made with knowledge of the state . Once each player has made his choice,
the game proceeds to the state fe with probability PH^.J^ Here F

y
1
...
;w
^>0
and ^Pij
r
..j
n
k = 1. The cost to player h of being in state and having the
vector ; = (/ , , /

) chosen is C
h j
. Each player furthermore chooses to
discount his projected cost by a factor

, where 0 <

< l . Thus if a
sequence of states {
n
} and alternative choices {]
n
} have been made, the cost
to player h is given by
(1)
The analysis of this game is simplified by a slight change in the outlook
and notation. Let g(h, ) be the cost to player h given that the game started
at the state . Then equation (1) can be rewritten
90 A. M. FINK
(2) g(h, o) = C
hio
j
Q
+ a
h
g(h, ).
Since is not strictly determined, we are interested in the expected value of
g(k, ). If the selection of the vector ;
0
is a function of the state &
5
i.e. a pure
strategy, one would get for every such strategy a relation
Of course, we may expect that the players will not use pure strategies.
Mixed strategies thus must be introduced. For every state of the game,
player h will give a probability distribution x
h
() on the alternative set J
h
(\
i.e. a; (0 = (&(0> > *m(0) where Xj()~>0, * j() = l and m is the cardinality
3
of J
h
(). If such a set of probability distributions x is given for every alter
native set, then letting e
hi
(x)=E\^g(h, OJ in (3) we have
(4) e
hi
(x) =
J w &
= 1 , , 7Z .
LEMMA 1. F or even/ ;= {^
m
(Ol^
w
G) is probability vector}, there exist
numbers e
hi
(x) satisfying relation (4). This set furthermore is unique.
Proof: The relation (4) for fixed h is a linear system whose coefficient
matrix A has the following properties.
a
k
= * *7, . ( )P, *, ( fc).
J m
By Hadamard's theorem all eigenvalues of A have absolute value at least
| f* I = l tf*> 0. Thus zero is not an eigenvalue of A and the linear
k+i
system has a unique solution.
Now each player h will use a strategy that tends to minimize his expected
cost. That is, if v
hi
mine
z
(X), then from (4) we get
(5) V
M
= min ^COCC o H tf

l]P
o
^], i = 1, ..., .
x
h
U~) ~J m k
h = 1, , n.
We will say that the vector x is an equilibrium point if and only if no player
can improve his cost by changing his strategy, i.e., if relation (5) is considered
Equilibrium in a Stochastic ^Person Game 91
for every player h and every state , the minimization of the right hand side
leads back to the same vector x. We will show t hat such equilibrium points
do in fact exist.
Let X^ {(VOL), , ^( ) , *
2
(1), ..., J( \ ..., *(!), ..., *"(<r)); **(0 is a
probability distribution on / * ()} , with euclidean metric and
R={(vn
9
, t7
n
); Vij real}, d(u, )=max|M/ y ;/y|. X is compact and R is
*./
complete. In the proofs t hat follow one notes t hat one can confine the
discussion to any closed convex subset of X and results are not altered. Thus
we simultaneously establish the existence of equilibrium points in constrained
games.
To ease the notational bulk we introduce the vector function / . Define
(6) / (*, y, I,)* =
J m=f=h k
where x e X and y\) is a probability vector. Thus (5) can be written
(7) v
hi
= mi n / 0, y, )
M
.
y
h
(.D
We note t hat f(x
9
y, v) has t he following properties :
(a) f(x, y, v) is continuous
(b) f(x, y, v)
hj
f(x, y, u)
hj
< a max \V
M
U
M
\, a = max a
h
k,l h
(c) f(x, y, v) is linear in y.
Let x 6 X and define a mapping T
x
of R into itself by the equation
(8) ( ,fO

y = min/ (*,
y
, fO

y
y
h
D
THEOREM 1 : For every x e X, T
x
is a contraction mapping of R.
Proof : Let u and be two elements of R and let x be a fixed element of
X. Let T
x
u=f(x,y,u) and T
x
v = f(x, z, )
9
then T
x
u<f(x,z,u) and 7 > <
/ (*,y,O Thus
(T
x
u)
h
j (T
x
v)
hj
<f(x, z, u)
hj
f( , z, v)
h
j<ad(v, u) by (b) and
(T
x
v)
hj
(T
x
u)
h
j<f(x, y, v)
h
j f(x, y, u)
h
j<ad(v, u).
Hence
x
u, T
x
v) = max | (T
x
v)
hj
(T
x
u)
hj
\ <ad(v, u).
Corollary 1. For every % e X, T
x
has a unique fixed point v.
Corollary 2. The set {T
x
\xtX} is equicontinuous.
Let x 6 X and define mappings and /? by
92 A. M. FINK
(9) 0*0 = {v I = min f(x, y, v)
y
(10) ( )
We note t hat (x) is a single valued function by Corollary 1. By (c) it is
clear t hat (x) is convex and closed for every xtX.
LEMMA 2. The range of (x) is bounded.
Proof. By Theorem 1 the sequence v
0
= Q, v
n+
= T
x
v
n
converges to (x).
Furthermore, we have d(v
m9
v
m
)<ad(v
m
_, v
m
2)< <oc
m
'
l
d(vi, VQ) so that
d(v
n9
*^d(v
l9
vo). Thus d( ( \ 0) < d(T
x
0
9
0) ^

1
max | ( , 0)
|
| <
J. OC 1 O J. O h,i
max|C
/
v| . Hence (x) is bounded as x takes on all values in X.
1 O hi j
Define S
v
(x) T
x
v. S
v
is a mapping of X into R.
LEMMAS. S
v
is continuous on X. Furthermore, {S
v
\v is bounded} is
equicontinuous.
Proof: Let S

(x) = T
x
v = f(x, 7, v)<Lf(x, 2, v) and S
v
(x')=T
x
,v = f(x', z, }<
f(x
f
, y, v}', then S
v
(x'} S
v
(x)<f(oc', y, ) f(x, y, v) and S
v
(x) S
v
(x'^f(x, z, v)
f(x
f
, z
9
v).
If v is restrained to be in a bounded region, then the right hand sides
can be made uniformly small because of the uniform continuity of / on
compact sets.
LEMMA 4. / / x
n
+x and (x^~^v^ then (x) = v
Q
.
Proof: d(
9
T
x
vo)<d(v
0
, (x
n
}} + d( (x
n
\ T
x
(x
n
}} + d(T
x
(x
n
\ T
x
v
0
) =
d(v
Q9
( n)) + d(S
(Xn}
(x
n
\ S
(Xn)
(x)) + d(T
x
(x
n
\ T
x
v
0
) *Q as ?i>oo because (x
n
^v^
and { (
n
)} is bounded by Lemma 2 so that Lemma 3 applies to the second
term.
LEMMAS. / / x
n
^x, j
n
~^j and y
n
(x
n
\ then y e (x).
Proof: By taking subsequences we can consider # GO >t?
0
. By Lemma 4
(x) = v
0
. Now d(f(x
9
y, v
0
\ vo)<d(f(x, y, V
Q
\ f(x
n
, y
n
> /?(*))) + <* (/ (*, Jn, (*n)\
VQ) = d(f(x
9
y, V
Q
\ f(x
n9
y
n9
/?GO)) + d( (x
n
\ VQ) + 0 as n > oo . Thus v
0
=f(x
9
y, V
Q
\
and by Lemma 4, v
0
is the fixed point so t hat f(x, y, VQ) =V
Q
= min/ (# , 2, V
Q
\
thus y (x).
THEOREM 2. There exist x 6 X, v e R such that v = f(x
9
x, v) = mmf(x, , v).
Equilibrium in a Stochastic rc Person Game 93
i.e. x 6 (x).
Proof: This is a consequence of the Kakutani fixed point theorem, i.e.
the mapping takes points into convex closed sets, by Lemma 5 the set
\J (x) is closed, hence sequentially compact and by Lemma 5 it is an upper
X
semi continuous set function.
I t is also interesting to note t hat /?(#) is continuous (Lemmas 2 and 4)
and thus its range is compact and connected. In case the set of states / is
denumerable, one replaces R by the space of bounded sequences and requires
t hat I Chj i < M for some M, and the results are still valid for X the appropriate
cartesian product of probability spaces. One also notes t hat Theorem 1 and
its corollaries are valid if the cardinality of J
h
() is arbitrary and min is
replaced by inf. In this case the techniques of this paper yield effective
strategies.
If n = 2 the results of this paper are those given by Shapely [2 \ and for
n = l the problem is a well known dynamic program problem. See Takahashi
[ J] for a different generalization of Shapely's results.
References
1. S. Kakut ani, A Generalisation of Brouwefs Fixed Point Theorem, Duke Math. Jour n al, vol. 8 (1941),
pp. 457 459.
2. L. S. Shapely, Stochastic games, Proc. Nat. Acad. Sc., vol. 39 (1953), pp. 10951100.
3. M. Takahashi, Stochastic Games with Infinitely Many Strategies, Jour n al of Science of the Hiroshima
University, Series A I, vol. 26 (1963), pp. 123134.
University of Nebraska