6.

254 : Game Theory with Engineering Applications

Lecture 15: Repeated Games

Asu Ozdaglar

MIT

April 1, 2010

Game Theory: Lecture 15

Introduction

Outline

Repeated Games (perfect monitoring)

The problem of cooperation
Finitely-repeated prisoners dilemma
Innitely-repeated games and cooperation

Folk Theorems

Reference:
Fudenberg and Tirole, Section 5.1.

Game Theory: Lecture 15

Introduction

Prisoners Dilemma
How to sustain cooperation in the society?
Recall the prisoners dilemma, which is the canonical game for
understanding incentives for defecting instead of cooperating.

Cooperate
Defect

Cooperate
1, 1
2, 1

Defect
1, 2
0, 0

Recall that the strategy prole (D, D ) is the unique NE. In fact, D
strictly dominates C and thus (D, D ) is the dominant equilibrium.
In society, we have many situations of this form, but we often observe
some amount of cooperation.
Why?

Game Theory: Lecture 15

Introduction

Repeated Games
In many strategic situations, players interact repeatedly over time.
Perhaps repetition of the same game might foster cooperation.
By repeated games, we refer to a situation in which the same stage game
(strategic form game) is played at each date for some duration of T periods.
Such games are also sometimes called supergames.
We will assume that overall payo is the sum of discounted payos at each
stage.
Future payos are discounted and are thus less valuable (e.g., money
and the future is less valuable than money now because of positive
interest rates; consumption in the future is less valuable than
consumption now because of time preference).
We will see in this lecture how repeated play of the same strategic game
introduces new (desirable) equilibria by allowing players to condition their
actions on the way their opponents played in the previous periods.
4

Game Theory: Lecture 15

Introduction

Discounting
We will model time preferences by assuming that future payos are
discounted proportionately (exponentially) at some rate [0, 1),
called the discount rate.
For example, in a two-period game with stage payos given by u 1 and
u 2 , overall payos will be
U = u 1 + u 2 .
With the interest rate interpretation, we would have
=

1
,
1+r

where r is the interest rate.

Game Theory: Lecture 15

Introduction

Mathematical Model
More formally, imagine that I players playing a strategic form game
G = I , (Ai )i I , (gi )i I for T periods.
At each period, the outcomes of all past periods are observed by all players
perfect monitoring
Let us start with the case in which T is nite, but we will be particularly
interested in the case in which T = .
Here Ai denotes the set of actions at each stage, and
gi : A R,
where A = A1 AI .

t
That is, gi ait , a
is the stage payo to player i when action prole
i

t t
t
a = ai , ai is played.

Game Theory: Lecture 15

Introduction

Mathematical Model (continued)

T

We use the notation a = {at }t =0 to denote the sequence of action proles.

T
We use the notation = {t }t =0 to be the prole of mixed strategies.
The payo to player i in the repeated game
T

ui (a) =

t gi (ait , at i )

t =0

where [0, 1).

We denote the T -period repeated game with discount factor by G T ().

Game Theory: Lecture 15

Introduction

Finitely-Repeated Prisoners Dilemma

Recall
Cooperate
Defect

Cooperate
1, 1
2, 1

Defect
1, 2
0, 0

What happens if this game was played T < times?

We rst need to decide what the equilibrium notion is. Natural
choice, subgame perfect Nash equilibrium (SPE).
Recall: SPE backward induction.
Therefore, start in the last period, at time T . What will happen?

Game Theory: Lecture 15

Introduction

Finitely-Repeated Prisoners Dilemma (continued)

In the last period,defect is a dominant strategy regardless of the
history of the game. So the subgame starting at T has a dominant
strategy equilibrium: (D, D ).
Then move to stage T 1. By backward induction, we know that at

T , no matter what, the play will be (D, D ). Then given this, the

subgame starting at T 1 (again regardless of history) also has a

dominant strategy equilibrium.

With this argument, we have that there exists a unique SPE: (D, D )

at each date.

In fact, this is a special case of a more general result.

Game Theory: Lecture 15

Introduction

Equilibria of Finitely-Repeated Games

Theorem
Consider repeated game G T () for T < . Suppose that the stage game
G has a unique pure strategy equilibrium a . Then G T has a unique SPE.
In this unique SPE, at = a for each t = 0, 1, ..., T regardless of history.
Proof: The proof has exactly the same logic as the prisoners dilemma
example. By backward induction, at date T , we will have that (regardless
of history) aT = a . Given this, then we have aT 1 = a , and continuing
inductively, at = a for each t = 0, 1, ..., T regardless of history.

10

Game Theory: Lecture 15

Innitely-Repeated Games

Innitely-Repeated Games
Now consider the innitely-repeated game G , i.e., players play the
game repeatedly at times t = 0, 1, . . ..

The notation a = {at }t=0 now denotes the (innite) sequence of

action proles.
A period-t history is ht = {a0 , . . . , at 1 } (action proles at all
periods before t), and the set of all period-t histories is H t .
A pure strategy for player i is si = {sit }, where sit : H t Ai
The payo to player i for the entire repeated game is then

ui (a) = (1 )

t gi (ait , at i )

t =0

where, again, [0, 1).

Note: this summation is well dened because < 1.

The term (1 ) is introduced as a normalization, to measure stage

and repeated game payos in the same units.

The normalized payo of having a utility of 1 per stage is 1.

11

Game Theory: Lecture 15

Innitely-Repeated Games

Trigger Strategies
In innitely-repeated games we can consider trigger strategies.
A trigger strategy essentially threatens other players with a worse,
punishment, action if they deviate from an implicitly agreed action prole.
A non-forgiving trigger strategy (or grim trigger strategy) s would involve
this punishment forever after a single deviation.
A non-forgiving trigger strategy (for player i) takes the following form:

a i
if a =
a for all < t
ait =
ai if a = a for some < t
Here a is the implicitly agreed action prole and ai is the punishment action.
This strategy is non-forgiving since a single deviation from a induces player i
to switch to ai forever.

12

Game Theory: Lecture 15

Innitely-Repeated Games

Cooperation with Trigger Strategies in the Repeated

Prisoners Dilemma
Recall
Cooperate
Defect

Cooperate
1, 1
2, 1

Defect
1, 2
0, 0

Suppose this game is played innitely often.

Is Both defect in every period still an SPE outcome?
Suppose both players use the following non-forgiving trigger strategy s :
Play C in every period unless someone has ever played D in the past
Play D forever if someone has played D in the past.
We next show that the preceding strategy is an SPE if 1/2 using
one-stage deviation principle.
13

Game Theory: Lecture 15

Innitely-Repeated Games

Cooperation with Trigger Strategies in the Repeated

Prisoners Dilemma
Step 1: cooperation is best response to cooperation.
Suppose that there has so far been no D. Then given s being played
by the other player, the payos to cooperation and defection are:
1 =1
Payo from C : (1 )[1 + + 2 + ] = (1 ) 1

Payo from D : (1 )[2 + 0 + 0 + ] = 2(1 )

Cooperation better if 2(1 ) 1.

This shows that for 1/2, deviation to defection is not protable.

14

Game Theory: Lecture 15

Innitely-Repeated Games

Cooperation with Trigger Strategies in the Repeated

Prisoners Dilemma (continued)
Step 2: defection is best response to defection.
Suppose that there has been some D in the past, then according to s ,
the other player will always play D. Against this, D is a best response.

This argument is true in every subgame, so s is a subgame perfect

equilibrium.
Note: Cooperating in every period would be a best response for a
player against s . But unless that player herself also plays s , her
opponent would not cooperate. Thus SPE requires both players to
use s .

15

Game Theory: Lecture 15

Innitely-Repeated Games

Remarks

Cooperation is an equilibrium, but so are many other strategy proles

(depending on the size of the discount factor)

Multiplicity of equilibria endemic in repeated games.

If a is the NE of the stage game (i.e., it is a static equilibrium), then the
strategies each player, plays ai form an SPE.
Note that with these strategies, future play of the opponent is
independent of how I play today, therefore, the optimal play is to
maximize the current payo, i.e., play a static best response.)
Sets of equilibria for nite and innite horizon versions of the same game
can be quite dierent.
Multiplicity of equilibria in prisoners dilemma only occurs at T = .
In particular, for any nite T (and thus by implication for T ),
prisoners dilemma has a unique SPE.
Why? The set of Nash equilibria is an upper semicontinuous
correspondence in parameters. It is not necessarily lower
semicontinuous.
16

Game Theory: Lecture 15

Innitely-Repeated Games

Repetition Can Lead to Bad Outcomes

The following example shows that repeated play can lead to worse outcomes
than in the one shot game:

A
B
C

A
2, 2
1, 2
0, 0

B
2, 1
1, 1
0, 1

C
0, 0
1, 0
1, 1

For the game dened above, the action A strictly dominates B, C for both
players, therefore the unique Nash equilibrium of the stage game is (A, A).
If 1/2, this game has an SPE in which (B, B ) is played in every period.
It is supported by a slightly more complicated strategy than grim trigger:
I. Play B in every period unless someone deviates, then go to II.
II. Play C . If no one deviates go to I. If someone deviates stay in II.

17

Game Theory: Lecture 15

Folk Theorems

Folk Theorems

In fact, it has long been a folk theorem that one can support
cooperation in repeated prisoners dilemma, and other
non-one-stageequilibrium outcomes in innitely-repeated games
with suciently high discount factors.
These results are referred to as folk theorems since they were
believed to be true before they were formally proved.
Here we will see a relatively strong version of these folk theorems.

18

Game Theory: Lecture 15

Folk Theorems

Feasible Payos
Consider stage game G = I , (Ai )i I , (gi )i I and innitely-repeated

game G ().

Let us introduce the set of feasible payos:

V = Conv{v RI | there exists a A such that g (a) = v }.

That is, V is the convex hull of all I - dimensional vectors that can be
obtained by some action prole. Convexity here is obtained by public
randomization.
Note: V is not equal to {v RI | there exists such that
g () = v }, where is the set of mixed strategy proles in the stage
game.

19

Game Theory: Lecture 15

Folk Theorems

Minmax Payos
Minmax payo of player i: the lowest payo that player is opponent
can hold him to:

v i = min max gi (i , i ) .
i

The player can never receive less than this amount.

Minmax strategy prole against i:

i
=
arg
min
max
g
(

)
m
i
i
i
i
i

Finally, let mii denote the strategy of player i such that

i ) =v .
gi (mii , m
i
i

20

Game Theory: Lecture 15

Folk Theorems

Example
Consider
U
M
D

L
2, 2
1, 2
0, 1

R
1, 2
2, 2
0, 1

To compute v 1 , let q denote the probability that player 2 chooses

action L.
Then player 1s payos for playing dierent actions are given by:
U 1 3q
M 2 + 3q
D0

21

Game Theory: Lecture 15

Folk Theorems

Example
Therefore, we have

v 1 = min

0q 1

[max{1 3q, 2 + 3q, 0}] = 0,

and m21 [ 13 , 23 ].

Similarly, one can show that: v 2 = 0, and m12 = (1/2, 1/2, 0) is the
unique minimax prole.

22

Game Theory: Lecture 15

Folk Theorems

Minmax Payo Lower Bounds

Theorem
1

Let be a (possibly mixed) Nash equilibrium of G and gi () be the

payo to player i in equilibrium . Then

gi () v i .

Let be a (possibly mixed) Nash equilibrium of G () and ui ( )

be the payo to player i in equilibrium . Then

ui ( ) v i .

Proof: Player i can always guarantee herself

v i = minai [maxai ui (ai , ai )] in the stage game and also in each stage
of the repeated game, since v i = maxai [minai ui (ai , ai )], meaning that
she can always achieve at least this payo against even the most
adversarial strategies.
23

Game Theory: Lecture 15

Folk Theorems

Folk Theorems

Denition
A payo vector v RI is strictly individually rational if vi > v i for all i.
Theorem
(Nash Folk Theorem) If (v1 , . . . , vI ) is feasible and strictly individually
rational, then there exists some < 1 such that for all > , there is a
Nash equilibrium of G () with payos (v1 , , vI ).

24

Game Theory: Lecture 15

Folk Theorems

Proof
Suppose for simplicity that there exists an action prole
a = (a1 , , aI ) s.t. gi (a) = vi [otherwise, we have to consider
mixed strategies, which is a little more involved].
i these the minimax strategy of opponents of i and mi be is
Let m
i
i
i .
best response to m
i

Now consider the following grim trigger strategy.

For player i: Play (a1 , , aI ) as long as no one deviates. If some
player j deviates, then play mij thereafter.
We next check if player i can gain by deviating form this strategy
prole. If i plays the strategy, his payo is vi .

25

Game Theory: Lecture 15

Folk Theorems

Proof (continued)
If i deviates from the strategy in some period t, then denoting
vi = maxa gi (a), the most that player i could get is given by:

(1 ) vi + vi + + t 1 vi + t v i + t +1 v i + t +2 v i + .
Hence, following the suggested strategy will be optimal if
vi
1 t
t +1

vi + t v i +
v ,

1
1
1 i
thus if

vi

1 t vi + t (1 ) v i + t +1 v i

= vi t [vi (1 )v i v i + (vi vi )].

The expression in the bracket is non-negative for any
v i vi
max
.
i vi vi
This completes the proof.
26

Game Theory: Lecture 15

Folk Theorems

Problems with Nash Folk Theorem

The Nash folk theorem states that essentially any payo can be
obtained as a Nash Equilibrium when players are patient enough.
However, the corresponding strategies involve this non-forgiving
punishments, which may be very costly for the punisher to carry out
(i.e., they represent non-credible threats).
This implies that the strategies used may not be subgame perfect.
The next example illustrates this fact.
U
D

L (q)
6, 6
7, 1

R (1 q)
0, 100
0, 100

The unique NE in this game is (D, L). It can also be seen that the
minmax payos are given by
v 1 = 0,

v 2 = 1,

and the minmax strategy prole of player 2 is to play R.

27

Game Theory: Lecture 15

Folk Theorems

Problems with the Nash Folk Theorem (continued)

Nash Folk Theorem says that (6,6) is possible as a Nash equilibrium
payo of the repeated game, but the strategies suggested in the proof
require player 2 to play R in every period following a deviation.
While this will hurt player 1, it will hurt player 2 a lot, it seems
unreasonable to expect her to carry out the threat.
Our next step is to get the payo (6, 6) in the above example, or
more generally, the set of feasible and strictly individually rational
payos as subgame perfect equilibria payos of the repeated game.

28

MIT OpenCourseWare
http://ocw.mit.edu

6.254 Game Theory with Engineering Applications

Spring 2010

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.