Reinforcement Learning and Bandits Spring 2019

Exploration in Contextual Bandits

Instructors: Alekh Agarwal, Sham Kakade Lecture 8-9

In the last lecture we studied a contextual bandit setting where the learning algorithm has no influence over the data
collection strategy. In this lecture, we will study a different setting where the learning algorithm actively engages in
an explore-exploit tradeoff, and controls the choice of actions in response to the contexts it observes. We will focus
attention on the i.i.d. setting, that is the contexts and
p rewards are drawn from a fixed distribution. Statistically, our
goal is to match the optimal regret guarantee of O( e KT ln |Π|) in the worst case.1 Computationally, our goal is to
develop algorithms which can be efficiently implemented.
Continuing from last lecture, we will assume access to a cost-sensitive classification algorithm over policies in Π. We
formalize this assumption through the notion of an ArgMax Oracle (AMO) below.

Definition 1 (ArgMax Oracle (AMO)). Given a policy class Π, an ArgMax Oracle (AMO) for short is an algorithm
which takes in an arbitrary dataset {(xs , rs )}s=1,2,...,t with xs ∈ X and rs ∈ [0, 1]K , and computes

t
X
arg max rs (π(xs )).
π∈Π
s=1

In words, an AMO is a reward maximization or equivalently cost minimization algorithm for the class Π.
Remark: Note that the AMO itself makes no assumption on the contexts and rewards coming from a distribution,
and this will be crucial to our algorithms. It is assumed to return the right answer for any dataset, even adversarially
generated.
Given such an oracle, we can clearly implement the τ -G REEDY algorithm using just one call to the AMO for com-
√ τ rounds. However, this gives an algorithm with a suboptimal regret bound scaling as
puting the G REEDY policy after
T 2/3 , instead of the optimal T dependence that we seek.
In this lecture, we will work towards attanining statistical optimality without sacrificing computational feasibility.
To build intuition though, we will start from an algorithm which is computationally inefficient. We will then see a
sophisticated variant of it which can indeed be implemented efficiently given an AMO.
Before describing the algorithms, we need a few additional pieces of notation. Recall that we use V (π) to denote the
expected reward of a policy. After the contextual bandit algorithm has run for t rounds, it has an empirical dataset of
the form (xs , as , ps , rs )ts=1 , where ps is the probability with which the action as was chosen at round s. Given such
a dataset, we can also use the off-policy techniques from the previous lecture to define an empirical estimate for the
reward of any policy π. Specifically, we will use the IPS estimator in this lecture for simplicity even though any other
estimator can be alternatively used. With this in mind, we define the empirical estimate of π’s value using t samples
as

t
1 X 1(as = π(xs ))
Vbt (π) = rs . (1)
t s=1 ps

With these expected and empirical values of a policy π, we also define corresponding shorthands for regret as:

1 Here, and throughout we use the O

e notation to suppress polylogarithmic factors in K, T and ln(|Π|/δ).

1
2 Lecture 8-9:

Reg(π) = V (π ? ) − V (π) = max

0
V (π 0 ) − V (π)
π ∈Π

Reg πt ) − Vbt (π) = max

d t (π) = Vbt (b
0
Vbt (π 0 ) − Vbt (π). (2)
π ∈Π

Note that while Vbt (π) is an unbiased estimator for V (π) under the conditions we discussed in previous lecture, Reg
d t (π)
is typically a biased estimate for Reg(π) since we base it on the empirical best policy π bt instead of π ? . However, this
bias is usually typically small and can be bounded.
P
We will use Q ∈ ∆(Π) to denote distributions over Π, that is Q(π) ≥ 0, π∈Π Q(π) = 1. Given a context x, we will
also abuse notation and use Q(a | x) to be the induced distribution over actions, given the context. That is,
X
Q(a|x) = Q(π). (3)
π∈Π : π(x)=a

Given a desired minimum probability for each action, we also use the shorthand Qγ (a|x) to be the mixture of Q with
the uniform distribution over actions as
X
Qγ (a|x) = (1 − Kγ) Q(π) + γ. (4)
π∈Π : π(x)=a

Note that we subtract Kγ instead of just γ, since the uniform distribution adds a probability γ over each of K actions.

1 Policy Elimination

Notice that the EXP4 algorithm, which gives the optimal regret guarantee, does so by maintaining and updating a
distribution over policies in Π, with larger weight being placed on policies which obtain a high reward. We will now
see an algorithm which uses a similar intuition in the i.i.d. setting.

1.1 The Algorithm

The algorithm, called P OLICY E LIMINATION [Dudı́k et al., 2011], is an iterative elimination based algorithm. At each
iteration, the algorithm evaluates every surviving policy according to the IPS estimator on the data collected so far.
Policies which have a low empirical regret according to these estimates are retained, while the rest are eliminated. The
algorithm then constructs a probability distribution for exploration at the next round over the surviving policies.
The main design question in the algorithm is how to construct a distribution over the policies, and for this we derive
guidance from the results on off-policy evaluation. Specifically, we would like that the set of policies retained based
on low-empirical regret resembles the set of policies with low expected regret. This would be true if we have good
estimates of the reward for each policy. Since the quality of our off-policiy estimates depends on the distribution of
propensity scores, the algorithm constructs a distribution for exploration that results in well-behaved propensity scores
on average. This intution is formalized in Algorithm 1.
The two main steps in the algorithm are those of finding a distribution respecting the constraints (5), and that of
eliminating bad policies in the last step. The constraint (5) corresponding to π captures the variance in evaluating the
reward of the policy π, if we were to collect samples using Qt . The variance depends, based on last lecture, inversely
on the probability Qγ (π(x)|x) for the context x. Taking expectation over this quantity results in the constraints of
Algorithm 1.
Lecture 8-9: 3

Algorithm 1 P OLICY E LIMINATION algorithm for i.i.d. contextual bandits

Require: Failure probability δ. number of rounds T .
Initialize Π0 =
qΠ.  q 
Define t = 2 2K ln(Nt
T /δ)
and γ = min 1
2K , ln(N T δ)
2KT .
for t = 1, 2, . . . , T do
Observe xt and choose a distribution Qt over Πt−1 such that
 
1
Ex∼D ≤ 2K, ∀π ∈ Πt−1 . (5)
Qγ (π(x)|x)

Choose action at ∼ Qγt (a|xt ).

Observe reward rt .
Update Πt = {π ∈ Πt−1 : Reg d t (π) ≤ 2t }.
end for

Remark: Note that Algorithm 1 assumes the knowledge of the distribution D over contexts (that is, the marginal
distribution over x, without the rewardS) as well as the number of rounds T . Both of these are avoidable, but assuming
these allows us to present simpler proofs without losing any of the key ideas. We refer interested readers to Dudı́k
et al. [2011] for the formal arguments in the more general setting.
Statistically, there are two properties of the algorithm which are important to understand. One is that the distribution
Qt which can satsify the constraint 5 must exist for the algorithm to work. Secondly, we would like to certify that the
algorithm incurs a low regret. For the regret, it seems sufficient to argue that our empirical estimates Vbt (π) are close
to V (π) for all π ∈ Πt and for all t. In the previous lectures, we saw arguments for doing this both in an off-policy
setting as well as for the τ -G REEDY algorithm using Hoeffding’s or Bernstein’s inequality. Specifically, consider the
random variable
1(at = π(xt ))
Yt (π) = rt − V (π).
pt
Then it is easily seen that
t
1X
Vbt (π) − V (π) = Ys (π). (6)
t s=1

In previous lectures, we argued that Yt (π) is a sequence of i.i.d. and mean-zero random variables, so that their average
is close to zero with high probability.
However, we cannot continue to use the same idea here. In both the off-policy and τ -G REEDY settings, the action
distribution is non-adaptive. That is, (xt , at , pt , rt ) as a tuple form an i.i.d. collection of random variables. In the off-
policy setting, we use a fixed policy µ, while our exploration is uniform in τ -G REEDY. In the P OLICY E LIMINATION
algorithm, on the other hand, the probability pt = Qγt (at |xt ) is heavily dependent on the previous actions, contexts
and rewards. Consequently, Yt (π) are not i.i.d. anymore and our earlier bounds from Hoeffding’s or Bernstein’s
inequality do not apply. In order to analyze such adaptive algorithms, we recall the notion of martingales from the
first lecture.

1.2 Digression: Martingale concentration

Definition 2 (Martingale). Suppose Z1 , . . . , Zn are mean zero random variables, which additionally satisfy E[Zi |Z1 , . . . , Zi−1 ] =
0 for all i = 2, 3, . . . , n, then the sequence Zi is called a martingale difference sequence.2
2 This definition is only a special case of the general formulation of martingales. This particular form of a martingale is often called a Doob

martingale.
4 Lecture 8-9:

Martingales are useful objects in probability theory as they satisfy many inequalities analogous to i.i.d. random
variables. It turns out they are also very relevant in the study of adaptive algorithms like P OLICY E LIMINATION.
Specifically, we can show that the sequence Yt (π) is a martingale difference sequence. Letting Zt = Yt (π), we notice
that conditioning on the previous Zi is analogous to conditioning on the previous samples (xi , ai , pi , ri ), since they
describe all the randomness contained in Yi (π). For brevity, we introduce the following two shorthands given any
random variable Y :

Et [Y ] = E[Y |(xi , ai , pi , ri )t−1

i=1 ] and Vart [Y ] = Var[Y |(xi , ai , pi , ri )t−1
i=1 ].

Given this notation, checking that Yt (π) form a martigale difference sequence is the same as verifying Et [Yt (π)] = 0.
This is done in our next result.

Lemma 3. The sequence Yt (π) is a martingale difference sequence.

Proof: As mentioned above, it suffices to verify that Et [Yt (π)] = 0. Notice that
    
1(at = π(xt )) 1(at = π(xt ))
Et rt = Et E rt |xt ,
pt pt
(a)
vIPS (xt , π(xt ))|xt , Qγt (xt )]]
= Et [E [b
(b)
= Et [E [v(xt , π(xt ))|xt , Qγt (xt )]]
(c)
= V (π).

Here, we have used the defintion of vbIPS in equality (a), and we recall that pt = Qγt (at |xt ). Then step (b) follows
by Lemma 7.3 in the previous lecture, since the expectation is only over the random choice of at according to its
distribution, and the random choice of rewards from their conditional distribution given xt and π(xt ). In particular,
observe that at is conditionally independent of the past given xt and the distribution Qγt (xt ) over actions. Similarly
rt is conditionally independent of the past given xt and at , whence the equality (b) follows. Finally, equality (c) uses
the fact that xt is independent of the past and drawn from D, so that the expectation is simply the value of π.
Having shown that Yt (π) form a martingale difference sequence, we next describe a Bernstein-like inequality which
they follow. This inequality will be crucial in showing that our estimates Vbt (π) and V (π) are close.

Lemma 4P (Freedman-style inequality). Let Z1 , . . . , Zn be a martingale difference sequence with Zi ≤ R for all i.
n
Let Vn = i=1 Varn [Zi ]. For any δ ∈ (0, 1) and any λ ∈ [0, 1/R], with probability at least 1 − δ
n
X ln(1/δ)
Zi ≤ (e − 2)λVn + .
i=1
λ

Remark: The inequality looks quite different from Bernstein’s at the first glance. However, this really is just a more
general form convenient for
psome of our proofs, and of course applies to martingales. We encourage the reader to
check that using λ = min( Vn / ln(1/δ), 1/R) yields a Bernstein-like bound for martingales.
With this digression, we now have all the tools for a statistical analysis of Algorithm 1.

1.3 Regret analysis

We begin by showing that the distribution Qt posited in Equation 5 does exist.

Lecture 8-9: 5

Theorem 5 (Feasibility). Let Π be any finite policy class. For any γ ∈ (0, 1/K] and all joint distributions D over
(x, r), we have
1 K
min max Ex∼D ≤ .
Q∈∆(Π) π∈Π Qγ (π(x)|x) 1 − Kγ

Remark: Applying the theorem to the policy set Πt−1 and noting that γ ≤ 1/2K in Algorithm 1, we observe that
there exists a distribution Qt ∈ ∆(Πt−1 ) such that the maximum LHS of constraints (5) is at most K/1 − Kγ ≤ 2K
using γ ≤ 1/2K.
Proof: The result is a consequence of a minimax theorem. To begin, we verify some regularity conditions on the
various objects in our problem. Note that ∆(Π) is the convex hull of a finite set. We further note that

" #
1 X 1
max Ex∼D γ = max Ex∼D P (π) γ
π∈Π Q (π(x)|x) P ∈∆(Π) Q (π(x)|x)
π∈Π
 
K
X X 1
= max Ex∼D  P (π) γ 
P ∈∆(Π)
a=1 π∈Π : π(x)=a
Q (a|x)
"K #
X 1
= max Ex∼D P (a|x) γ
P ∈∆(Π)
a=1
Q (a|x)
1
= max Ex∼D Ea∼P (·|x) .
P ∈∆(Π) Qγ (a|x)

Here the first equality is clearly an upper bound since we enlarged the maximization to a clearly larger set ∆(Π),
which in particular includes each policy π by choosing a distribution which puts its entire mass on π. However, the
inequality is really an equality because the objective of maximization is linear in P . So the maximization over all
distributions P simply returns a point mass over the policy π for which the objective is that largest.
This allows us to instead consider the problem

1
min max Ex∼D Ea∼P (·|x) .
Q∈∆(Π) P ∈∆(Π) Qγ (a|x)

Both P and Q are chosen from the convex hull of a finite set here. Furthermore, the objective is linear (and hence
concave) in P . Furthermore, it is easily seen to be convex in Q (from the convexity of f (x) = 1/x). Under these
conditions, we can invoke Sion’s minimax theorem [Sion, 1958], which allows us to swap the order of maximization
and minimization. That is,

1 1
min max Ex∼D Ea∼P (·|x) = max min Ex∼D Ea∼P (·|x) γ .
Q∈∆(Π) P ∈∆(Π) Qγ (a|x) P ∈∆(Π) Q∈∆(Π) Q (a|x)

We now proceed to upper bound the RHS of this equality. Since we are taking a minimum over Q, using any other
distribution gives a valid upper bound, and in particular we choose Q = P . This yields
6 Lecture 8-9:

1 1
max min Ex∼D Ea∼P (·|x) ≤ max Ex∼D Ea∼P (·|x) γ
P ∈∆(Π) Q∈∆(Π) Qγ (a|x) P ∈∆(Π) P (a|x)
K
X 1
= max Ex∼D P (a|x)
P ∈∆(Π)
a=1
(1 − Kγ)P (a|x) + γ
K
X 1
≤ max Ex∼D P (a|x)
P ∈∆(Π)
a=1
(1 − Kγ)P (a|x)
K
= .
1 − Kγ

Thus, we have proved that a distribution Qt satisfying the conditions (5) of Algorithm 1 exists. Next, we would like
to establish that if we play according to such a distribution at each round, then we incur low regret.
We will establish the following proposition, from which the regret guarantee will follow as a corollary.
Proposition 6 (Concentration of regret estimates). With probability at least 1 − 2δ, we have for all t

max |Vbt (π) − V (π)| ≤ t .

π∈Πt

Proof: Fix a time step t and a policy π ∈ Πt . Recall Equation 6 and Lemma 3 which collectively show that Vbt (π) −
V (π) is a sample-average of a martingale difference sequence. Furthermore, each Yt (π) satisfies |Yt (π)| ≤ 1/γ, since
the rewards are bounded in [0, 1] and pt = Qγt (·|xt ) ≥ γ. Furthermore, we have
 
2 1(at = π(xt ))
Vart [Yt (π)] ≤ Et rt
p2t
(a)
 
1(at = π(xt ))
≤ Et
p2t
"K #
X γ 1(a = π(xt ))
= Et Ext ∼D Qt (a|xt )
a=1
Qγt (a|xt )2
 
1
= Et Ext ∼D
Qγt (π(xt )|xt )
(b)
≤ 2K.

Here inequality (a) follows as rt ∈ [0, 1] and (b) is a result of the constraint (5), since Qt satisfies these constraints
and π ∈ Πt ⊆ Πt−1 . By the nested structure of Πt , any π ∈ Πt is contained in all the previous Πs so that the variance
bound applies to each Ys (π).
Thus we have a bound on the range and conditional variances of the Yt , and we can invoke Lemma 4 to bound their
sample average. Applying the lemma twice, once for Zt = Yt once for Zt = −Yt , we observe that with probability at
least 1 − 2δ 0 , we have for any λ ∈ [0, γ]
 
t
X  ln(1/δ 0 )
Yt (π) ≤ (e − 2)λ 2Kt +
 
λ

 
s=1

We use the inequality with δ 0 = δ/N T and λ = γ, which yields that with probability at least 1 − 2δ/N T , we have
Lecture 8-9: 7

 t 
X  ln(N T /δ)
Yt (π) ≤ 2(e − 2) γKt + (7)
 
γ

 
s=1

q
ln(N T /δ)
To simplify further, we assume that T is large enough so that γ = 2KT . If not, then we have
r
ln(N T /δ) 1
≥ ⇒ t ≥ 1 , t = 1, 2, . . . , T.
2KT 2K
Thus, the propisition is trivially true in the other case and we can focus on the desired setting of γ. Plugging this
setting into Equation 7, we obtain that with probability at least 1 − 2δ/N T

 t 
X  ln(N T /δ)
Yt (π) ≤ 2(e − 2)γKt + ≤ t , (8)
 
γ

 
s=1

where the last inequality uses e − 1 ≤ 2. Taking a union bound over all policies and rounds of the algorithm completes
the proof.
Remark: We observe that the constraints (5) provide a direct bound on the variance of our Yt (π) random variables.
As a result, we will often refer to these as the variance constraints.
Given the proposition, we can now state the main regret bound for Algorithm 1.
Theorem 7. With probability at least 1 − 2δ, for all t, we have:

1. π ? ∈ Πt , that is, π ? is never eliminated.

2. V (π) ≥ V (π ? ) − 4t for all π ∈ Πt .
q
Consequently, the regret of Algorithm 1 is bounded with probability at least 1 − 2δ by 17 2T K ln TδN .

Proof: To simplify our handling of probabilities, let E refer to the event that the conclusion of Proposition 6 holds for
all π, t. The proposition guarantees that P(E C ) ≤ 2δ.
We begin with the first part of the theorem. Let us inductively assume that π ? ∈ Πs for all s = 0, 1, 2, . . . , t. The base
case is clearly true as π ? ∈ Π = Π0 . We will now show that π ? ∈ Πt+1 . We apply Proposition 6 after round t twice.
Once with π ? and once with π bt . Our inductive assumption gives that π ? ∈ Πt and Reg
d t (b
πt ) = 0 by definition so that
it is also in Πt . Consequently, the preconditions of Proposition 6 are satisfied for both policies, and we get under the
event E,

Vbt (π ? ) ≥ V (π ? ) − t , and πt ) ≥ Vbt (b

V (b π t ) − t .

Adding the two inequalities, we obtain that

Vbt (π ? ) ≥ Vbt (b
πt ) + V (π ? ) − V (b
πt ) − 2t ≥ Vbt (b
πt ) − 2t , (9)

where the second inequality follows since V (π ? ) ≥ V (bπt ) by definition of π ? . This demonstrates that π ? is not
eliminated at round t + 1 and the induction is complete.
By rearranging the terms in Equation 9 a bit differently, we also see that under E, we have

πt ) ≥ V (π ? ) − 2t .
V (b
8 Lecture 8-9:

Similarly, we observe that under E, for any π ∈ Πt

πt ) − 2t ≥ Vbt (π ? ) − 2t ≥ V (π ? ) − 3t .

Vbt (π) ≥ Vbt (b

Further adding the deviation between Vbt (π) and V (π) completes the proof.
Finally, for the regret bound note that the algorithm always plays according to a distribution over Πt at round t, except
for the Kγ uniform random distribution which is added in. Hence, the cumulative regret of the algorithm is at most
T T
r r
X X 2K ln(N T /δ) ln(N T δ)
(1 − Kγ)4t + KγT ≤ 8 + KT
t=1 t=1
t 2KT
r
p KT ln(N T δ)
≤ 16 2KT ln(N T /δ) + .
2

Remark: Here we analyze regret as the expected reward of our chosen action, compared with the expected reward
of the best @inproceedingssyrgkanis2016improved, title=Improved regret bounds for oracle-based adversarial con-
textual bandits, author=Syrgkanis, Vasilis and Luo, Haipeng and Krishnamurthy, Akshay and Schapire, Robert E,
booktitle=Advances in Neural Information Processing Systems, pages=3135–3143, year=2016 policy. Recall a prob-
lem from Homework 2 to obtain a bound on the actual obtained reward in terms of the empirically best policy using
this expected regret.

2 A computationally efficient approach

The P OLICY E LIMINATION algorithm so far has little difference from EXP4, in that it is statistically optimal (albeit in
a more limited i.i.d. setting), but computationally impractical. It turns out though, that the structure of P OLICY E LIM -
INATION can provide crucial guidance towards the development of efficient algorithms for the i.i.d. setting.

The main computational bottlenecks of P OLICY E LIMINATION are the elimination step and the requirement to enforce
the variance constraints (5) for each surviving policy. It is natural to wonder if we can avoid elimination altogether,
enforce the variance constraints over all the policies and somehow encourage the distribution Qt to not place mass over
policies with a large empirical regret? The next example shows that we cannot hope to enforce the variance constraints
for all π ∈ Π, and also obtain low regret.
Example 8. Let us consider a policy class Π consisting of two policies πgood and πbad . Suppose further that we have
only two actions a1 and a2 with v(x, a1 ) = 1 and v(x, a2 ) = 0 for each context x. Furthermore, πgood always chooses
a1 and πbad always chooses a2 indpendent of the context. Now the variance constraint (5), if enforced for policy πbad
implies
 
1
Ex ≤ 2K
Qγ (πbad (x)|x)
 
1
⇒ Ex ≤ 2K
Qγ (a2 |x)
 
(a) 1
⇒ Ex ≤ 2K
(1 − Kγ)Q(πbad ) + γ
1
⇒ (1 − Kγ)Q(πbad ) + γ ≥
2K
1 − 2Kγ
⇒ Q(πbad ) ≥ .
2K(1 − Kγ)
Lecture 8-9: 9

Here the implication (a) follows since πbad is the only policy which takes a2 . If we choose γ ≤ 1/(4K), then the RHS
above is at least 1/(3K). Noting that K = 2, we pick πbad with probability at least 1/6 each round, meaning that our
cumulative reward is at most 5T /6, leading to a regret of at least T /6 for the algorithm.

This example tells us that it is damaging to our goal of low-regret, if we insist of having the variance constraints also
for the bad policies π. This makes intuitive sense too, we only want to focus on taking the actions which help us
distinguish amongst the good (rather not provably bad) policies. We will now see how to motivate the design of a
different, and computationally efficient algorithn.

2.1 A new optimization problem and its statistical analysis

The search for a distribution Q satisfying constraints (5) can be seen as an optimization problem. We are looking for a
distribution which puts mass only over the policies in Πt−1 and satisfies all the constraints. We now present a related,
but different optimization problem which was developed in Agarwal et al. [2014]. We will subsequently show that any
algorithm which picks a distribution based on this new optimization problem also enjoys low regret. The next section
will show an efficient algorithm to solve the problem.

Optimization Problem (OP)

Reg
d (π)
Given samples (xs , as , ps , rs )ts=1 , minimum probability γ and with bt (π) = t
4(e−2)γ ln T ,
find Q ∈ ∆(Π) such that
X
Q(π)bt (π) ≤ 2K and (10)
π∈Π
 
1
∀π ∈ Π : Ex∼D ≤ 2K + bt (π). (11)
Qγ (π(x)|x)

Figure 1: The optimization algorithm (OP) in the ILOVETOCONBANDITS algorithm of Agarwal et al. [2014].

Qualitatively, the optimization problem OP in Figure 2.1 is not so different from the constraints (5). We are still
looking for a distribution such that the estimation variance for the policies π can be controlled. Crucially though, the
algorithm does not explicitly eliminate bad policies now, and enforces the low-variance constraints for each policy π.
To circumvent the bad cases such as that of Example 8, we do allowpfor different bounds on the variance for each
policy π. Intuitively, if b(π) = O(K), meaning that Reg d t (π) = O( K/t ln(T N/δ)) by our setting of γ, then the
overall RHS in the variance constraints in OP is still O(K) like in the P OLICY E LIMINATION algorithm.
√ But for the
policies which are empirically poor, we allow for much larger estimation variance, as large as O( t) if the empirical
regret is a constant. This relaxed constraint on the estimation variance for bad policies means that we do not risk
over-exploring in the cases like Example 8.
Additionally, since the distribution Q is over all the policies and not just the empirically good policies, we require a
constraint saying that Q should have good empirical regret.p Recalling the setting of b(π), we observe that the empirical
regret of picking a policy according to Q is O(Kγ) = O( K/t ln(T N/δ)). Since we have low estimation variance
for all policies with regret at that level, our analysis will show that the actual regret of Q is not much worse either.
That is, we posit having a small empirical regret and a small estimation variance, which collectively imply a small
population regret.
We will now make this intuition formal in a few lemmas. We will show how to obtain the regret bound for the algorithm
by putting together these lemmas, while deferring the more technical steps to an appendix. Our analysis focuses on
the following intuitive algorithm:
10 Lecture 8-9:

1. At time t, find a distribution Qt which satisfies OP using past samples, i.e. with bt−1 (π).

2. Observe xt , and play an action according to Qγt (a|xt ).

3. Observe rt and incorporate (xt , at , pt , rt ) to the dataset.

We begin with the statement of the overall theorem we will prove about any algorithm which plays a distribution
satisfying OP at each round.

Theorem 9. Suppose an algorithm plays according to a distribution which isaqsolution to OP at each round
 t. Then
with probability at least 1 − δ, the algorithm incurs a regret no more than O KT ln TδN + K ln TδN .

Remark: Note that we have no explicitly shown that the optimization problem OP is even feasible, but only asserted
a bound on regret assuming that a distribution feasible according to it can be found. In the next section, we will show
an explicit construction for distributions satisfying it, which implicitly yields feasibility too.
Remark: The theorem shows that the less stringent constraints in OP, compared with (5) do not cost us statistically.
The next section will show how this creates the path to an efficient algorithm.
Remark: As before we assume that the constraints involve expectations under the actual distribution D over the
contexts. This can be relaxed to an empirical average, as detailed in Appendix B.2 of Agarwal et al. [2014].

2.2 Some helper lemmas

We now present a couple of lemmas which can be easily put together to obtain the theorem. The first lemma can be
seen as an analog of Proposition 6.

Lemma 10. Under conditions of Theorem 9, with probability at least 1 − δ, we have for all t = 1, 2, . . . , T all policies
π ∈ Π and λ ∈ [0, γ] !
t
  1X ln TδN
Vt (π) − V (π) ≤ (e − 2)λ 2K + bs−1 (π) + .
b 
t s=1 λt

The lemma follows almost directly from combining Freedman’s inequality (Lemma 4) along with our variance con-
straints in OP, similar to how we used the variance constraints in the P OLICY E LIMINATION algorithm.
Remark: In order to apply the lemma at round 1, we also define b0 (π) = 0, and 2K + b0 (π) = 2K is still an upper
bound on the variance at round 1, since a distribution satisfying this bound for all policies π exists by the Lemma 5
applied to the first step of the P OLICY E LIMINATION algorithm.
We next show how the deviation bound on reward estimates implies concentration of regret.

Lemma 11. Under conditions of Theroem 9, with probability at least 1 − δ, we have for all t = 1, 2, . . . , T and all
π ∈ Π:
Reg(π) ≤ 2Reg
d t (π) + t , and d t (π) ≤ 2Reg(π) + t ,
Reg
q
where t = 4 Kt ln TδN + 32 TN
γt ln δ .

Remark: We see that unlike in the P OLICY E LIMINATION algorithm, we are not guaranteeing that Reg(π) and
Reg d t (π) = O(t ),
d t (π) are necessarily close (due to the factor of 2). However, we do still guarantee that if Reg
then Reg(π) ≤ Reg
d t (π) + c t for some constant c. Note that t is quite similar to the elimination threshold we
Lecture 8-9: 11

used in the P OLICY E LIMINATION algorithm. So we are asserting that for all the good policies, we still have good
concentration of empirical regret around its expectation, but the concentration is allowed to be worse for bad policies.
This is a natural consequence of having a higher variance on bad policies.
The proof of the lemma effectively uses Lemma 10 twice, for π and π ? to bound the deviation of regret. We present a
rough sketch here, with a detailed proof in Appendix A.
We begin with the observation:

d t (π) ≤ V (π ? ) − Vbt (π ? ) − V (π) + Vbt (π),

Reg(π) − Reg
and invoke Lemma 10 on each of the deviation terms. The main departure from our previous analysis is that the
deviation now depends on the policy through the bs−1 (π) terms in Lemma 10.

In order to reason about them, Lemma 11 is proved inductively. Since bs−1 (π) is Reg
d s−1 (π) up to scaling factors, it
is also related to Reg(π) under the inductive assumption. That is, we obtain using the inductive assumption:

t
!
TN
d t (π) ≤ 2 ln δ + 4(e − 2)λK + (e − 2)λ
Reg(π) − Reg
1X
(bs−1 (π) + bs−1 (π ? ))
λt t s=1
t
!
2 ln TδN λ 1X ?
≤ + 4(e − 2)λK + (2Reg(π) + 2Reg(π ) + 2s−1 ) .
λt 4γ ln T t s=1

Rearranging terms along with some algebra now gives the upper bound on Reg(π) in Lemma 11. Following a similar
logic also yields the bound on Reg
d t (π) in terms of Reg(π). A detailed proof of this result can be found in Appendix A.

Given Lemma 11, the proof of Theorem 9 is quite straightforward.

2.3 Proof of Theorem 9

Under the conditions of Theorem 9, at each round we play according to a distribution Qt which satisfies the constraints
in OP. This implies, in particular that the distribution Q has a low, empirical regret. Specifically,
X
d t−1 (π) ≤ 8K(e − 2)γ ln T.
Qt (π)Reg
π∈Π

Using Lemma 11 this immediately yields a bound on the expected regret of picking policies according to Q. Now we
just have to factor in the additional regret we incur due to mixing in the uniform distribution, which is bounded by γT .
Substituting the value of γ and simplifying yields the desired guarantee.

3 An efficient algorithm to solve OP

The approach of solving OP instead of the P OLICY E LIMINATION algorithm eliminates the need to explicitly maintain
the set of good policies. However, checking the feasibility of a distribution Q for OP is still challenging since the
set of constraints is extremely large, with one constraint for each policy π. Naı̈vely, it appears just as intractable as
P OLICY E LIMINATION or EXP4 approaches. However, we will see that the structure of OP is amenable to efficient
solutions by leveraging the existence of AMO for Π. We now present an algorithm to solve OP which can be
12 Lecture 8-9:

implemented by appropriately invoking the AMO. We will also discuss the computational efficiency of this algorithm
in finding a solution to OP. Throughout, we will drop the time index t since the entire analysis is focusing on an
arbitrary round of the contextual bandit algorithm.

Algorithm 2 Coordinate descent algorithm for solving OP

Require: Initial Qinit . Qinit can be any elementwise non-negative vector with entries summing to no more than 1.
Intialize Q := Qinit .
loop
Define the following quantities for all π ∈ Π:
  " #
1 1
Varπ (Q) = Ex , Sπ (Q) = Ex 2 , and Dπ (Q) = Varπ (Q)−(2K +b(π) (12)
Qγ (π(x)|x) (Qγ (π(x)|x))
P
if π∈Π Q(π)(2K + b(π)) > 2K then
Replace Q by cQ where
2K
c= P < 1.
π∈Π Q(π)(2K + b(π))
end if
if There is a policy π for which Dπ (Q) > 0 then
Update Q(π 0 ) = Q(π 0 ) + α, if π 0 = π and leave Q(π 0 ) unchanged otherwise, where

Varπ (Q) + Dπ (Q)

α= .
2(1 − Kγ)Sπ (Q)

else P
Let θ = π∈Π Q(π).
Halt and output Q + (1 − θ)1(b
π ), where 1(b
π ) is a distribution which picks the empirically best policy π
b
with probability 1.
end if
end loop

The algorithm can be intuitively seen as trying to greedily satisfy the constraints in OP. The first check and renormal-
ization of the distribution Q is clearly enforcing the low-regret constraint in OP. The quantity Dπ (Q) is simply the
violation of the variance constraint for policy π in OP, and then algorithm increases the weight on π if this constraint
is violated, which should decrease the constraint violation. Given these properties, the following lemma is immediate.

Lemma 12. If Algorithm 2 halts and outputs a distribution Q, then it is feasible for OP.

Remark: The algorithm tacitly assumes that the quantity θ < 1 in the last line,
P which is also required for the above
assertion of correctness. However, this follows since we are assured that Pπ Q(π)(2K + b(π)) ≤ 2K when the
algorithm terminates. Recalling the non-negativity of b(π), this implies that π Q(π) ≤ 1.
Given the correctness, we will now study the computational properties of Algorithm 2. We begin by showing how the
algorithm can be implemented using an AMO for Π first. Subsequently, we will bound the number of iterations of
Algorithm 2 before termination, which will yield a bound on the number of AMO calls.

Lemma 13. Algorithm 2 can be implemented with one call to AMO before the loop is started, and one call for each
iteration of the loop thereafter.

Proof: Before starting the loop, we make a single call to the AMO to obtain the empirically best policy π
b so far,
which is required in order to evaluate b(π) (or rather Reg(π)).
d
Lecture 8-9: 13

P
Now we have two main computational steps in the algorithm. There is the computation of π∈Π Q(π)b(π) before
we potentially renormalize the distribution. This computation is efficient even for large Π as long as the distribution
Q has a sparse support. For instance, assuming Qinit is the all 0’s vector, since Q adds a non-zero weight to only one
policy at each iteration, the size of its support is bounded by the number of iterations.
The key computational burden then, is in the checking if there is a policy π satisfying Dπ (Q) > 0, that is there is a
policy whose variance constraint in OP is violated. Note that the Varπ (Q) term in Dπ (Q) is
  t
1 1X 1
Ex∼D γ
≈ γ
,
Q (π(x)|x) t s=1 Q (π(xt )|xt )

where we have replaced the true expectation with a sample average involving the previously seen contexts. While this
might seem like a heuristic argument, the paper [Agarwal et al., 2014] makes this rigorous by using sample averages
in the variance constraints of OP, and bounding the error from doing this. For simplicitly, we will continue to assume
that the variance terms are a sample average for this remainder of the computational analysis. Ignoring the constant
2K in Dπ (Q), the other π-dependent term is b(π), which can be simplified as
t
Reg
d t (π) Vbt (b
πt ) 1 X
bt (π) = = − vbt (xt , π(xt )),
ψγ ψγ ψγt s=1

where ψ = 4(e − 2) ln T and vbt (xt , a) is the IPS estimator for the reward of action a, given the tuple (xt , at , pt , rt )
observed by the algorithm.
In order to check whether a policy with Dπ (Q) > 0 exists, it suffices to find arg maxπ∈Π Dπ (Q) and compute the
corresponding maximum value. By the above simplifications, we see that we need to compute
t t
1X 1 1 X
arg max Dπ (Q) = arg max Varπ (Q) − 2K − b(π) = arg max γ
+ vbt (xt , π(xt )),
π∈Π π∈Π π∈Π t s=1 Q (π(xt )|xt ) ψγt s=1

where we dropped all the terms indepedent of π. Now, given any round t and action a, we define
ψγ
r˜t (a) = vbt (xt , a) + .
Qγ (a|x t)

We then create a dataset (xs , r˜s )ts=1 and feed it to the AMO. It is now easily seen that the output of the AMO can
be used to check whether Dπ (Q) > 0, which concludes the proof.
Thus, each iteration of Algorithm 2 is computationally feasible. It now remains to bound the number of iterations,
which controls both the sparsity of the distribution Q as well as the number of calls to the AMO. We present this
bound in the next theorem.
4 ln(1/(Kγ)
Theorem 14. Algorithm 2 with Qinit = 0 terminates in at most γ iterations and outputs a solution to OP.

p
Remark: Recalling the setting of γ, we see that the algorithm terminates in at most O(e T /K ln(T N/δ)) iterations.
√
That is, we make O( e 1.5 ) calls across all rounds. In Agarwal
e T ) calls to the AMO at every round, and a total of O(T
et al. [2014],
√ a more sophisticated epoch and warm-starting strategy is used to bring the overall number of calls down
to O(
e T ) across all T rounds. That is, we just need a sublinear number of AMO calls, and this is unavoidable for
any algorithm which solves OP due to a corresponding lower bound in that paper.
Proof: The proof uses a potential function argument, and we will only present the ideas at a very high-level. The key
insight of the algorithm is in finding a potential function Φ(Q), such that minimizing Φ(Q) leads to finding a solution
to OP. Since the function is acting over the N -dimensional variable Q, and we also have N variance constraints in
OP, the function Φ is designed such that its directional derivative along policy π is effectively the constraint violation
14 Lecture 8-9:

Dπ (Q). Since the derivative vanishes at optimum, we can have no violation of the variance constraints. The handling
of the low-regret constraint requires additional care, but is ensured by the re-normalization step in Algorithm 2.
P
Specifically Agarwal et al. [2014] show that if π Q(π)(2K + b(π)) > 2K, then Φ(cQ) ≤ Φ(Q) for the constant c
in Algorithm 2. This implies that the renormalization step never increases the potential.
They further show that everytime we find a π such that Dπ (Q) > 0 and update the distribution to add α mass on π,
this leads to a significant decrease in the potential.
The convergence result now follows since the potential function is chosen to be upper and lower bounded, and is de-
creased significantly everytime we find a variance constraint violation. Hence, after the number of updates postulated
in the theorem statement, the function has to be reduced to its minimum value.
The above argument is rather vague, but intentionally so. We encourage the reader to study the exact form of the
potential, and the details of the convergence analysis in the paper.
√
In summary, the whole exercise yields an algorithm which invokes the AMO at most O( e T ) times over T rounds
and enjoys the statistically optimal regret guarantee for i.i.d. contextual bandit problems. In contrast, τ -G REEDY
requires exactly one AMO call but is suboptimal in regret, while EXP4 has no known implementation via the AMO.
Subsequent works [Syrgkanis et al., 2016a, Rakhlin and Sridharan, 2016, Syrgkanis et al., 2016b] have explored other
trade-offs in terms of the computational and statistical efficiencies, as well as the distributional assumptions on the
problem.
For practical implementation, τ -G REEDY or its -G REEDY version are still the most convenient and efficient to im-
plement. While the approach presented here can be also implemented using an AMO, doing so is non-trivial and
expensive. In the paper [Agarwal et al., 2014], we discuss epoching strategies to ease the computational cost, as
well as online approximations to develop a more practical algorithm. Having strong regret guarantees for an online
algorithm remains a challenging open question in this area.

Appendix A Proof of Lemma 10

We give a more formal version of our proof sketch here. As stated earlier, we will prove the lemma by induction.
Starting with the base case, we check that 1 > 1. Since the Reg(π) ≤ 1, the upper bound on population regret easily
follows. For the upper bound on empirical regret, we note that 1 ≥ 1/γ as well, but the empirical regret is no more
than 1/γ meaning that the base case is satisfied. We will now assume that the statement of the lemma holds for all
s = 1, 2, . . . , t − 1 and all policies π and establish it at round t. To simplify our treatment of probabilities, we will
assume that the 1 − δ probability event in Lemma 10 holds.

We will only work out the upper bound on Reg(π) in terms of Reg
d t (π) since the proof of the other bound follows
similarly. Now we observe that

d t (π) = V (π ? ) − V (π) − Vbt (b

Reg(π) − Reg πt ) − Vbt (π)
≤ V (π ? ) − V (π) − Vbt (π ? ) − Vbt (π),

πt ) ≥ Vbt (π ? ) by the definition of π

where the inequality follows since Vbt (b bt . Next we apply Lemma 10 twice, once
?
with π and once with π to obtain
Lecture 8-9: 15

t
!
TN
d t (π) ≤ 2 ln δ + 4(e − 2)λK + (e − 2)λ
Reg(π) − Reg
1X
(bs−1 (π) + bs−1 (π ? ))
λt t s=1
t
!
2 ln TδN λ X
d s−1 (π ? ))
≤ + 4(e − 2)λK + (Reg
d s−1 (π) + Reg
λt 4γt ln T s=2
t
!
TN
2 ln δ λ X
?
≤ + 4(e − 2)λK + (2Reg(π) + 2Reg(π ) + 2s−1 )
λt 4γt ln T s=2
t
!
2 ln TδN λ λ X
= + 4(e − 2)λK + (Reg(π) + Reg(π ? )) + s−1 . (13)
λt 2γ ln T 2γt ln T s=2

Here the first inequality uses the definition of bs−1 (π) (and b0 (π) = 0), while the second inequality uses the inductive
hypothesis twice, once for π and once for π ? . In order to simplify further, we bound the sum of s−1 terms first. We
have

t t r !
X X K TN 32 TN
s−1 = 4 ln + ln
s=2 s=1
s−1 δ γ(s − 1) δ
r
TN 64 ln t T N
≤8 Kt ln + ln ,
δ γ δ
Pt √ √ Pt
since s=1 1/ s ≤ 2 t and s=1 1/s ≤ 2 ln t. Plugging this into our earlier bound (13) and noting that Reg(π ? ) =
0, we obtain

r !
TN
d t (π) ≤ 2 ln δ + 4(e − 2)λK +
Reg(π) − Reg
λ
Reg(π) +
λ
8 Kt ln
TN
+
64 ln t T N
ln .
λt 2γ ln T 2γt ln T δ γ δ

In order to collect terms, we pick λ = γ/4 ∈ [0, γ] and obtain

r !
TN
8 ln δ 1 1 T N 64 ln t T N
Reg(π) − Reg
d t (π) ≤ + (e − 2)γK + Reg(π) + 8 Kt ln + ln
γt 8 ln T 8t ln T δ γ δ
r
16 ln TδN Reg(π) K TN
≤ + +2 ln .
γt 2 t δ
q
K TN
In the last inequality, we have also used the definition of γ = t ln δ , along with algebraic simplifcations.
Rearranging terms completes the proof.

References
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert E Schapire. Taming the monster: A
fast and simple algorithm for contextual bandits. In ICML, 2014.
16 Lecture 8-9:

Miroslav Dudı́k, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang.
Efficient optimal learning for contextual bandits. In UAI, 2011.
Alexander Rakhlin and Karthik Sridharan. Bistro: An efficient relaxation-based method for contextual bandits. In
ICML, 2016.
Maurice Sion. On general minimax theorems. Pacific J. Math., 8(1):171–176, 1958.

Vasilis Syrgkanis, Akshay Krishnamurthy, and Robert E Schapire. Efficient algorithms for adversarial contextual
learning. In ICML, 2016a.
Vasilis Syrgkanis, Haipeng Luo, Akshay Krishnamurthy, and Robert E Schapire. Improved regret bounds for oracle-
based adversarial contextual bandits. In Advances in Neural Information Processing Systems, pages 3135–3143,
2016b.