Reinforcement Learning

Unit-1
Model Question & Answer
(Aimed to Cover 10 Questions of QB)

Sabyasachi Chakraborty

1) Explain PAC Learning Framework and Hypothesis Space

The Probably Approximately Correct (PAC) learning framework is a fundamental concept
in machine learning that provides a theoretical basis for understanding how algorithms
generalize from training data to unseen data. It offers probabilistic guarantees about the
performance of a learning algorithm in terms of accuracy and reliability.

Key Components of PAC Learning

1. Hypothesis Space (H)

The hypothesis space consists of all possible hypotheses (functions) that the learning
algorithm can consider as solutions.
o Example: For a binary classifier, H could include all possible decision
boundaries.
o A smaller hypothesis space is easier to search but may lack expressiveness; a
larger space increases flexibility but may lead to overfitting.
2. Error Tolerance (ϵ)
ϵ\epsilonϵ defines the maximum acceptable error rate for the hypothesis. For example,
an ϵ=0.05 means the hypothesis can misclassify up to 5% of instances.
3. Confidence (1−δ)
(1−δ) represents the probability that the hypothesis will perform within the error
tolerance on unseen data. δ=0.5 corresponds to a 95% confidence level.
4. Sample Complexity (m)
The number of training examples required to ensure the hypothesis is both
approximately correct and confident.
o Larger H or stricter ϵ and δ values increase the required m.

Mathematical Framework

The PAC learning framework provides the following inequality for the minimum number of
training samples (m):
 𝑚 ≥ 𝜖1( 𝑙𝑜𝑔 ∣ 𝐻 ∣ +𝑙𝑜𝑔𝛿1 )

Explanation of Terms:

 ∣H∣: The size of the hypothesis space (or its complexity, such as the Vapnik-
Chervonenkis (VC) dimension).
 ϵ : The error tolerance.
 δ : The probability of failure.
 This formula illustrates how the complexity of H, the desired accuracy (ϵ), and
confidence (1−δ1 influence the amount of data needed for learning.

Hypothesis Space and Generalization

1. Small Hypothesis Space:

o Easier to train but may lack flexibility (risk of underfitting).
o Example: Using linear functions for complex patterns.
2. Large Hypothesis Space:
o More expressive but prone to overfitting without sufficient training data.
o Example: Deep neural networks with many parameters.

Generalization in PAC Learning:

The goal is to balance the size of the hypothesis space and the number of training samples to
ensure good performance on unseen data.

2) Explain in detail the Multi-Armed Bandit and UCB Algorithm

with all mathematical rules and explanations

Solution :

Multi-Armed Bandit Problem

The multi-armed bandit problem is a classic problem in reinforcement learning and

decision theory. The problem is often illustrated using the analogy of a slot machine with
multiple arms, where each arm has a different, unknown reward distribution. The objective is
to choose which arm to pull (i.e., which action to take) in order to maximize the cumulative
reward over time.

Key Elements of the Multi-Armed Bandit Problem

1. Arms:
Each arm corresponds to an action, and each action (arm) has an associated unknown
 probability distribution of rewards. In a simple case, each arm has a fixed expected
reward but an unknown distribution.
2. Action Selection:
The decision-maker must decide which arm to pull at each step. The challenge is to
balance two competing objectives:
o Exploitation: Choose the arm with the highest known reward.
o Exploration: Choose arms that have been tried less frequently in order to
learn more about their reward distribution.
3. Reward Distribution:
Each arm i has a reward distribution with mean μ_i, and the goal is to maximize the
expected cumulative reward by selecting the best arm. The true mean of each arm is
unknown and must be estimated over time.
4. Goal:
Maximize the cumulative reward by selecting arms based on observed rewards, while
balancing exploration (to estimate the true mean rewards) and exploitation (to choose
the best arm based on current knowledge).

Upper Confidence Bound (UCB) Algorithm

The Upper Confidence Bound (UCB) algorithm is a popular method for solving the multi-
armed bandit problem. UCB balances exploration and exploitation by selecting arms based
on an upper confidence bound that considers both the estimated mean reward and the
uncertainty about that estimate.

Key Idea of UCB

The UCB algorithm selects the arm with the highest upper confidence bound. This upper
bound is calculated using both the estimated mean reward and its variability. The algorithm
encourages exploration for arms with higher uncertainty and exploitation for arms with
higher estimated rewards.

Mathematical Formulation of UCB

Steps in UCB Algorithm

Theoretical Analysis of UCB

Advantages of UCB Algorithm

 Theoretical Guarantee: UCB provides an upper bound on the cumulative regret,

ensuring efficient learning with provable performance.
 Efficient Exploration and Exploitation: UCB balances exploration (through the
confidence bound term) and exploitation (by choosing the arm with the highest mean
reward).
 Limitations of UCB Algorithm

 Computational Complexity: Calculating UCB requires keeping track of the mean

and the number of pulls for each arm, which can be computationally expensive if the
number of arms is large.
  Assumption of Known Reward Distributions: UCB assumes that the reward
distributions are stationary and independent, which may not hold in real-world
dynamic settings.
 Extensions to UCB

To address limitations like non-stationary environments (where reward distributions change

over time), modified versions of UCB have been proposed, such as:

 Sliding Window UCB: This version only considers the most recent rewards, making
it more suitable for environments where the reward distribution changes over time.
 UCB1-Tuned: A variant that adapts the exploration term based on the variance of the
rewards.

Conclusion

The UCB Algorithm is a powerful and widely used approach for solving the multi-armed
bandit problem, balancing exploration and exploitation. Its simplicity and strong theoretical
guarantees make it a preferred choice in many applications requiring a trade-off between
exploration and exploitation. By using the confidence bounds, UCB ensures that the
algorithm will quickly exploit the best-performing arms while still exploring less-tried arms
to refine the estimates and improve overall performance.

3 ) Explain Bandit Algorithms and their Real-World Applications

Bandit algorithms are a class of reinforcement learning algorithms used to solve the multi-
armed bandit problem. These algorithms are designed to balance the exploration of new
options (arms) with the exploitation of known options that provide high rewards. The goal is
to maximize the cumulative reward over time by selecting actions that offer the most benefit
while still learning about less-explored options.

Overview of Bandit Algorithms

The multi-armed bandit problem is typically framed as a scenario where there are multiple
actions (arms), each associated with an unknown probability distribution of rewards. The
decision maker must choose which action to take in each round to maximize the cumulative
reward. The challenge is to explore the arms (try them out) to estimate their rewards while
also exploiting the best-performing ones to maximize the accumulated reward.

Key Components of Bandit Algorithms:

1. Actions (Arms): The set of possible actions or arms to choose from. Each arm has an
associated reward distribution.
2. Exploration: Trying out different arms to gather information about their expected
rewards.
3. Exploitation: Choosing the arm with the highest observed reward based on past trials.
4. Regret: The difference between the reward obtained by the chosen actions and the
reward that would have been obtained by always choosing the best arm.

Types of Bandit Algorithms

There are several types of bandit algorithms, each focusing on different ways to balance
exploration and exploitation.

UCB focuses on arms that are uncertain (based on the size of the confidence interval), thus
encouraging exploration, while still exploiting the arms with high rewards.
3. Thompson Sampling
 Exploration vs. Exploitation: Thompson Sampling is a Bayesian approach that
probabilistically selects the arm to pull based on the posterior distributions of the
 rewards. It explores arms based on their potential to perform well, using the
distribution of past rewards.
 Mathematical Explanation: At each time step:
o Sample from the posterior distribution of each arm.
o Select the arm with the highest sampled value.

Thompson Sampling has shown to perform well empirically and is often used in practice
because it naturally balances exploration and exploitation.
4. Softmax Algorithm
 Exploration vs. Exploitation: In the Softmax approach, the probability of selecting an
arm is based on the estimated rewards, but arms with higher rewards are more likely
to be chosen. It introduces randomness into the decision-making process, making it a
more exploratory strategy.

Real-World Applications of Bandit Algorithms

Bandit algorithms are widely used in various domains where the goal is to make sequential
decisions with uncertain outcomes. Below are a few real-world applications:
1. Online Advertising
In online advertising, companies often need to choose which ads to display to users to
maximize click-through rates (CTR). Bandit algorithms, particularly UCB and Thompson
Sampling, are frequently used in ad selection. The system explores different ads to gather
information and exploits the best-performing ones to maximize revenue.
 Use case: A website showing advertisements chooses which ad to display to users
based on observed click-through rates (CTR). It balances showing ads that have
historically worked well with exploring new ones to gather more data.
2. Recommendation Systems
Recommendation systems (e.g., Netflix, Amazon) use bandit algorithms to suggest content
(movies, products) to users. The algorithm can use past user interactions (clicks, ratings) to
recommend items that are likely to generate the highest user engagement, while continuing
to explore new items to learn more about user preferences.
 Use case: A video streaming platform recommends new movies or shows to users
based on their previous viewing behavior and interactions. Bandit algorithms help to
dynamically adjust recommendations based on user feedback, maximizing user
engagement.

3. A/B Testing
A/B testing is a standard method in marketing and product development where two or more
versions of a webpage or app feature are tested with users to determine which one
performs best. Bandit algorithms are used to adaptively allocate traffic to the better-
performing variations over time.

 Use case: In a website redesign, two versions of the homepage are tested to see
which performs better in terms of conversions. Bandit algorithms can automatically
allocate more traffic to the version with a higher conversion rate, while continuing to
test the other version to learn more.
4. Robotics and Autonomous Systems

In robotics, bandit algorithms are used to control exploration and exploitation in tasks such
as path planning, robotic manipulation, or optimizing control parameters. Bandit algorithms
help the robot decide which actions to take based on uncertain information about the
environment.
 Use case: A robot exploring an environment uses bandit algorithms to decide which
direction to move in next, balancing the need to explore new areas while also
exploiting areas it has already identified as productive (e.g., with more objects of
interest).
5. Healthcare and Drug Trials
Bandit algorithms are increasingly used in clinical trials to optimize the allocation of patients
to different treatment arms. By selecting the most promising treatments based on observed
outcomes, the algorithm can improve patient outcomes while reducing the number of
patients who receive less effective treatments.
 Use case: In a clinical trial for a new drug, patients are dynamically assigned to either
the experimental treatment or a control treatment. Bandit algorithms ensure that
patients who receive the most effective treatments are prioritized.